Keyword extraction method and apparatus

Abstract

The embodiment of the present application provides a keyword extraction method and device, and relates to the technical field of data processing, wherein the method comprises the following steps: preprocessing original data to obtain to-be-extracted data; determining the language type of the to-be-extracted data; and extracting keywords from the to-be-extracted data based on the language type of the to-be-extracted data to obtain target keywords. The keyword extraction method and device provided by the embodiment of the present application automatically extract keywords from to-be-extracted data based on the language type of the to-be-extracted data, thereby improving the keyword extraction efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a keyword extraction method and apparatus. Background Technology

[0002] Indicators of compromise (IOCs) can be Internet Protocol (IP) addresses, domain names, Uniform Resource Locators (URLs), hashes, file paths, registry entries, etc. A single IOC hit can determine whether a Trojan virus is running on the corresponding host, or whether a host on the internal network has accessed the control command server of a botnet. Typically, keyword extraction from IOCs is used to detect whether a compromised host has been identified.

[0003] In related technologies, multiple preset keywords are stored in a vocabulary and context rules are defined. First, the data to be extracted is matched with the vocabulary to obtain candidate keywords in the data to be extracted. Then, the candidate keywords are filtered based on the context rules to obtain the final keywords.

[0004] However, the aforementioned technologies require manual formulation of word lists and context rules, resulting in low efficiency in keyword extraction. Summary of the Invention

[0005] To address the problems in the prior art, embodiments of the present invention provide a keyword extraction method and apparatus.

[0006] Specifically, the embodiments of the present invention provide the following technical solutions:

[0007] In a first aspect, embodiments of the present invention provide a keyword extraction method, including:

[0008] The raw data is preprocessed to obtain the data to be extracted;

[0009] Determine the language type of the data to be extracted;

[0010] Based on the language type of the data to be extracted, keywords are extracted from the data to obtain target keywords.

[0011] Furthermore, the preprocessing of the original data to obtain the data to be extracted includes:

[0012] Determine whether the raw data includes article data of the non-invalid index (IOC) type;

[0013] When the original data includes article data of non-IOC type, the non-IOC type article data is filtered to obtain filtered article data.

[0014] The filtered article data is converted into data in a preset format; the preset format data includes at least one of the following: title data, abstract data, link data, body text data, and keyword data;

[0015] The data in the preset format is determined as the data to be extracted.

[0016] Furthermore, the step of converting the filtered article data into data in the preset format includes:

[0017] The target article data is obtained by deleting the data after K segments from the filtered article data; where K is an integer greater than or equal to 1.

[0018] The target article data is converted into data in the preset format.

[0019] Further, the step of converting the target article data into data in the preset format includes:

[0020] Determine the information of each IOC in the target article data;

[0021] The corresponding IOC information is replaced with the target entity based on the type of each IOC information; the character length of the target entity is less than the character length of the corresponding IOC information;

[0022] The target article data obtained after replacement is converted into data in the preset format.

[0023] Furthermore, the step of extracting keywords from the data to be extracted based on the language type to obtain target keywords includes:

[0024] When the language type of the data to be extracted is English, the data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, which is output by the target semantic recognition model; the semantic vector is used to represent the contextual semantic information of the corresponding word and the position information of the corresponding word in the data to be extracted;

[0025] The semantic vector corresponding to each word in the data to be extracted is input into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model; the label data corresponding to each word includes one of the following: the position label of the keyword that is not divided into sub-words, the position label of the first sub-word of the keyword that is divided into sub-words, the position labels of the other sub-words of the keyword that is divided into sub-words except the first sub-word, and the position label of the non-keyword;

[0026] The target keywords in the data to be extracted are determined based on the tag data corresponding to each word.

[0027] The target semantic recognition model is trained based on training data samples and the labeled tags of each word in the training data samples; the target keyword extraction model is trained based on the labeled tags of each word in the training data samples and the semantic vector samples corresponding to each word.

[0028] Further, the step of inputting the data to be extracted into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, output by the target semantic recognition model, includes:

[0029] When the length of the data to be extracted is greater than a preset length, the data of the preset length is truncated starting from the beginning position of the data to be extracted, and a first preset character is added to the beginning position of the data of the preset length to obtain the target data to be extracted;

[0030] When the length of the data to be extracted is less than the preset length, a second preset character of the target length is added to the end position of the data to be extracted, and a first preset character is added to the beginning position of the data to be extracted, so as to obtain the target data to be extracted;

[0031] The target data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the target data to be extracted, output by the target semantic recognition model;

[0032] The step of inputting the semantic vector corresponding to each word in the data to be extracted into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model includes:

[0033] The semantic vector corresponding to each word in the target data to be extracted is input into the target keyword extraction model to obtain the tag data corresponding to the target word output by the target keyword extraction model; the target word includes words other than the first preset character and the second preset character.

[0034] Further, the training data samples and the labeled tags of each word in the training data samples are input into the initial semantic recognition model to obtain the semantic vector samples corresponding to each word output by the initial semantic recognition model;

[0035] The semantic vector sample corresponding to each word is input into the initial keyword extraction model to obtain the predicted label of each word output by the initial keyword extraction model;

[0036] The model parameters of the initial semantic recognition model and the initial keyword extraction model are optimized based on the labeled and predicted labels of each word to obtain the target semantic recognition model and the target keyword extraction model.

[0037] Further, before inputting the training data samples and the labeled words in the training data samples into the initial semantic recognition model, the method includes:

[0038] Obtain each keyword sample from the training data sample;

[0039] Determine the position of each keyword sample in the training data sample;

[0040] The training data samples are segmented to obtain sub-words corresponding to each word in the training data samples; when the word is a word that can be divided into sub-words, the word includes at least two sub-words, and a third preset character is added between adjacent sub-words in the at least two sub-words; when the word is a word that cannot be divided into sub-words, the sub-words are the word itself.

[0041] Each sub-word is labeled based on the position of each keyword sample, resulting in a label for each sub-word; the label for each word includes the label for each sub-word.

[0042] When the subword is the first subword in a keyword sample divided into subwords, the label of the subword includes the position label of the first subword in the keyword sample;

[0043] When the sub-word is any sub-word in the keyword sample that is divided into sub-words, excluding the first sub-word, the tag of the sub-word includes the position tag of the sub-word in the keyword sample excluding the first sub-word;

[0044] When the sub-word is a keyword sample that has not been divided into sub-words, the tagging label of the sub-word includes the position label of the keyword sample that has not been divided into sub-words;

[0045] When the subword is a non-keyword sample, the label of the subword includes the position label of the non-keyword sample.

[0046] Furthermore, the step of extracting keywords from the data to be extracted based on the language type to obtain target keywords includes:

[0047] When the language type of the data to be extracted is not English, each English word in the data to be extracted is extracted, and each English word is determined as a candidate keyword;

[0048] For each candidate keyword, determine the syntactic feature value of the candidate keyword;

[0049] The weights of the candidate keywords are determined based on their syntactic feature values.

[0050] The target keyword is determined based on the weight of each candidate keyword.

[0051] Further, determining the syntactic feature values ​​of the candidate keywords includes:

[0052] Determine whether the candidate keyword is located at at least one preset position in the data to be extracted;

[0053] When it is determined that the candidate keyword is located in at least one preset position in the data to be extracted, a score is determined for each preset position;

[0054] The score corresponding to each preset position is determined as the syntactic feature value of the candidate keyword;

[0055] The preset positions include at least one of the following: the position of the title data, the position before the first preset number of punctuation marks in the summary data, the position between the last second preset number of punctuation marks in the summary data, the position before the third preset number of punctuation marks in the body data, the position between the last fourth preset number of punctuation marks in the body data, and the position of the first M paragraphs of data in the body data; where M is an integer greater than or equal to 1.

[0056] Furthermore, determining the syntactic feature values ​​of the candidate keywords further includes:

[0057] Determine the word frequency of the candidate keywords in the data to be extracted;

[0058] The word frequencies of the candidate keywords are normalized, and the normalized values ​​are determined as the syntactic feature values ​​of the candidate keywords.

[0059] Further, determining the weight of the candidate keywords based on their syntactic feature values ​​includes:

[0060] The weight of a candidate keyword is determined by the sum of the products of each syntactic feature value and its corresponding coefficient; the coefficient is used to characterize the proportion of the corresponding syntactic feature value.

[0061] Further, determining the target keyword based on the weight of each candidate keyword includes:

[0062] For each candidate keyword, if the weight of the candidate keyword is greater than or equal to a preset threshold, the candidate keyword is determined as the target keyword.

[0063] Furthermore, after extracting keywords from the data to be extracted based on the language type to obtain target keywords, the method further includes:

[0064] Based on a preset list, determine whether there are any invalid keywords among all target keywords; the preset list stores at least one invalid keyword.

[0065] Once it is determined that there are invalid keywords among all target keywords, the invalid keywords among all target keywords are filtered out to obtain the filtered target keywords.

[0066] Secondly, embodiments of the present invention also provide a keyword extraction device, comprising:

[0067] The first processing unit is used to preprocess the raw data to obtain the data to be extracted;

[0068] The first determining unit is used to determine the language type of the data to be extracted;

[0069] The first extraction unit is used to extract keywords from the data to be extracted based on the language type of the data to be extracted, so as to obtain target keywords.

[0070] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the keyword extraction method as described in the first aspect.

[0071] Fourthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the keyword extraction method as described in the first aspect.

[0072] Fifthly, embodiments of the present invention also provide a computer program product having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the steps of the keyword extraction method described in the first aspect.

[0073] The keyword extraction method and apparatus provided in this invention identify the language type of the preprocessed data to be extracted, and automatically extract keywords based on the language type of the data to be extracted, thereby improving the efficiency of keyword extraction. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is one of the flowcharts illustrating the keyword extraction method provided in this embodiment of the invention;

[0076] Figure 2 This is a second flowchart illustrating the keyword extraction method provided in this embodiment of the invention;

[0077] Figure 3 This is the third flowchart illustrating the keyword extraction method provided in this embodiment of the invention;

[0078] Figure 4 This is the fourth flowchart illustrating the keyword extraction method provided in this embodiment of the invention;

[0079] Figure 5 This is a schematic diagram of the model structure provided in an embodiment of the present invention;

[0080] Figure 6 This is the fifth flowchart illustrating the keyword extraction method provided in this embodiment of the invention;

[0081] Figure 7 This is the sixth flowchart illustrating the keyword extraction method provided in this embodiment of the invention;

[0082] Figure 8 This is a schematic diagram of the keyword extraction device provided in an embodiment of the present invention;

[0083] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] Figure 1 This is one of the flowcharts illustrating the keyword extraction method provided in this embodiment of the invention, such as... Figure 1 As shown, this keyword extraction method includes the following steps:

[0086] Step 101: Preprocess the raw data to obtain the data to be extracted.

[0087] For example, the original data comes from the Internet. The original data is multilingual and unformatted, which means it contains noisy information. In order to prevent this noisy information from affecting the extraction of keywords, the original data needs to be preprocessed. The purpose of preprocessing is to filter out this noisy information and finally obtain the data to be extracted.

[0088] Step 102: Determine the language type of the data to be extracted.

[0089] For example, when the data to be extracted is obtained, the language of the data to be extracted is identified. If the language of the data to be extracted is English in a first preset proportion and the language of the data to be extracted is non-English in a second preset proportion, wherein the first preset proportion is greater than the second preset proportion, then the language type of the data to be extracted is determined to be English. If the language of the data to be extracted is non-English (Chinese, Japanese, Korean or Russian, etc.) in a first preset proportion and the language of the data to be extracted is English in a second preset proportion, then the language type of the data to be extracted is determined to be non-English.

[0090] Step 103: Extract keywords from the data to be extracted based on the language type to obtain target keywords.

[0091] If the data to be extracted includes IOC-type article data, the target keywords can be words related to IOC information, including "virus family name, attack behavior, vulnerability number, affected platform", etc.

[0092] For example, when the language type of the data to be extracted is determined to be English, the contextual semantic information and syntactic structure information of the data to be extracted are extracted, and the target keywords in the data to be extracted are determined based on the contextual semantic information and syntactic structure information of the data to be extracted; when the language type of the data to be extracted is determined to be non-English, each English word in the data to be extracted is obtained, and the target keywords in the data to be extracted are determined based on the syntactic structure information such as the position of each English word in the data to be extracted.

[0093] The keyword extraction method provided in this embodiment of the invention identifies the language type of the preprocessed data to be extracted, and automatically extracts keywords based on the language type of the data to be extracted, thereby improving the efficiency of keyword extraction.

[0094] Optionally, Figure 2 This is a second flowchart illustrating the keyword extraction method provided in this embodiment of the invention, such as... Figure 2 As shown, step 101 above can be implemented through the following steps:

[0095] Step 1011: Determine whether the original data includes article data of the non-invalid index (IOC) type.

[0096] For example, the raw data obtained may come from the security field, but this raw data is not necessarily all IOC type article data. IOC keywords cannot be extracted from non-IOC type article data, so it is necessary to filter the non-IOC type article data in the raw data. Specifically, a pre-defined thesaurus can be used for filtering. If the article data in the raw data contains words from the thesaurus, then the article data is considered to be of IOC type; if the article data in the raw data does not contain words from the thesaurus, then the article data is considered to be of non-IOC type.

[0097] It should be noted that before determining whether the original data includes non-IOC type article data, the HTML tags, emoticons, garbled characters, etc. contained in the original data can be cleaned to reduce the impact on subsequent keyword extraction. Data cleaning is performed using regular expressions and a preset database to clean the content in the original data that belongs to the preset database.

[0098] Step 1012: When the original data includes non-IOC type article data, filter the non-IOC type article data to obtain filtered article data.

[0099] For example, if the article data in the original data does not contain words from the thesaurus, the article data is considered to be of non-IOC type and is directly deleted. This process continues until all non-IOC type article data in the original data is deleted, and the IOC type article data is filtered out and used as the filtered article data.

[0100] Step 1013: Convert the filtered article data into data in a preset format.

[0101] The preset formatted data includes at least one of the following: title data, abstract data, link data, body text data, and keyword data.

[0102] The keyword data consists of the main words and phrases summarized manually from the article data.

[0103] For example, when the filtered article data is obtained, it is organized according to a preset format to obtain data in the preset format, specifically including at least one of the following: title data, abstract data, link data, body text data, and keyword data; for example, the preset format data is shown below:

[0104] "title":"Snake Keylogger Spreads Through Malicious PDFs",

[0105] "url":"https: / / threatpost.com / snake-keylogger-pdfs / 179703",

[0106] "text":"a recently discovered campaign uses a malicious PDF file and a 22-year-old Office bug to propagate the Snake Keylogger malware,researchershave found….https: / / docs.google.com / forms / d / e / ",

[0107] "keywords":["MALWARE","KEYLOGGER","SNAKE","XORDDOS"]

[0108] In this context, title represents the title, and Snake Keylogger Spreads Through Malicious PDFs represents the title data; url represents the link, and https: / / threatpost.com / snake-keylogger-pdfs / 179703 represents the link data; text represents the body text, and the data after text is the body text data; keywords represent keywords, and "MALWARE", "KEYLOGGER", "SNAKE", and "XORDDOS" represent keyword data.

[0109] It should be noted that when obtaining keyword data, since the keyword data was summarized by manual reading of articles, some keyword data was not originally in the article data. To improve the accuracy of subsequent keyword extraction, keyword data that does not exist in the article data needs to be deleted. Also, since keywords are all nouns, they can be singular or plural. Both singular and plural can be used as keywords, so part-of-speech tagging is required to obtain the singular and plural forms of all keywords. That is, the singular and plural forms of each keyword data are expanded, resulting in an expanded keyword set. For example, the original keywords include ["MALWARE","KEYLOGGER","SNAKE","XORDDOS"], and the expanded keywords include ["MALWARE","KEYLOGGER","SNAKE","XORDDOS","XORDDO"], where "MALWARE", "KEYLOGGER", and "SNAKE" are proper nouns and do not have plural forms.

[0110] Step 1014: Determine the data in the preset format as the data to be extracted.

[0111] The keyword extraction method provided in this invention filters out non-IOC type article data in the original data and converts the filtered article data into data in a preset format. On the one hand, it filters out noise information in the original data, improving the accuracy of subsequent keyword extraction; on the other hand, it extracts keywords from the preset format data, making it easier for electronic devices to process the data uniformly and simplifying the processing flow.

[0112] Optionally, step 1013 above can be implemented in the following ways:

[0113] Delete the data after K segments from the filtered article data to obtain the target article data; convert the target article data into the preset format.

[0114] Wherein, K is an integer greater than or equal to 1.

[0115] For example, IOC article data often contains a lot of code data, command operation data, file address data, etc. This data can interfere with electronic devices' keyword extraction, so it needs to be deleted. Manually reviewing IOC article data reveals that it typically uses a general-to-specific structure. The first K paragraphs mainly describe the article's main content, containing most of the keywords. The subsequent paragraphs mainly consist of case studies and specific IOC information, code data, command operation data, file address data, etc. The specific IOC information does not contain keywords, so the paragraphs after the first K paragraphs need to be filtered out. Here, K is a parameter that can be determined empirically; this invention does not impose any limitations on it.

[0116] The keyword extraction method provided in this embodiment of the invention deletes data after K segments of filtered article data to reduce interference from data that does not contain keywords to the extraction of keywords by electronic devices.

[0117] Optionally, the target article data can be converted into data in a preset format, which can be achieved in the following ways:

[0118] Determine the information of each IOC in the target article data;

[0119] The corresponding IOC information is replaced with the target entity based on the type of each IOC information; the character length of the target entity is less than the character length of the corresponding IOC information;

[0120] The target article data obtained after replacement is converted into data in the preset format.

[0121] For example, the content of IOC information often contains disorganized and unusual characters, and is quite long, which can interfere with keyword extraction. However, directly deleting the IOC information would affect the contextual semantic and syntactic structure information of the keywords. Therefore, this invention uses some special target entities to replace the IOC information. This way, the contextual semantic and syntactic structure information of the keywords can be fully preserved, while avoiding the impact of the specific content of the IOC information on keyword extraction. The specific target entities for IOC information replacement are shown in Table 1.

[0122] Table 1

[0123] URL https: / / docs.google.com / forms / d / e / S_URL IP 10.97.223.155 S_IP DOMAIN apache.org S_DOMAIN Hash 0925b3c05cef6d…663b9c02070e587b3f2d S_HASH CVE CVE-2014-2570 S_CVE CNVD CNVD-2022-47474 S_CVND

[0124] It should be noted that in Table 1, DOMAIN represents a domain name, CVE represents Common Vulnerabilities & Exposures (CVE), and CNVD represents the China National Vulnerability Database (CNVD), which is an information security vulnerability information sharing knowledge base jointly established by the National Computer Network Emergency Response Technical Team / Coordination Center. In Table 1 above, IOCs of URL type are replaced with S_URL, IOCs of IP type are replaced with S_IP, IOCs of DOMAIN type are replaced with S_DOMAIN, IOCs of Hash type are replaced with S_HASH, and IOCs of CNVD type are replaced with S_CVND.

[0125] The keyword extraction method provided in this embodiment of the invention replaces each IOC information with the corresponding target entity to simplify the complexity of the data to be extracted, thereby reducing interference with keyword extraction.

[0126] Optionally, Figure 3 This is the third flowchart illustrating the keyword extraction method provided in this embodiment of the invention, as shown below. Figure 3 As shown, step 103 above can be achieved through the following steps:

[0127] Step 1031: When the language type of the data to be extracted is English, the data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, which is output by the target semantic recognition model.

[0128] The semantic vector is used to represent the contextual semantic information of the corresponding word and the position information of the corresponding word in the data to be extracted.

[0129] Specifically, when the length of the data to be extracted is greater than a preset length, the data of the preset length is truncated from the starting position of the data to be extracted, and a first preset character is added to the starting position of the data of the preset length to obtain the target data to be extracted;

[0130] When the length of the data to be extracted is less than the preset length, a second preset character of the target length is added to the end position of the data to be extracted, and a first preset character is added to the beginning position of the data to be extracted, so as to obtain the target data to be extracted;

[0131] The target data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the target data to be extracted, which is output by the target semantic recognition model.

[0132] The first preset character can be [CLS], and the second preset character can be [PAD].

[0133] For example, since some paragraphs are very long, exceeding the input length requirement of the target semantic recognition model, regular expressions can be used to divide each paragraph in the data to be extracted into multiple sentences. Then, according to the input length requirement of the target semantic recognition model, each sentence begins with the character [CLS], and the sentence length is N (N is a parameter, set empirically). If the sentence length is less than N, the character [PAD] is used to expand it so that the sentence sample length is N; if the sentence length is greater than N, the sentence needs to be truncated to make its length N. For example, the original sentence is "sinclairbroadcast'disrupted by ransom##ware attack", and the modified sentence is "[CLS]sinclairbroadcast'disrupted by ransom##ware attack[PAD][PAD]…[PAD]". The target semantic recognition model inputs a corresponding number of sentences of length N. The target semantic recognition model performs feature analysis on each input sentence in conjunction with the context, and finally outputs a semantic vector for each word in each sentence, representing the contextual semantic information and the word's position information in the sentence.

[0134] Step 1032: Input the semantic vector corresponding to each word in the data to be extracted into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model.

[0135] The tag data corresponding to each word includes one of the following: the position tag of the keyword that is not divided into sub-words, the position tag of the first sub-word of the keyword that is divided into sub-words, the position tags of the other sub-words of the keyword that are divided into sub-words excluding the first sub-word, and the position tag of the non-keyword.

[0136] The target semantic recognition model is trained based on the training data samples and the labeled tags of each word in the training data samples; the target keyword extraction model is trained based on the labeled tags of each word in the training data samples and the semantic vector samples corresponding to each word.

[0137] Specifically, the semantic vector corresponding to each word in the target data to be extracted is input into the target keyword extraction model to obtain the tag data corresponding to the target word output by the target keyword extraction model; the target word includes words other than the first preset character and the second preset character.

[0138] For example, the semantic vector corresponding to each word output by the target semantic recognition model is input into the target keyword extraction model. The target keyword extraction model performs feature analysis on the semantic vector corresponding to each word and finally outputs the label data corresponding to each word. For example, the data input to the target semantic recognition model includes [CLS]sinclairbroadcast'disrupted by ransomware attack[PAD][PAD]…[PAD]. The label data output by the target keyword extraction model includes O S-KEY OOO B-KEY I-KEY O. Among them, [CLS] and each [PAD] do not have corresponding label data. S-KEY represents the position label of the keyword that has not been divided into sub-words. B-KEY represents the position label of the first sub-word of the keyword that has been divided into sub-words. I-KEY represents the position label of the other sub-words of the keyword that has been divided into sub-words except for the first sub-word. O represents the position label of the non-keyword that has not been divided into sub-words. In addition, all sub-words that have been divided into sub-words are also labeled as O. Additionally, the ## in ransomware indicates that the combination of the two preceding and following words belongs to one word, i.e., ransomware is a single word.

[0139] Step 1033: Determine the target keywords in the data to be extracted based on the tag data corresponding to each word.

[0140] For example, when the tag data corresponding to each word is obtained, the words corresponding to the tags representing keywords can be found in the corresponding sentences based on the tag data. The words corresponding to the tags representing keywords are all identified as target keywords in the data to be extracted. For example, in the example of step 1042 above, S-KEY, B-KEY and I-KEY in the tag data are used to represent keywords. Therefore, broadcast corresponding to S-KEY in "sinclair broadcast'disrupted by ransom##ware attack" is identified as the target keyword, ransom corresponding to B-KEY is identified as the target keyword, and ##ware corresponding to I-KEY is identified as the target keyword.

[0141] It should be noted that when the language of the data to be extracted is English, the final target keywords are in the form of sub-words, and are extracted on a sentence-by-sentence basis. Therefore, it is necessary to merge the sub-words into words, and the merged words are then used as the final target keywords. For example, the sub-words corresponding to B-KEY (ransom) and I-KEY (ware) are merged, and "ransomware" is used as the final target keyword.

[0142] The keyword extraction method provided in this invention determines the semantic vector corresponding to each word in the data to be extracted based on the target semantic recognition model, and then analyzes the semantic vector corresponding to each word based on the target keyword extraction model, finally obtaining the tag data corresponding to each word. Based on the tag data, the target keywords in the data to be extracted can be obtained, realizing the automatic extraction of IOC keywords in English data with high extraction efficiency.

[0143] Optionally, Figure 4 This is the fourth flowchart illustrating the keyword extraction method provided in this embodiment of the invention, as shown below. Figure 4 As shown, this keyword extraction method also includes the following steps:

[0144] Step 104: Input the training data sample and the annotation label of each word in the training data sample into the initial semantic recognition model to obtain the semantic vector sample corresponding to each word output by the initial semantic recognition model.

[0145] The initial semantic recognition model can be a BERT model, which is trained on a large amount of text data using an unsupervised algorithm and then saved. Through training, the BERT model can acquire semantic and syntactic information, allowing it to be directly transferred to various other tasks without needing to train model parameters from scratch.

[0146] The training data samples are preprocessed data samples. The specific preprocessing process is the same as the preprocessing process of the original data. That is, the training data samples also include data samples in a preset format. The data samples in the preset format include at least one of the following: title data samples, abstract data samples, link data samples, body text data samples, and keyword data samples.

[0147] For example, when training data samples are obtained, annotation methods such as "BIOS" can be used to label whether each word in the training data samples is a keyword, and obtain the annotation label of each word; then the training data samples and the annotation labels of each word are input into the initial semantic recognition model. After the initial semantic recognition model is processed, the semantic vector sample of each word can be obtained.

[0148] It should be noted that manual observation revealed that the text parameters in the URL often include keywords. Therefore, to improve the model's accuracy, the text parameters of the URL can be used as training data during model training to increase the amount of training data and make the model training more thorough. Specifically, the extraction of text parameters from the URL is based on rules summarized manually. For example, if the URL includes "https: / / threatpost.com / snake-keylogger-pdfs / 179703", the data between the last two forward slashes " / " can be used as the text parameter, that is, the text parameter of the URL is "snake-keylogger-pdfs".

[0149] It should be noted that the overall data sample can be divided into training data samples, test data samples, and validation data samples in an 8:1:1 ratio. The training data samples are used to train the model, the validation data samples are used to verify the model's performance in a timely manner during training to prevent overfitting, and the test data samples are used to test and evaluate the model's actual performance.

[0150] Step 105: Input the semantic vector sample corresponding to each word into the initial keyword extraction model to obtain the predicted label of each word output by the initial keyword extraction model.

[0151] The initial keyword extraction model can be a CRF model.

[0152] For example, when obtaining the semantic vector sample of each word, the semantic vector sample of each word is input into the CRF model to obtain the predicted label corresponding to each word in the final output of the CRF model.

[0153] It's important to note that before training the model, the training data samples need to be modified according to the input requirements of the BERT model. The training data samples should be divided into multiple sentence samples, each beginning with the character [CLS] and having a length of N (N is a parameter set empirically). If the sentence sample length is less than N, it is expanded using the character [PAD] to make the sentence sample length N; if the sentence sample length is greater than N, it needs to be truncated to make its length N. For example, the original sentence sample might include "sinclair broadcast'disrupted by ransom##ware attack," and the modified sentence sample might include "[CLS]sinclair broadcast'disrupted by ransom##ware attack[PAD][PAD]…[PAD]." Furthermore, in the labeled samples output by the CRF model, the characters [CLS] and [PAD] will not have corresponding labels. For example, if the training data samples input to the BERT model include [CLS]sinclair broadcast'disrupted byransom##ware attack[PAD][PAD]…[PAD], then the label samples output by the CRF model include O S-KEY OO OB-KEY I-KEY O.

[0154] Step 106: Optimize the model parameters of the initial semantic recognition model and the initial keyword extraction model based on the labeled and predicted labels of each word to obtain the target semantic recognition model and the target keyword extraction model.

[0155] For example, when the predicted label for each word is obtained, a loss function can be constructed based on the predicted label for each word and the labeled label for each word. The model parameters of the initial semantic recognition model and the initial keyword extraction model can be optimized based on the loss function until the convergence condition is met, and finally the target semantic recognition model and the target keyword extraction model are obtained.

[0156] Figure 5 This is a schematic diagram of the model structure provided in the embodiments of the present invention, such as... Figure 5 As shown, it includes an initial semantic recognition model (BERT model) and an initial keyword extraction model (CRF model). The BERT model takes the representation corresponding to each token as input. Figure 5 In the string, Tok 1, Tok 2, Tok 3...Tok N are all tokens for each string, and E1, E2, E3...E N These are the representations corresponding to the token, E

CLS

CLS

[0157] The keyword extraction method provided in this invention trains a target semantic recognition model and a target keyword extraction model based on training data samples and the labeled tags of each word in the training data samples. This facilitates the automatic extraction of keywords from the data to be extracted based on the target semantic recognition model and the target keyword extraction model, thereby improving the extraction efficiency and accuracy of keywords.

[0158] Optionally, before performing step 104 above, the keyword extraction method further includes the following steps:

[0159] Obtain each keyword sample from the training data sample;

[0160] Determine the position of each keyword sample in the training data sample;

[0161] The training data samples are segmented to obtain sub-words corresponding to each word in the training data samples; when the word is a word that can be divided into sub-words, the word includes at least two sub-words, and a third preset character is added between adjacent sub-words in the at least two sub-words; when the word is a word that cannot be divided into sub-words, the sub-words are the word itself.

[0162] Each sub-word is labeled based on the position of each keyword sample, resulting in a label for each sub-word; the label for each word includes the label for each sub-word.

[0163] When the subword is the first subword in a keyword sample divided into subwords, the label of the subword includes the position label of the first subword in the keyword sample;

[0164] When the sub-word is any sub-word in the keyword sample that is divided into sub-words, excluding the first sub-word, the tag of the sub-word includes the position tag of the sub-word in the keyword sample excluding the first sub-word;

[0165] When the sub-word is a keyword sample that has not been divided into sub-words, the tagging label of the sub-word includes the position label of the keyword sample that has not been divided into sub-words;

[0166] When the subword is a non-keyword sample, the label of the subword includes the position label of the non-keyword sample.

[0167] The third preset character can be ##.

[0168] For example, first, regular expressions are used to match each keyword sample and sentence sample to find the position of each keyword sample in the sentence sample. Then, a word segmentation tool is used to divide all words in the sentence sample into sub-word forms. For example, the original sentence sample includes "sinclair broadcast'disrupted by ransomware attack," and the segmented sub-word forms include "sinclair broadcast'disrupted by ransomware attack." attack; for example, if sinclair is an indivisible word, then the subword corresponding to sinclair is still sinclair; ransomware can be divided into two subwords, ransom and ware. The second subword ware starts with "##", indicating the connection relationship between this subword and the preceding subword; finally, each subword is labeled based on the position of each keyword sample. When labeling, if the keyword sample can be divided into multiple subwords, then the first subword is labeled as "B-KEY", and the other subwords are labeled as "I-KEY"; if the keyword is not divided into subwords, then it is labeled as "S-KEY"; all non-keywords are labeled as "O", and all subwords that make up non-keywords are also labeled as "O". For example, the original sentence sample includes sinclair broadcast'disrupted by ransom##ware attack, and the label of each subword includes O S-KEY OOO B-KEY I-KEY O.

[0169] The keyword extraction method provided in this invention uses sub-words for annotation, which, compared to annotation by word, can solve the problem of semantic loss caused by a large number of out-of-vocabulary words in IOC domain data, thereby improving the accuracy of keyword extraction.

[0170] Optionally, Figure 6 This is the fifth flowchart illustrating the keyword extraction method provided in this embodiment of the invention, as shown below. Figure 6 As shown, step 103 above can be implemented through the following steps:

[0171] Step 1034: When the language type of the data to be extracted is not English, extract each English word in the data to be extracted and determine each English word as a candidate keyword.

[0172] For example, when the language of the data to be extracted is not English, regular expressions can be used to extract English words from the non-English IOC data. In IOC articles in Chinese, Korean, Russian, and Japanese, English words are used to describe IOC terms. Therefore, every English word contained in IOC articles in these languages ​​can be extracted, and each extracted English word can be used as a candidate keyword.

[0173] Step 1035: For each candidate keyword, determine the syntactic feature value of the candidate keyword.

[0174] Among them, the syntactic feature value is determined based on the position of each candidate keyword in the sentence.

[0175] For example, for each candidate keyword, determine the position of the candidate keyword in the data to be extracted, give a score based on the position of the candidate keyword in the data to be extracted, and use each given score as the syntactic feature value of the candidate keyword.

[0176] Step 1036: Determine the weight of the candidate keywords based on their syntactic feature values.

[0177] Specifically, the sum of the products of each syntactic feature value of the candidate keyword and its corresponding coefficient is determined as the weight of the candidate keyword.

[0178] The coefficient is used to characterize the proportion of the corresponding syntactic feature value.

[0179] For example, when obtaining the syntactic feature values ​​of each candidate keyword, the weight of each candidate keyword is obtained by weighted summation of the syntactic feature values ​​of each candidate keyword. Specifically, the weight of each candidate keyword can be calculated using the following formula (1).

[0180] The weight of a candidate keyword = α1w1 + α2w2 + α3w3 + α4w4...α n w n (1)

[0181] Among them, α1, α2, α3, α4...α n All are constants, representing the proportion of each syntactic feature value, α1, α2, α3, α4...α n The values ​​can be obtained experimentally, w1, w2, w3, w4...w n denoted as each of the syntactic feature values ​​of the candidate keyword, and n is the number of syntactic feature values ​​of the candidate keyword.

[0182] Step 1037: Determine the target keyword based on the weight of each candidate keyword.

[0183] Specifically, for each candidate keyword, when the weight of the candidate keyword is determined to be greater than or equal to a preset threshold, the candidate keyword is determined to be the target keyword.

[0184] For example, when determining the weight of each candidate keyword, the weight of each candidate keyword is compared with a preset threshold. If the weight of a candidate keyword is greater than or equal to the preset threshold, the candidate keyword is determined as the final target keyword; if the weight of a candidate keyword is less than the preset threshold, the candidate keyword is deleted.

[0185] The keyword extraction method provided in this invention uses syntactic structure information to determine multiple syntactic feature values ​​of candidate keywords, integrates multiple syntactic feature values ​​to obtain the weight of candidate keywords, and uses a preset threshold to filter out suitable target keywords, making the keyword extraction results more robust.

[0186] Optionally, step 1035 above can be implemented in the following ways:

[0187] Determine whether the candidate keyword is located at at least one preset position in the data to be extracted;

[0188] When it is determined that the candidate keyword is located in at least one preset position in the data to be extracted, a score is determined for each preset position;

[0189] The score corresponding to each preset position is determined as the syntactic feature value of the candidate keyword;

[0190] The preset positions include at least one of the following: the position of the title data, the position before the first preset number of punctuation marks in the summary data, the position between the last second preset number of punctuation marks in the summary data, the position before the third preset number of punctuation marks in the body data, the position between the last fourth preset number of punctuation marks in the body data, and the position of the first M paragraphs of data in the body data; where M is an integer greater than or equal to 1.

[0191] Further, determine the word frequency of the candidate keywords in the data to be extracted;

[0192] The word frequencies of the candidate keywords are normalized, and the normalized values ​​are determined as the syntactic feature values ​​of the candidate keywords.

[0193] For example, when each candidate keyword is obtained, its position in the data to be extracted is determined, specifically as follows:

[0194] (1) Determine whether the candidate keyword exists in the title data. If the candidate keyword exists in the title data, a score w1 is given; if the candidate keyword does not exist in the title data, a score 0 is given.

[0195] (2) Determine whether the candidate keyword exists in the position before the first preset number of punctuation marks in the summary data. If it exists in the position before the first preset number of punctuation marks in the summary data, a score w2 is given; if it does not exist in the position before the first preset number of punctuation marks in the summary data, a score of 0 is given.

[0196] (3) Determine whether the candidate keyword exists in the position between the last second preset number of punctuation marks in the summary data. If it exists in the position between the last second preset number of punctuation marks in the summary data, a score of w3 is given; if it does not exist in the position between the last second preset number of punctuation marks in the summary data, a score of 0 is given.

[0197] (4) Determine whether the candidate keyword exists in the text before the third preset number of punctuation marks. If it exists in the text before the third preset number of punctuation marks, a score of w4 is given; if it does not exist in the text before the third preset number of punctuation marks, a score of 0 is given.

[0198] (5) Determine whether the candidate keyword exists in the position between the last four preset number of punctuation marks in the main text data. If it exists in the position between the last four preset number of punctuation marks in the main text data, a score of w5 is given; if it does not exist in the position between the last four preset number of punctuation marks in the main text data, a score of 0 is given.

[0199] (6) Determine whether the candidate keyword is located in the first M paragraphs of the main text data. If it exists in the first M paragraphs of the main text data, a score of w6 is given; if it does not exist in the first M paragraphs of the main text data, a score of 0 is given.

[0200] (7) It can also determine the word frequency of the candidate keyword in the data to be extracted, that is, the number of times the candidate keyword appears in the data to be extracted, and then normalize the word frequency corresponding to each candidate keyword to obtain a given score w7; for example, w7 corresponding to a word frequency of 10 is equal to 1, w7 corresponding to a word frequency of 8 is equal to 0.8, w7 corresponding to a word frequency of 6 is equal to 0.6, and so on.

[0201] It should be noted that the present invention does not limit the values ​​of the first preset quantity, the second preset quantity, the third preset quantity and the fourth preset quantity, and can be set according to actual needs.

[0202] The keyword extraction method provided in this invention utilizes syntactic structure information to determine multiple syntactic feature values ​​of candidate keywords. That is, multiple syntactic feature values ​​are determined based on the specific position of the candidate keyword in the data to be extracted and the number of times the candidate keyword appears. This facilitates obtaining the weight of the candidate keyword based on multiple syntactic feature values, thereby determining the importance of each candidate keyword. Furthermore, suitable target keywords are selected through a preset threshold, thus improving the accuracy of keyword extraction.

[0203] Optionally, Figure 7 This is the sixth flowchart illustrating the keyword extraction method provided in this embodiment of the invention, as shown below. Figure 7 As shown, after step 103 above, this keyword extraction method further includes the following steps:

[0204] Step 107: Determine whether there are any invalid keywords among all target keywords based on the preset list; the preset list stores at least one invalid keyword.

[0205] Step 108: When it is determined that there are invalid keywords in all target keywords, filter out the invalid keywords in all target keywords to obtain the filtered target keywords.

[0206] For example, since some keywords may meet the definition of IOC keywords, such as malware, IP, hash, ransomware, etc., but these words are not needed in subsequent tasks, a preset list containing multiple invalid keywords can be created in advance. This preset list can be used to filter all the extracted target keywords to obtain the filtered keywords. Each of the filtered keywords can be used as the final keyword in the data to be extracted. For example, if all target keywords include ["broadcast", "ransomware"], then the final keyword will include ["broadcast"].

[0207] The keyword extraction method provided in this invention filters all the obtained target keywords based on a preset list that includes multiple invalid keywords, making the final target keywords more accurate and easier to apply to subsequent tasks.

[0208] Figure 8 This is a schematic diagram of the keyword extraction device provided in an embodiment of the present invention, such as... Figure 8 As shown, the keyword extraction device includes a first processing unit 801, a first determining unit 802, and a first extraction unit 803; wherein:

[0209] The first processing unit 801 is used to preprocess the raw data to obtain the data to be extracted;

[0210] The first determining unit 802 is used to determine the language type of the data to be extracted;

[0211] The first extraction unit 803 is used to extract keywords from the data to be extracted based on the language type of the data to be extracted, so as to obtain target keywords.

[0212] The keyword extraction device provided in this embodiment of the invention identifies the language type of the preprocessed data to be extracted, and automatically extracts keywords based on the language type of the data to be extracted, thereby improving the efficiency of keyword extraction.

[0213] Based on any of the above embodiments, the first processing unit 801 is specifically used for:

[0214] Determine whether the raw data includes article data of the non-invalid index (IOC) type;

[0215] When the original data includes article data of non-IOC type, the non-IOC type article data is filtered to obtain filtered article data.

[0216] The filtered article data is converted into data in a preset format; the preset format data includes at least one of the following: title data, abstract data, link data, body text data, and keyword data;

[0217] The data in the preset format is determined as the data to be extracted.

[0218] Based on any of the above embodiments, the first processing unit 801 is further specifically used for:

[0219] The target article data is obtained by deleting the data after K segments from the filtered article data; where K is an integer greater than or equal to 1.

[0220] The target article data is converted into data in the preset format.

[0221] Based on any of the above embodiments, the first processing unit 801 is further specifically used for:

[0222] Determine the information of each IOC in the target article data;

[0223] The corresponding IOC information is replaced with the target entity based on the type of each IOC information; the character length of the target entity is less than the character length of the corresponding IOC information;

[0224] The target article data obtained after replacement is converted into data in the preset format.

[0225] Based on any of the above embodiments, the first extraction unit 802 is specifically used for:

[0226] When the language type of the data to be extracted is English, the data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, which is output by the target semantic recognition model; the semantic vector is used to represent the contextual semantic information of the corresponding word and the position information of the corresponding word in the data to be extracted;

[0227] The semantic vector corresponding to each word in the data to be extracted is input into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model; the label data corresponding to each word includes one of the following: the position label of the keyword that is not divided into sub-words, the position label of the first sub-word of the keyword that is divided into sub-words, the position labels of the other sub-words of the keyword that is divided into sub-words except the first sub-word, and the position label of the non-keyword;

[0228] The target keywords in the data to be extracted are determined based on the tag data corresponding to each word.

[0229] The target semantic recognition model is trained based on training data samples and the labeled tags of each word in the training data samples; the target keyword extraction model is trained based on the labeled tags of each word in the training data samples and the semantic vector samples corresponding to each word.

[0230] Based on any of the above embodiments, the first extraction unit 802 is specifically used for:

[0231] When the length of the data to be extracted is greater than a preset length, the data of the preset length is truncated starting from the beginning position of the data to be extracted, and a first preset character is added to the beginning position of the data of the preset length to obtain the target data to be extracted;

[0232] When the length of the data to be extracted is less than the preset length, a second preset character of the target length is added to the end position of the data to be extracted, and a first preset character is added to the beginning position of the data to be extracted, so as to obtain the target data to be extracted;

[0233] The target data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the target data to be extracted, output by the target semantic recognition model;

[0234] The semantic vector corresponding to each word in the target data to be extracted is input into the target keyword extraction model to obtain the tag data corresponding to the target word output by the target keyword extraction model; the target word includes words other than the first preset character and the second preset character.

[0235] Based on any of the above embodiments, the device further includes:

[0236] The recognition unit is used to input the training data sample and the annotation label of each word in the training data sample into the initial semantic recognition model to obtain the semantic vector sample corresponding to each word output by the initial semantic recognition model;

[0237] The second extraction unit is used to input the semantic vector sample corresponding to each word into the initial keyword extraction model to obtain the predicted label of each word output by the initial keyword extraction model;

[0238] An optimization unit is used to optimize the model parameters of the initial semantic recognition model and the model parameters of the initial keyword extraction model based on the annotation label and the predicted label of each word, so as to obtain the target semantic recognition model and the target keyword extraction model.

[0239] Based on any of the above embodiments, the device further includes:

[0240] An acquisition unit is used to acquire each keyword sample in the training data sample;

[0241] The second determining unit is used to determine the position of each keyword sample in the training data sample;

[0242] The second processing unit is used to perform word segmentation on the training data samples to obtain sub-words corresponding to each word in the training data samples; when the word is a word that can be divided into sub-words, the word includes at least two sub-words, and a third preset character is added between adjacent sub-words in the at least two sub-words; when the word is a word that cannot be divided into sub-words, the sub-words are the word itself.

[0243] The annotation unit is used to annotate each of the sub-words based on the position of each keyword sample, so as to obtain the annotation label of each sub-word; the annotation label of each word includes the annotation label of each sub-word;

[0244] When the subword is the first subword in a keyword sample divided into subwords, the label of the subword includes the position label of the first subword in the keyword sample;

[0245] When the sub-word is any sub-word in the keyword sample that is divided into sub-words, excluding the first sub-word, the tag of the sub-word includes the position tag of the sub-word in the keyword sample excluding the first sub-word;

[0246] When the sub-word is a keyword sample that has not been divided into sub-words, the tagging label of the sub-word includes the position label of the keyword sample that has not been divided into sub-words;

[0247] When the subword is a non-keyword sample, the label of the subword includes the position label of the non-keyword sample.

[0248] Based on any of the above embodiments, the first extraction unit 802 is further specifically used for:

[0249] When the language type of the data to be extracted is not English, each English word in the data to be extracted is extracted, and each English word is determined as a candidate keyword;

[0250] For each candidate keyword, determine the syntactic feature value of the candidate keyword;

[0251] The weights of the candidate keywords are determined based on their syntactic feature values.

[0252] The target keyword is determined based on the weight of each candidate keyword.

[0253] Based on any of the above embodiments, the first extraction unit 802 is further specifically used for:

[0254] Determine whether the candidate keyword is located at at least one preset position in the data to be extracted;

[0255] When it is determined that the candidate keyword is located in at least one preset position in the data to be extracted, a score is determined for each preset position;

[0256] The score corresponding to each preset position is determined as the syntactic feature value of the candidate keyword;

[0257] The preset positions include at least one of the following: the position of the title data, the position before the first preset number of punctuation marks in the summary data, the position between the last second preset number of punctuation marks in the summary data, the position before the third preset number of punctuation marks in the body data, the position between the last fourth preset number of punctuation marks in the body data, and the position of the first M paragraphs of data in the body data; where M is an integer greater than or equal to 1.

[0258] Based on any of the above embodiments, the first extraction unit 802 is further specifically used for:

[0259] Determine the word frequency of the candidate keywords in the data to be extracted;

[0260] The word frequencies of the candidate keywords are normalized, and the normalized values ​​are determined as the syntactic feature values ​​of the candidate keywords.

[0261] Based on any of the above embodiments, the first extraction unit 802 is further specifically used for:

[0262] The weight of a candidate keyword is determined by the sum of the products of each syntactic feature value and its corresponding coefficient; the coefficient is used to characterize the proportion of the corresponding syntactic feature value.

[0263] Based on any of the above embodiments, the first extraction unit 802 is further specifically used for:

[0264] For each candidate keyword, if the weight of the candidate keyword is greater than or equal to a preset threshold, the candidate keyword is determined as the target keyword.

[0265] Based on any of the above embodiments, the device further includes:

[0266] The third determining unit is used to determine whether there are invalid keywords among all target keywords based on a preset list; the preset list stores at least one invalid keyword.

[0267] The filtering unit is used to filter out the invalid keywords from all target keywords when it is determined that there are invalid keywords among all target keywords, so as to obtain the filtered target keywords.

[0268] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 9 As shown, the electronic device may include a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute the following method: preprocessing the raw data to obtain the data to be extracted;

[0269] Determine the language type of the data to be extracted;

[0270] Based on the language type of the data to be extracted, keywords are extracted from the data to obtain target keywords.

[0271] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0272] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the keyword extraction method provided by the above methods, the method including: preprocessing the raw data to obtain the data to be extracted;

[0273] Determine the language type of the data to be extracted;

[0274] Based on the language type of the data to be extracted, keywords are extracted from the data to obtain target keywords.

[0275] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the keyword extraction methods provided by the above methods, the method comprising: preprocessing the raw data to obtain the data to be extracted;

[0276] Determine the language type of the data to be extracted;

[0277] Based on the language type of the data to be extracted, keywords are extracted from the data to obtain target keywords.

[0278] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0279] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0280] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A keyword extraction method, characterized in that, include: The raw data is preprocessed to obtain the data to be extracted; Determine the language type of the data to be extracted; Based on the language type of the data to be extracted, keywords are extracted from the data to be extracted to obtain target keywords; The preprocessing of the raw data to obtain the data to be extracted includes: Determine whether the raw data includes article data of the non-invalid index (IOC) type; When the original data includes article data of the non-lost index IOC type, the article data of the non-lost index IOC type is filtered to obtain the filtered article data. The filtered article data is converted into data in a preset format; the preset format data includes at least one of the following: title data, abstract data, link data, body text data, and keyword data; The data in the preset format is determined as the data to be extracted; The step of extracting keywords from the data to be extracted based on the language type to obtain target keywords includes: When the language type of the data to be extracted is English, the data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, output by the target semantic recognition model; the semantic vector is used to represent the contextual semantic information of the corresponding word and the position information of the corresponding word in the data to be extracted; the semantic vector corresponding to each word in the data to be extracted is input into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted, output by the target keyword extraction model; the label data corresponding to each word includes one of the following: the position label of the keyword not divided into sub-words, the position label of the first sub-word of the keyword divided into sub-words, the position labels of the other sub-words of the keyword divided into sub-words excluding the first sub-word, and the position label of the non-keyword; the target keyword in the data to be extracted is determined based on the label data corresponding to each word; wherein, the target semantic recognition model is trained based on the training data sample and the labeled labels of each word in the training data sample; the target keyword extraction model is trained based on the labeled labels of each word in the training data sample and the semantic vector sample corresponding to each word; When the language type of the data to be extracted is not English, each English word in the data to be extracted is extracted and each English word is identified as a candidate keyword; for each candidate keyword, the syntactic feature value of the candidate keyword is determined; the weight of the candidate keyword is determined based on the syntactic feature value of the candidate keyword; and the target keyword is determined based on the weight of each candidate keyword.

2. The keyword extraction method according to claim 1, characterized in that, The step of converting the filtered article data into the preset format includes: The target article data is obtained by deleting the data after K segments from the filtered article data; where K is an integer greater than or equal to 1. The target article data is converted into data in the preset format.

3. The keyword extraction method according to claim 2, characterized in that, The step of converting the target article data into the preset format includes: Determine the information of each IOC in the target article data; The corresponding IOC information is replaced with the target entity based on the type of each IOC information; the character length of the target entity is less than the character length of the corresponding IOC information; The target article data obtained after replacement is converted into data in the preset format.

4. The keyword extraction method according to claim 1, characterized in that, The step of inputting the data to be extracted into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted output by the target semantic recognition model includes: When the length of the data to be extracted is greater than a preset length, the data of the preset length is truncated starting from the beginning position of the data to be extracted, and a first preset character is added to the beginning position of the data of the preset length to obtain the target data to be extracted; When the length of the data to be extracted is less than the preset length, a second preset character of the target length is added to the end position of the data to be extracted, and a first preset character is added to the beginning position of the data to be extracted, so as to obtain the target data to be extracted; The target data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the target data to be extracted, output by the target semantic recognition model; The step of inputting the semantic vector corresponding to each word in the data to be extracted into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model includes: The semantic vector corresponding to each word in the target data to be extracted is input into the target keyword extraction model to obtain the tag data corresponding to the target word output by the target keyword extraction model; the target word includes words other than the first preset character and the second preset character.

5. The keyword extraction method according to claim 1, characterized in that, The method further includes: The training data samples and the labeled tags of each word in the training data samples are input into the initial semantic recognition model to obtain the semantic vector samples corresponding to each word output by the initial semantic recognition model; The semantic vector sample corresponding to each word is input into the initial keyword extraction model to obtain the predicted label of each word output by the initial keyword extraction model; The model parameters of the initial semantic recognition model and the initial keyword extraction model are optimized based on the labeled and predicted labels of each word to obtain the target semantic recognition model and the target keyword extraction model.

6. The keyword extraction method according to claim 5, characterized in that, Before inputting the training data samples and the labeled tags of each word in the training data samples into the initial semantic recognition model, the method includes: Obtain each keyword sample from the training data sample; Determine the position of each keyword sample in the training data sample; The training data samples are segmented to obtain sub-words corresponding to each word in the training data samples; when the word is a word that can be divided into sub-words, the word includes at least two sub-words, and a third preset character is added between adjacent sub-words in the at least two sub-words; when the word is a word that cannot be divided into sub-words, the sub-words are the word itself. Each sub-word is labeled based on the position of each keyword sample, resulting in a label for each sub-word; the label for each word includes the label for each sub-word. When the subword is the first subword in a keyword sample divided into subwords, the label of the subword includes the position label of the first subword in the keyword sample; When the sub-word is any sub-word in the keyword sample that is divided into sub-words, excluding the first sub-word, the tag of the sub-word includes the position tag of the sub-word in the keyword sample excluding the first sub-word; When the sub-word is a keyword sample that has not been divided into sub-words, the tagging label of the sub-word includes the position label of the keyword sample that has not been divided into sub-words; When the subword is a non-keyword sample, the label of the subword includes the position label of the non-keyword sample.

7. The keyword extraction method according to claim 1, characterized in that, Determining the syntactic feature values ​​of the candidate keywords includes: Determine whether the candidate keyword is located at at least one preset position in the data to be extracted; When it is determined that the candidate keyword is located in at least one preset position in the data to be extracted, a score is determined for each preset position; The score corresponding to each preset position is determined as the syntactic feature value of the candidate keyword; The preset positions include at least one of the following: the position of the title data, the position before the first preset number of punctuation marks in the summary data, the position between the last second preset number of punctuation marks in the summary data, the position before the third preset number of punctuation marks in the body data, the position between the last fourth preset number of punctuation marks in the body data, and the position of the first M paragraphs of data in the body data; where M is an integer greater than or equal to 1.

8. The keyword extraction method according to claim 7, characterized in that, The determination of the syntactic feature values ​​of the candidate keywords further includes: Determine the word frequency of the candidate keywords in the data to be extracted; The word frequencies of the candidate keywords are normalized, and the normalized values ​​are determined as the syntactic feature values ​​of the candidate keywords.

9. The keyword extraction method according to claim 7 or 8, characterized in that, Determining the weight of the candidate keywords based on their syntactic feature values ​​includes: The weight of a candidate keyword is determined by the sum of the products of each syntactic feature value and its corresponding coefficient; the coefficient is used to characterize the proportion of the corresponding syntactic feature value.

10. The keyword extraction method according to claim 1, characterized in that, Determining the target keyword based on the weight of each candidate keyword includes: For each candidate keyword, if the weight of the candidate keyword is greater than or equal to a preset threshold, the candidate keyword is determined as the target keyword.

11. The keyword extraction method according to claim 1, characterized in that, After extracting keywords from the data to be extracted based on the language type to obtain target keywords, the method further includes: Based on a preset list, determine whether there are any invalid keywords among all target keywords; the preset list stores at least one invalid keyword. Once it is determined that there are invalid keywords among all target keywords, the invalid keywords among all target keywords are filtered out to obtain the filtered target keywords.

12. A keyword extraction device, characterized in that, include: The first processing unit is used to preprocess the raw data to obtain the data to be extracted; The first determining unit is used to determine the language type of the data to be extracted; The first extraction unit is used to extract keywords from the data to be extracted based on the language type of the data to be extracted, so as to obtain target keywords; The first processing unit is specifically used for: Determine whether the raw data includes article data of the non-invalid index (IOC) type; When the original data includes article data of the non-lost index IOC type, the article data of the non-lost index IOC type is filtered to obtain the filtered article data. The filtered article data is converted into data in a preset format; the preset format data includes at least one of the following: title data, abstract data, link data, body text data, and keyword data; The data in the preset format is determined as the data to be extracted; The first extraction unit is specifically used for: When the language type of the data to be extracted is English, the data to be extracted is input into the target semantic recognition model to obtain the semantic vector corresponding to each word in the data to be extracted, which is output by the target semantic recognition model; the semantic vector is used to represent the contextual semantic information of the corresponding word and the position information of the corresponding word in the data to be extracted; The semantic vector corresponding to each word in the data to be extracted is input into the target keyword extraction model to obtain the label data corresponding to each word in the data to be extracted output by the target keyword extraction model; the label data corresponding to each word includes one of the following: the position label of the keyword that is not divided into sub-words, the position label of the first sub-word of the keyword that is divided into sub-words, the position labels of the other sub-words of the keyword that is divided into sub-words except the first sub-word, and the position label of the non-keyword; The target keywords in the data to be extracted are determined based on the label data corresponding to each word; wherein, the target semantic recognition model is trained based on the training data samples and the labeled tags of each word in the training data samples; the target keyword extraction model is trained based on the labeled tags of each word in the training data samples and the semantic vector samples corresponding to each word; When the language type of the data to be extracted is not English, each English word in the data to be extracted is extracted and each English word is identified as a candidate keyword; for each candidate keyword, the syntactic feature value of the candidate keyword is determined; the weight of the candidate keyword is determined based on the syntactic feature value of the candidate keyword; and the target keyword is determined based on the weight of each candidate keyword.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the keyword extraction method as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the keyword extraction method as described in any one of claims 1 to 11.

15. A computer program product having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to implement the keyword extraction method as described in any one of claims 1 to 11.

Images