Keyword group extraction method and device for cyberspace asset retrieval

Abstract

The embodiment of the invention provides a cyberspace asset retrieval-oriented keyword extraction method and device. The method comprises the following steps of: obtaining a plurality of target text entries of a target site for cyberspace asset retrieval; performing word segmentation on the target text entries through a pre-trained improved hidden Markov model to obtain a target word segmentation result corresponding to each target text entry; combining the target word segmentation results to generate a plurality of target word groups; through a text mining technology, according to a plurality of target text entries, mining statistics is carried out on a plurality of target word groups, a target keyword group is generated, keyword group extraction is carried out based on an improved Markov model and the text mining technology, the accuracy of a word segmentation result is improved, and therefore the target keyword group is accurately extracted; and the accuracy of the asset retrieval result is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a keyword group extraction method and device for network space asset retrieval. Background Art

[0002] Cyberspace asset retrieval technology is a technical means specifically used to search, identify and locate various types of assets in the global Internet space. These assets include but are not limited to Internet Protocol (IP) addresses, domain names, servers, applications, databases and cloud services. The common asset detection method is to enter keywords in the network asset retrieval engine to match and query the target assets. The determination of keywords is crucial to the accuracy of cyberspace asset retrieval results. In related technologies, sentence segmentation is usually performed based on rules or statistics, and then keywords are extracted. However, the rule-based method requires a manually established word segmentation dictionary and word segmentation rule library in advance, which is highly dependent on manual labor and has a low word segmentation accuracy; the statistical-based method, such as the hidden Markov model, has two internal independence assumptions that are too absolute and are not accurate in practical applications, resulting in low word segmentation accuracy, which further leads to low accuracy of asset retrieval results. Summary of the invention

[0003] One object of the present invention is to provide a keyword group extraction method for cyberspace asset retrieval, which extracts keyword groups based on an improved Markov model and text mining technology, improves the accuracy of word segmentation results, and thus accurately extracts target keyword groups, further improving the accuracy of asset retrieval results. Another object of the present invention is to provide a keyword group extraction device for cyberspace asset retrieval. Another object of the present invention is to provide a computer-readable medium. Another object of the present invention is to provide a computer device.

[0004] In order to achieve the above objectives, the present invention discloses a keyword group extraction method for cyberspace asset retrieval, comprising:

[0005] Acquire multiple target text terms of a target site for cyberspace asset retrieval;

[0006] Through the pre-trained improved hidden Markov model, the target text entries are segmented to obtain the target segmentation results corresponding to each target text entry;

[0007] Combine the target word segmentation results to generate multiple target phrases;

[0008] Through text mining technology, multiple target phrases are mined and counted based on multiple target text entries to generate target keyword phrases.

[0009] Preferably, obtaining a plurality of target text entries of a target site for network space asset retrieval includes:

[0010] Use asset search engine tools to search the target site and generate full text entries;

[0011] According to the preset target conditions, the full text entries are extracted to generate target text entries.

[0012] Preferably, the method further comprises:

[0013] Obtaining a term sample training set and a term sample test set, wherein the term sample training set includes a term state value set and a term observation value set;

[0014] The initial improved hidden Markov algorithm is trained by using the entry state value set and the entry observation value set to generate an initial improved hidden Markov model;

[0015] Through the model optimization algorithm, the model parameters of the initial improved hidden Markov model are optimized according to the term sample test set to generate the final improved hidden Markov model.

[0016] Preferably, the target word segmentation results are combined to generate multiple target phrases, including:

[0017] Combine all the target word segmentation results to generate full-quantity phrases;

[0018] The full phrase is determined as the target phrase.

[0019] Preferably, the target word segmentation results are combined to generate multiple target phrases, including:

[0020] According to the preset number of keywords, the target word segmentation results are combined to generate multiple target phrases.

[0021] Preferably, by using text mining technology, multiple target phrases are mined and counted based on multiple target text entries to generate target keyword phrases, including:

[0022] Through text mining technology, multiple target phrases are accurately matched and fuzzy matched according to multiple target text entries, and the importance index corresponding to the target phrases is generated;

[0023] According to the importance index, a target keyword phrase is determined from multiple target phrases.

[0024] Preferably, by using text mining technology, multiple target phrases are precisely matched and fuzzy matched according to multiple target text entries to generate importance indicators corresponding to the target phrases, including:

[0025] Based on multiple target text entries, multiple target phrases are accurately matched to generate word frequency indicators;

[0026] Based on multiple target text terms, multiple target phrases are matched accurately and fuzzily to generate an inverse document frequency index;

[0027] According to the word frequency index and the inverse document frequency index, the importance index corresponding to the target phrase is generated.

[0028] The present invention also discloses a keyword group extraction device for network space asset retrieval, comprising:

[0029] An entry acquisition unit, used for acquiring a plurality of target text entries of a target site for cyberspace asset retrieval;

[0030] A word segmentation unit is used to segment target text terms through a pre-trained improved hidden Markov model to obtain a target word segmentation result corresponding to each target text term;

[0031] A combining unit, used for combining the target word segmentation results to generate multiple target phrases;

[0032] The phrase extraction unit is used to mine and count multiple target phrases based on multiple target text entries through text mining technology to generate target keyword phrases.

[0033] Preferably, the entry acquisition unit is specifically used to search the target site through an asset search engine tool to generate full text entries; according to preset target conditions, the full text entries are extracted to generate target text entries.

[0034] Preferably, the device further comprises:

[0035] A sample acquisition unit, used to acquire a term sample training set and a term sample test set, wherein the term sample training set includes a term state value set and a term observation value set;

[0036] A model training unit, used for training the initial improved hidden Markov algorithm through the entry state value set and the entry observation value set to generate an initial improved hidden Markov model;

[0037] The model optimization unit is used to optimize the model parameters of the initial improved hidden Markov model according to the term sample test set through the model optimization algorithm to generate the final improved hidden Markov model.

[0038] Preferably, the combination unit is specifically used to perform full combination on the target word segmentation results to generate a full phrase; and determine the full phrase as the target phrase.

[0039] Preferably, the combining unit is specifically used to combine the target word segmentation results according to a preset number of keywords to generate a plurality of target phrases.

[0040] Preferably, the phrase extraction unit is specifically used to perform exact matching and fuzzy matching on multiple target phrases based on multiple target text entries through text mining technology to generate importance indicators corresponding to the target phrases; and determine the target keyword phrase from the multiple target phrases based on the importance indicators.

[0041] Preferably, the phrase extraction unit is specifically used to perform precise matching of multiple target phrases based on multiple target text entries to generate a word frequency index; perform precise matching and fuzzy matching of multiple target phrases based on multiple target text entries to generate an inverse document frequency index; and generate an importance index corresponding to the target phrase based on the word frequency index and the inverse document frequency index.

[0042] The present invention also discloses a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0043] The present invention also discloses a computer device, including a memory and a processor, wherein the memory is used to store information including program instructions, the processor is used to control the execution of program instructions, and the processor implements the above method when executing the program.

[0044] The present invention also discloses a computer program product, including a computer program / instruction, and the method described above is implemented when the computer program / instruction is executed by a processor.

[0045] The present invention obtains multiple target text entries of a target site for cyberspace asset retrieval; performs word segmentation on the target text entries through a pre-trained improved hidden Markov model to obtain a target word segmentation result corresponding to each target text entry; combines the target word segmentation results to generate multiple target phrases; performs mining statistics on multiple target phrases according to multiple target text entries through text mining technology to generate target keyword phrases, extracts keyword phrases based on the improved Markov model and text mining technology, improves the accuracy of word segmentation results, thereby accurately extracting target keyword phrases, and further improves the accuracy of asset retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a keyword group extraction method for network space asset retrieval provided by an embodiment of the present invention;

[0048] Figure 2 A flowchart of another method for extracting keyword phrases for network space asset retrieval provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a keyword extraction device for network space asset retrieval provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] It should be noted that the keyword group extraction method and device for cyberspace asset retrieval disclosed in the present application can be used in the field of artificial intelligence technology, and can also be used in any field outside the field of artificial intelligence technology. The application field of the keyword group extraction method and device for cyberspace asset retrieval disclosed in the present application is not limited.

[0053] In order to facilitate the understanding of the technical solution provided by this application, the relevant contents of the technical solution of this application are first explained below. With the continuous expansion and complexity of cyberspace, cyberspace asset retrieval technology plays an increasingly important role in network security, asset management, business analysis and other fields. A common asset detection method is to enter keywords in a network asset retrieval engine for fuzzy matching. For example, by entering keywords such as "source, network, load and storage" in the asset search engine tool to match the assets of the power website, assets include but are not limited to Internet Protocol (IP) addresses, domain names, servers, applications, databases and cloud services.

[0054] The structure of the traditional hidden Markov model is as follows:

[0055] There are two independence assumptions: 1. The observations are independent of each other. 2. The state value is only related to the previous state value and has nothing to do with other state values. The initial state matrix, state transfer matrix and observation matrix of the learning model are learned using maximum likelihood estimation through training sample data; according to the given observation value sequence, the state value sequence is calculated. For the word segmentation system, the state is a sequence of B (beginning of the word), M (middle of the word), E (end of the word), S (independent word), and the observation value is a Chinese character. The application of the hidden Markov model in the word segmentation system is manifested in learning the initial state matrix, state transfer matrix and observation matrix through training samples; according to the given Chinese sentence, the sequence composed of BMES is calculated to realize Chinese word segmentation. Its probability distribution calculation formula can be expressed as follows:

[0056] Given the observed value, the conditional probability distribution P(state|observation) of the state value can be expressed as follows according to the Bayesian formula:

[0057] P(state|observation)=P(observation|state)×P(state) / P(observation). Since P(observation) is fixed for all states, P(state|observation)≈P(observation|state)×P(state).

[0058] According to the requirements of independence assumption 1, the observations are independent of each other, so the formula can be expressed as:

[0059] P(state|observation)≈P(observation|state)×P(state)=(P(observation1|state)×P(state))×(P(observation2|state)×P(state))….

[0060] According to the requirements of independence assumption 2, the state is only related to the state at the previous moment, so P(state) in the formula can be further expressed as:

[0061] P(state t)=P(state t|state t-1).

[0062] According to Bayes' theorem, the traditional hidden Markov algorithm formula is expanded:

[0063]

[0064] Among them, P(Y|X) is the probability of generating the term state value set Y from the term observation value set X; P(X|Y) is the probability of generating the term observation value set X from the term state value set Y; P(Y) is the probability of the term state value set Y; P(X) is the probability of the term observation value set X. Among them, the maximum value of P(Y|X) is recorded as maxP(Y|X), maxP(Y|X) = P(X|Y) × P(Y), and maxP(Y|X) is the word segmentation result.

[0065] According to Hidden Markov Independence Assumption 1: The observations are independent of each other, the above formula is written as:

[0066]

[0067] Among them, maxP(Y|X) is the word segmentation result; P(X i |Y) is to generate an observation value X from the entry state value set Y i The probability of; P(Y) is the probability of the entry state value set Y.

[0068] According to Hidden Markov Independence Assumption 2: the state value at this moment is only related to the state value at the previous moment, the above formula is written as:

[0069]

[0070] Among them, maxP(Y|X) is the word segmentation result; P(X i |Y i ) is a state Y i Generate an observation X i The probability of i |Y i-1 ) is a state Y i-1 Transition to another state Y i The probability of. Calculate P(X i |Y i ) and P(Y i |Y i-1 ), find the state chain with the largest P(Y|X), that is, the most likely word segmentation result maxP(Y|X).

[0071] The following takes the keyword group extraction device for cyberspace asset retrieval as an example of the execution subject to illustrate the implementation process of the keyword group extraction method for cyberspace asset retrieval provided by the embodiment of the present invention. It can be understood that the execution subject of the keyword group extraction method for cyberspace asset retrieval provided by the embodiment of the present invention includes but is not limited to the keyword group extraction device for cyberspace asset retrieval.

[0072] Figure 1 A flowchart of a keyword group extraction method for network space asset retrieval provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0073] Step 101: Acquire multiple target text entries of a target site for cyberspace asset retrieval.

[0074] Step 102: Segment the target text terms using the pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text term.

[0075] Step 103: Combine the target word segmentation results to generate multiple target phrases.

[0076] Step 104: Using text mining technology, based on multiple target text entries, multiple target phrases are mined and counted to generate target keyword phrases.

[0077] In the technical solution provided by the embodiment of the present invention, multiple target text entries of a target site for cyberspace asset retrieval are obtained; the target text entries are segmented by a pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text entry; the target segmentation results are combined to generate multiple target phrases; through text mining technology, multiple target phrases are mined and counted according to multiple target text entries to generate target keyword phrases, and keyword phrases are extracted based on the improved Markov model and text mining technology to improve the accuracy of the segmentation results, thereby accurately extracting the target keyword phrases, and further improving the accuracy of the asset retrieval results.

[0078] Figure 2 A flowchart of another method for extracting keyword phrases for network space asset retrieval provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0079] Step 201: Search the target site through the asset search engine tool to generate full text entries.

[0080] In the embodiment of the present invention, each step is performed by a keyword group extraction device for network space asset retrieval.

[0081] In the embodiment of the present invention, the target site is a website that needs to perform asset retrieval, including but not limited to the main site and sub-sites of the website. Specifically, through the asset search engine tool, the main site and sub-sites that need to perform asset retrieval are fully searched to generate full text entries, which include all sentence entries of the main site and sub-sites.

[0082] Step 202: extract all text entries according to preset target conditions to generate target text entries.

[0083] In the embodiment of the present invention, the target condition can be set according to actual needs, and the embodiment of the present invention does not limit this. As an optional solution, the target condition can be the target asset type to be retrieved.

[0084] Specifically, according to the target condition, text terms that meet the target condition are extracted from the full amount of text terms; and the extracted text terms are determined as target text terms.

[0085] In the embodiment of the present invention, extracting text terms that meet the target conditions can reduce the amount of subsequent model calculations, save computing resources, and improve computing efficiency.

[0086] As another optional solution, determining the full text entry as the target text entry can ensure the comprehensiveness and accuracy of subsequent calculation results.

[0087] Step 203: training an improved hidden Markov model.

[0088] In the embodiment of the present invention, the hidden Markov model includes a state value set, an observation value set, a state initial matrix, a state transition matrix, and an observation matrix. The state value is a hidden state, denoted by Y; the observation value is an observable result, which is closely related to the state value, denoted by X; the state initial matrix is ​​the probability of various states at the initial time, denoted by π; the state transition matrix is ​​the probability of a state Y from a state i-1 Transition to another state Y i The probability of i |Y i-1 ). The observation matrix consists of a state Y i Generate observations X i The probability P(X i |Y i ).

[0089] In the hidden Markov model for Chinese word segmentation, all Chinese characters constitute an observation set, and the position information of Chinese characters in the word (B (beginning of the word), E (end of the word), M (in the middle of the word), S (independent word)) constitutes an observation set. By training samples and using the maximum likelihood estimation method to approximate the frequency as probability, the initial state matrix, state transfer matrix and observation matrix of the training samples can be obtained, and finally the learning of the hidden Markov model can be completed.

[0090] In the embodiment of the present invention, step 203 specifically includes:

[0091] Step 2031: Obtain a term sample training set and a term sample test set, wherein the term sample training set includes a term state value set and a term observation value set.

[0092] In an embodiment of the present invention, a term sample training set and a term sample test set are pre-constructed. The term sample training set includes a term state value set and a term observation value set used in the training process; the term sample test set includes a term state value set and a term observation value set used in the testing process.

[0093] Step 2032: Perform model training on the initial improved hidden Markov algorithm using the entry state value set and the entry observation value set to generate an initial improved hidden Markov model.

[0094] In the embodiments of the present invention, the independence assumption 1 of the traditional Hidden Markov Model (HMM) holds that the observed values are independent of each other. However, in reality, not all observed values are independent. Some words are strongly correlated. For example, for the word "taotie", if "tao" appears, it is very likely that "tie" will follow. Therefore, it is necessary to improve the formula of the independence assumption 1 of the traditional HMM. First, use the HMM algorithm to segment the text; then, perform word frequency statistics on the segmented results to distinguish whether two characters are independent. For example, for a phrase MN composed of Chinese characters M and N, the co-occurrence frequency of MN is: where P(MN) is the co-occurrence frequency of the phrase MN; count(MI N) is the number of sentences in the sample that contain both M and N; count(MYN) is the number of sentences in the sample that contain M or N.

[0095] When P(MN) < 0.5, it indicates that Chinese characters M and N are independent, and then:

[0096] P(MN) = P(M)P(N)

[0097] where P(MN) is the co-occurrence frequency of the phrase MN; P(M) is the probability of the appearance of Chinese character M; P(N) is the probability of the appearance of Chinese character N.

[0098] When P(MN) > 0.5, it indicates that Chinese characters M and N are not independent, and then:

[0099] P(MN) = P(M)P(N) + μ(1 - P(M)P(N))

[0100] where P(MN) is the co-occurrence frequency of the phrase MN; P(M) is the probability of the appearance of Chinese character M; P(N) is the probability of the appearance of Chinese character N; μ is the difference measurement parameter. The value of μ can be obtained through learning from the training samples, and the value range of μ is greater than 0 and less than 1.

[0101] In summary, the formula of the independence assumption 1 of the improved HMM is:

[0102]

[0103] where maxP(Y|X) is the segmentation result; P(X i |Y) is the probability of generating an observed value X from the set of word entry state values Y i ; P(X j |Y) is the probability of generating an observed value X from the set of word entry state values Y j ; P(X k |Y) is the probability of generating an observed value X from the set of word entry state values Y k ; μ is the difference measurement parameter; P(Y) is the probability of the set of word entry state values Y.

[0104] The independence assumption 2 of the traditional hidden Markov model believes that the state value at this moment is only related to the state value at the previous moment. In fact, the state value at this moment (expressed as word division in the word segmentation problem) is not only related to the state value of the previous word, but may also be related to a previous state value. For example, in the sentence: "Although xxxx but xxxx", "but" is divided into a state phrase precisely because the state division "although" appears in front. Therefore, it is necessary to improve the formula of the independence assumption 2 of the traditional hidden Markov model. First, the hidden Markov algorithm is used to segment the text; then the frequency of associated words is counted for the results of the segmentation to measure the cross-word correlation. For example, if word M' and word N' are associated, the frequency P(M'N') of word M' and word N' being associated words is: Wherein, count(M'I N') indicates the number of sentences in the sample that contain both word M' and word N', and count(M'YN') indicates the number of sentences in the sample that contain word M' or word N'.

[0105] When P(M'N') < 0.5, it indicates that the word M' and the word N' are not related, then:

[0106] P(Y M' |Y M'-1 ,Y M'-2 ...Y1)=P(Y M' |Y M'-1 )

[0107] Among them, P(Y M' |Y M'-1 ) indicates that the probability of the word M' state occurring is only related to the probability of the previous state, P(Y M' |Y M'-1 ,Y M'-2 ...Y1) indicates that Y is known M'-1 ,Y M'-2 ...the probability of the M' state of the word Y1 occurring.

[0108] When P(MN)>0.5, it indicates that word M' and word N' are related, then:

[0109] P(Y M' |Y M'-1 ,Y i-2 ...Y1)=P(Y M' |Y M'-1 ,Y N' )

[0110] Among them, P(Y M' |Y M'-1 ,Y i-2 ...Y1) is the known YM'-1 ,Y i-2 ...the conditional probability of the occurrence of the word M' state of Y1, P(Y M' |Y M'-1 ,Y N' ) indicates that the probability of word M' occurring is only related to the previous state and the state of word N'.

[0111] In summary, the formula for independence assumption 2 of the improved hidden Markov model is:

[0112]

[0113] Among them, maxP(Y|X) is the word segmentation result; P(X i |Y i ) is a state Y i Generate an observation X i The probability of i |Y i-1 ,Y t ) is the known Y i-1 ,Y t Status Y i Probability of occurrence; Y t Representation and state Y i The associated state.

[0114] Combining the formula of the improved hidden Markov model independence assumption 1 with the formula of the improved hidden Markov model independence assumption 2, the expression of the improved hidden Markov algorithm is obtained as follows:

[0115]

[0116] Among them, maxP(Y|X) is the word segmentation result; P(X i |Y) is to generate an observation value X from the entry state value set Y i The probability of P(X j |Y) is to generate an observation value X from the entry state value set Y j The probability of P(X k |Y) is to generate an observation value X from the entry state value set Y k The probability of; μ is the difference measurement parameter; P(Y i |Y i-1 ,Y t ) is the known Y i-1 ,Y t Status Y i Probability of occurrence; Y t Representation and state Y i The associated state.

[0117] Specifically, the entry state value set and the entry observation value set used in the training process are input into the initial improved hidden Markov algorithm for preliminary model training to generate an initial improved hidden Markov model.

[0118] Step 2033: Optimize the model parameters of the initial improved hidden Markov model according to the entry sample test set through the model optimization algorithm to generate the final improved hidden Markov model.

[0119] In the embodiment of the present invention, the model optimization algorithm is a gradient descent algorithm. Specifically, the entry sample test set used in the test process is input into the initial improved hidden Markov model, and the model parameters are optimized by the gradient descent algorithm to generate the final improved hidden Markov model.

[0120] Step 204: Segment the target text terms using the pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text term.

[0121] Specifically, each target text entry is input into the improved hidden Markov model for word segmentation calculation to generate a target word segmentation result corresponding to each target text entry.

[0122] Step 205: Combine the target word segmentation results to generate multiple target phrases.

[0123] In the embodiment of the present invention, step 205 specifically includes:

[0124] Step 2051: Fully combine the target word segmentation results to generate a full phrase.

[0125] Specifically, the target word segmentation results corresponding to the same target text term are fully combined to generate a full phrase corresponding to the target text term.

[0126] Step 2052: determine the full phrase as the target phrase.

[0127] In the embodiment of the present invention, the full phrase is determined as the target phrase corresponding to the target text entry, which can ensure the comprehensiveness and accuracy of subsequent calculation results.

[0128] As another optional solution, the target word segmentation results are combined according to a preset number of keywords to generate multiple target phrases.

[0129] It is worth noting that the number of keywords can be set according to actual needs, and the embodiment of the present invention does not limit this. As an optional solution, if the number of keywords is 3, then the target word segmentation results are combined in groups of three using the three keywords to generate multiple target phrases, each of which includes 3 target word segments.

[0130] In the embodiment of the present invention, generating target phrases according to the number of keywords can reduce the amount of combination calculations, save computing resources, and improve computing efficiency.

[0131] Step 206: Using text mining technology, perform exact matching and fuzzy matching on multiple target phrases based on multiple target text entries to generate importance indicators corresponding to the target phrases.

[0132] In the embodiment of the present invention, step 206 specifically includes:

[0133] Step 2061: Based on the multiple target text entries, multiple target phrases are accurately matched to generate a word frequency index (TF).

[0134] In the embodiment of the present invention, exact matching means that the current target text entry contains all target word segments of the target phrase.

[0135] Specifically, each target phrase is accurately matched among multiple target text entries, and the number of target text entries containing all target word segments of the target phrase is counted; the counted number is determined as the number of accurately matched hit entries; the number of accurately matched hit entries is divided by the total number of accurately matched target text entries to generate a term frequency index (TF), that is: TF = A1 / A, where TF is the term frequency index, A1 is the number of accurately matched hit entries, and A is the total number of accurately matched target text entries.

[0136] Step 2062: Based on the multiple target text terms, perform exact matching and fuzzy matching on the multiple target phrases to generate an inverse document frequency (IDF).

[0137] In the embodiment of the present invention, fuzzy matching means that the current target text entry contains at least one target word segment of the target phrase.

[0138] Specifically, the number of target text terms that contain at least one target word segment of the target phrase is counted; the counted number is determined as the number of fuzzy matching hit terms; the number of fuzzy matching hit terms and the number of exact matching hit terms are calculated by IDF=log(A2 / 1+A1) to generate an inverse document frequency index (IDF). Among them, IDF is an inverse document frequency index, A1 is the number of exact matching hit terms, and A2 is the number of fuzzy matching hit terms.

[0139] Step 2063: Generate an importance index corresponding to the target phrase based on the word frequency index and the inverse document frequency index.

[0140] Specifically, the term frequency index (TF) and the inverse document frequency index (IDF) are multiplied to generate the importance index (TF-IDF) corresponding to the target phrase, that is, TF-IDF=TF×IDF.

[0141] In the embodiment of the present invention, the higher the importance index, the more accurate the target segmentation in the target phrase is for the asset retrieval application, and the better the effect of using the target segmentation in the target phrase as a keyword in the retrieval; the lower the importance index, the less accurate the target segmentation in the target phrase is for the asset retrieval application, and the worse the effect of using the target segmentation in the target phrase as a keyword in the retrieval.

[0142] Step 207: Determine a target keyword phrase from multiple target phrases based on the importance index.

[0143] In the embodiment of the present invention, a plurality of target phrases are arranged in descending order according to their corresponding importance indexes, and the first k groups of target phrases are obtained and determined as target keyword phrases.

[0144] It is worth noting that the value of k can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0145] In the embodiment of the present invention, the first k groups of target phrases with higher importance indexes are determined as target keyword groups for asset retrieval, and subsequently, accurate search based on the target keywords can be completed in the network asset search engine.

[0146] The present invention completes the selection of data sets and the calculation of TF-IDF through fuzzy matching and exact matching of search engines. When searching for network assets, the combination of multiple keywords makes it easier to detect the target entry, thereby improving the accuracy of determining the target keyword.

[0147] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of laws and regulations. The user information in the embodiments of this application is obtained through legal and compliant channels, and the acquisition, storage, use, and processing of user information are authorized and agreed by the customer.

[0148] It is worth noting that the information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0149] It is worth noting that the technical solution provided in this application provides users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0150] In the technical solution of the keyword phrase extraction method for cyberspace asset retrieval provided by the embodiment of the present invention, multiple target text entries of the target site for cyberspace asset retrieval are obtained; the target text entries are segmented through a pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text entry; the target segmentation results are combined to generate multiple target phrases; through text mining technology, based on the multiple target text entries, multiple target phrases are mined and counted to generate target keyword phrases, and keyword phrases are extracted based on the improved Markov model and text mining technology to improve the accuracy of the segmentation results, thereby accurately extracting the target keyword phrases, and further improving the accuracy of the asset retrieval results.

[0151] Figure 3 A schematic diagram of the structure of a keyword group extraction device for network space asset retrieval provided by an embodiment of the present invention, the device is used to execute the keyword group extraction method for network space asset retrieval, such as Figure 3 As shown, the device includes: a term acquisition unit 11, a word segmentation unit 12, a combination unit 13 and a phrase extraction unit 14.

[0152] The term acquisition unit 11 is used to acquire a plurality of target text terms of a target site for network space asset retrieval.

[0153] The word segmentation unit 12 is used to segment the target text terms through a pre-trained improved hidden Markov model to obtain a target word segmentation result corresponding to each target text term.

[0154] The combining unit 13 is used to combine the target word segmentation results to generate multiple target phrases.

[0155] The phrase extraction unit 14 is used to perform mining statistics on multiple target phrases based on multiple target text entries through text mining technology to generate target keyword phrases.

[0156] In the embodiment of the present invention, the term acquisition unit 11 is specifically used to search the target site through the asset search engine tool to generate full text terms; according to the preset target conditions, the full text terms are extracted to generate target text terms.

[0157] In the embodiment of the present invention, the device further includes: a sample acquisition unit 15 , a model training unit 16 and a model optimization unit 17 .

[0158] The sample acquisition unit 15 is used to acquire a term sample training set and a term sample test set. The term sample training set includes a term state value set and a term observation value set.

[0159] The model training unit 16 is used to perform model training on the initial improved hidden Markov algorithm through the entry state value set and the entry observation value set to generate an initial improved hidden Markov model.

[0160] The model optimization unit 17 is used to optimize the model parameters of the initial improved hidden Markov model according to the entry sample test set through the model optimization algorithm to generate the final improved hidden Markov model.

[0161] In the embodiment of the present invention, the combining unit 13 is specifically used to perform full combination on the target word segmentation results to generate a full phrase; and determine the full phrase as the target phrase.

[0162] In the embodiment of the present invention, the combining unit 13 is specifically configured to combine the target word segmentation results according to a preset number of keywords to generate a plurality of target phrases.

[0163] In the embodiment of the present invention, the phrase extraction unit 14 is specifically used to perform precise matching and fuzzy matching on multiple target phrases based on multiple target text entries through text mining technology to generate importance indicators corresponding to the target phrases; and determine the target keyword phrase from the multiple target phrases based on the importance indicators.

[0164] In the embodiment of the present invention, the phrase extraction unit 14 is specifically used to perform precise matching of multiple target phrases based on multiple target text entries to generate a word frequency index; perform precise matching and fuzzy matching of multiple target phrases based on multiple target text entries to generate an inverse document frequency index; and generate an importance index corresponding to the target phrase based on the word frequency index and the inverse document frequency index.

[0165] In the scheme of the embodiment of the present invention, multiple target text entries of a target site for cyberspace asset retrieval are obtained; the target text entries are segmented by a pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text entry; the target segmentation results are combined to generate multiple target phrases; through text mining technology, multiple target phrases are mined and counted according to multiple target text entries to generate target keyword phrases, and keyword phrases are extracted based on the improved Markov model and text mining technology to improve the accuracy of the segmentation results, thereby accurately extracting the target keyword phrases, and further improving the accuracy of the asset retrieval results.

[0166] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device, and specifically, the computer device may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] An embodiment of the present invention provides a computer device, including a memory and a processor, the memory is used to store information including program instructions, the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the steps of the embodiment of the above-mentioned keyword group extraction method for cyberspace asset retrieval are implemented. For a specific description, please refer to the embodiment of the above-mentioned keyword group extraction method for cyberspace asset retrieval.

[0168] Reference below Figure 4 , which shows a schematic diagram of the structure of a computer device 600 suitable for implementing an embodiment of the present application.

[0169] like Figure 4 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate operations and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0170] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal feedback device (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed as needed as the storage section 608.

[0171] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from the removable medium 611.

[0172] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0173] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0174] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0178] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0179] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0181] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0182] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A keyword extraction method for cyberspace asset retrieval, characterized in that: The method comprises: Acquire multiple target text terms of a target site for cyberspace asset retrieval; Segmenting the target text terms through a pre-trained improved hidden Markov model to obtain a target segmentation result corresponding to each target text term; Combining the target word segmentation results to generate multiple target phrases; Through text mining technology, multiple target phrases are mined and counted based on multiple target text entries to generate target keyword phrases.

2. The keyword extraction method for cyberspace asset retrieval according to claim 1 is characterized in that: The step of obtaining a plurality of target text entries of a target site for network space asset retrieval includes: Using an asset search engine tool, search the target site to generate full text entries; According to preset target conditions, the full text entries are extracted to generate target text entries.

3. The keyword extraction method for cyberspace asset retrieval according to claim 1 is characterized in that: The method further comprises: Obtaining a term sample training set and a term sample test set, wherein the term sample training set includes a term state value set and a term observation value set; Performing model training on the initial improved hidden Markov algorithm through the entry state value set and the entry observation value set to generate an initial improved hidden Markov model; The model parameters of the initial improved hidden Markov model are optimized according to the entry sample test set by using a model optimization algorithm to generate the final improved hidden Markov model.

4. The keyword extraction method for cyberspace asset retrieval according to claim 1 is characterized in that: The target word segmentation results are combined to generate multiple target phrases, including: Performing full combination on the target word segmentation results to generate full-quantity phrases; The full phrase is determined as the target phrase.

5. The keyword extraction method for cyberspace asset retrieval according to claim 1 is characterized in that: The target word segmentation results are combined to generate multiple target phrases, including: According to the preset number of keywords, the target word segmentation results are combined to generate multiple target phrases.

6. The keyword extraction method for cyberspace asset retrieval according to claim 1 is characterized in that: The text mining technology is used to mine and count multiple target phrases according to multiple target text entries to generate target keyword phrases, including: Using text mining technology, based on multiple target text entries, multiple target phrases are accurately matched and fuzzy matched to generate importance indicators corresponding to the target phrases; The target keyword phrase is determined from a plurality of target phrases according to the importance index.

7. The keyword extraction method for cyberspace asset retrieval according to claim 6 is characterized in that: The text mining technology is used to perform precise matching and fuzzy matching on multiple target phrases according to multiple target text entries, and generate importance indicators corresponding to the target phrases, including: Based on multiple target text entries, multiple target phrases are accurately matched to generate word frequency indicators; Based on multiple target text terms, multiple target phrases are matched accurately and fuzzily to generate an inverse document frequency index; An importance index corresponding to the target phrase is generated according to the word frequency index and the inverse document frequency index.

8. A keyword extraction device for network space asset retrieval, characterized in that: The device comprises: An entry acquisition unit, used for acquiring a plurality of target text entries of a target site for cyberspace asset retrieval; A word segmentation unit is used to segment the target text terms through a pre-trained improved hidden Markov model to obtain a target word segmentation result corresponding to each target text term; A combining unit, used for combining the target word segmentation results to generate a plurality of target phrases; The phrase extraction unit is used to mine and count multiple target phrases based on multiple target text entries through text mining technology to generate target keyword phrases.

9. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The entry acquisition unit is specifically used to search the target site through an asset search engine tool to generate full text entries; according to preset target conditions, the full text entries are extracted to generate target text entries.

10. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The device also includes: A sample acquisition unit, used to acquire a term sample training set and a term sample test set, wherein the term sample training set includes a term state value set and a term observation value set; A model training unit, used to perform model training on an initial improved hidden Markov algorithm through the entry state value set and the entry observation value set to generate an initial improved hidden Markov model; The model optimization unit is used to optimize the model parameters of the initial improved hidden Markov model according to the term sample test set through the model optimization algorithm to generate the final improved hidden Markov model.

11. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The combination unit is specifically used to perform full combination on the target word segmentation results to generate a full phrase; and determine the full phrase as the target phrase.

12. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The combining unit is specifically used to combine the target word segmentation results according to a preset number of keywords to generate a plurality of target phrases.

13. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The phrase extraction unit is specifically used to perform precise matching and fuzzy matching on multiple target phrases according to multiple target text entries through text mining technology to generate importance indicators corresponding to the target phrases; and determine the target keyword phrase from the multiple target phrases according to the importance indicators.

14. The keyword extraction device for cyberspace asset retrieval according to claim 8, characterized in that: The phrase extraction unit is specifically used to perform precise matching on multiple target phrases according to multiple target text entries to generate a word frequency index; perform precise matching and fuzzy matching on multiple target phrases according to multiple target text entries to generate an inverse document frequency index; and generate an importance index corresponding to the target phrase according to the word frequency index and the inverse document frequency index.

15. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for extracting keyword phrases for cyberspace asset retrieval as described in any one of claims 1 to 7 is implemented.

16. A computer device comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, characterized in that: When the program instructions are loaded and executed by the processor, the keyword group extraction method for cyberspace asset retrieval according to any one of claims 1 to 7 is implemented.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the keyword group extraction method for cyberspace asset retrieval according to any one of claims 1 to 7 is implemented.

Images