An evaluation method and system based on semantic similarity analysis

Abstract

The application provides a kind of based on semantic similarity analysis evaluation method and system, wherein the method comprises obtaining the investigation form of the environment and system information of evaluation system;Extract the first description sentence related to test item in investigation form;Determine the subject corresponding to the first description sentence;Obtain the standard description sentence of subject;The first description sentence corresponding to the subject and the standard description sentence of subject are analyzed by similarity, and the test item result is obtained;According to the content of test item result and investigation form, generate evaluation report.The application uses historical data to learn and train, realizes automatic evaluation model to improve the effect and quality of compliance evaluation business such as security protection, password, solves the problem that service quality and efficiency are seriously dependent on the ability of individual evaluator, and cannot support the fast expansion of business.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network security, and particularly relates to a method and system for evaluation based on semantic similarity analysis. BACKGROUND

[0002] The evaluation of the equivalent protection is the information security level protection, which is to implement the security protection in different levels for the important information of the state, the exclusive information of the legal person and other organizations and the public, and the information system (network equipment) for storing, transmitting and processing the information, to implement the management according to the level for the information security products used in the information system, and to respond and dispose the information security events in the information system in different levels.

[0003] The equivalent evaluation refers to the evaluation work of the network equipment according to the technical data in the information system security level protection evaluation requirements. The traditional evaluation work of the network equipment is to distinguish the type of the network equipment manually, to perform the corresponding evaluation on the network equipment, and to manually calculate the data obtained by the evaluation to obtain the evaluation result. The traditional evaluation process has great limitations. When a large number of network equipments need to be evaluated automatically, the evaluation result will have errors, and the evaluation time is too long, and the whole evaluation time is not efficient. SUMMARY

[0004] The present application provides a method and system for evaluation based on semantic similarity analysis to solve the problems in the prior art.

[0005] In the first aspect, the present application provides a method for evaluation based on semantic similarity analysis, comprising:

[0006] Obtaining a survey table of the environment and system information of the evaluation system;

[0007] Extracting a first description sentence related to the test item in the survey table;

[0008] Determining the theme corresponding to the first description sentence;

[0009] Obtaining the standard description sentence of the theme;

[0010] Performing similarity analysis on the first description sentence corresponding to the theme and the standard description sentence of the theme to obtain the test item result;

[0011] Generating an evaluation report according to the test item result and the content of the survey table.

[0012] Further, the determination of the theme corresponding to the first description sentence comprises:

[0013] The LDA model algorithm is used to analyze the topic to which the first description sentence belongs, and compared with the full topic of the evaluation item in the database. If it does not belong to the topic related to the evaluation item, the target first description sentence is deleted, and records are made;

[0014] For the explicit first description sentence, the implicit topic of the first description sentence is extracted using the LDA model, and is associated with the evaluation item.

[0015] Further, the LDA model algorithm is used to analyze the topic to which the first description sentence belongs, including:

[0016] Before training the LDA model, a target number of topics are obtained;

[0017] The text data set is subjected to word segmentation and stop word filtering operations;

[0018] The LDA model is trained using historical data to generate document topic distribution and word topic distribution;

[0019] The trained LDA model is subjected to denoising and model parameter adjustment and topic prediction.

[0020] Further, the first description sentence corresponding to the topic and the standard description sentence of the topic are subjected to similarity analysis to obtain the test item result, including:

[0021] The first description sentence is subjected to word parsing using the BM25 algorithm to generate word q i ;

[0022] The relevance score of word q i to the first description sentence is determined;

[0023] The relevance score of word q i to the first description sentence is weighted and summed to obtain the relevance of the first description sentence to the standard description sentence.

[0024] Further, the relevance score of word q i to the first description sentence is weighted and summed to obtain the relevance of the first description sentence to the standard description sentence, including:

[0025] The relevance Score(Q,d) of the first description sentence to the standard description sentence is calculated according to the following formula:

[0026]

[0027] Wherein, Q is the first description sentence; d is the standard description sentence; n is the number of words in the first description sentence; q i The i-th word after parsing the first description sentence; IDF(q i ) is the word qi The weight of; N is the number of standard description sentences in the index; n(q i ) contains word q i The number of standard description statements; k1 is the first adjustment factor; b is the second adjustment factor; f i For word q i The frequency of occurrence in standard description sentences; dl is the length of the standard description sentence; avgdl is the average length of all standard description sentences.

[0028] In a second aspect, the present invention provides an evaluation system based on semantic similarity analysis, comprising:

[0029] The first acquisition module is used to obtain a survey form for the environment and system information of the evaluation system;

[0030] A first extraction module is used to extract a first description sentence related to the test item in the survey form;

[0031] a determination module, configured to determine a topic corresponding to the first description statement;

[0032] The second acquisition module is used to obtain a standard description sentence of the subject;

[0033] A similarity analysis module is used to perform similarity analysis on the first description sentence corresponding to the topic and the standard description sentence of the topic to obtain a test item result;

[0034] The evaluation report generation module is used to generate an evaluation report based on the test item results and the content of the survey form.

[0035] Furthermore, the determining module includes:

[0036] An analysis unit is configured to analyze the topic to which the first description statement belongs using an LDA model algorithm, and compare the topic with all topics of the evaluation items in the database. If the topic does not belong to a topic related to the evaluation item, the target first description statement is deleted and recorded.

[0037] The extraction unit is used to extract the implicit topic of the first description sentence using the LDA model for the clear first description sentence, and associate it with the evaluation item.

[0038] Furthermore, the analysis unit includes:

[0039] An acquisition device, used to acquire a target number of topics before training the LDA model;

[0040] A text data set processing device, used for performing vocabulary segmentation and stop word filtering operations on the text data set;

[0041] A model training device for training the LDA model using historical data to generate document topic distribution and word topic distribution;

[0042] The model processing device is used to denoise and adjust the model parameters of the trained LDA model and perform topic prediction.

[0043] Furthermore, the similarity analysis module includes:

[0044] The word parsing unit is used to parse the first description sentence using the BM25 algorithm to generate word q i ;

[0045] Determination unit, used to determine word q i Relevance score to the first descriptive statement;

[0046] Weighted summation unit, used to sum word q i The relevance scores of the first description statement are weighted and summed to obtain the relevance of the first description statement with the standard description statement.

[0047] Furthermore, the weighted summation unit includes:

[0048] The calculation device is used to calculate the correlation Score(Q,d) between the first description statement and the standard description statement according to the following formula:

[0049]

[0050] Where Q is the first description sentence; d is the standard description sentence; n is the number of words in the first description sentence; q i The i-th word after parsing the first description sentence; IDF(q i ) is the word q i The weight of; N is the number of standard description sentences in the index; n(q i ) contains word q i The number of standard description statements; k1 is the first adjustment factor; b is the second adjustment factor; f i For word q i The frequency of occurrence in standard description sentences; dl is the length of the standard description sentence; avgdl is the average length of all standard description sentences.

[0051] The present invention provides an evaluation method and system based on semantic similarity analysis, wherein the method includes obtaining a survey form for the environment and system information of the evaluation system; extracting a first descriptive statement related to a test item in the survey form; determining a subject corresponding to the first descriptive statement; obtaining a standard descriptive statement for the subject; performing a similarity analysis between the first descriptive statement corresponding to the subject and the standard descriptive statement for the subject to obtain a test result; and generating an evaluation report based on the test result and the contents of the survey form. The present invention utilizes historical data for learning and training to implement an automated evaluation model to improve the effectiveness and quality of compliance evaluation services such as security and cryptography, and addresses the problem that service quality and efficiency are heavily dependent on the individual abilities of the evaluator and cannot support rapid business expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A flowchart of an evaluation method based on semantic similarity analysis provided by an embodiment of the present invention;

[0054] Figure 2 A diagram showing the process of establishing a subject library and a vocabulary library provided in an embodiment of the present invention;

[0055] Figure 3 A diagram of the LDA model modeling and training process provided by an embodiment of the present invention;

[0056] Figure 4 A structural diagram of an evaluation system based on semantic similarity analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] In the embodiment of the present application, first, the test target and the evaluation item are interpreted according to the evaluation implementation guide, each evaluation item corresponds to a topic, a topic baseline library is established, and the evaluation result and result explanation baseline library is established for each evaluation item. The analysis benchmark library is established according to the understanding of the topic in different environments, different equipment and different evaluators. The topic library and the word library constructed by the evaluation item are the basic premise for the implementation of the embodiment of the present application. The establishment process of the topic library and the word library is shown in Figure 2 .

[0059] In an embodiment, as shown in Figure 1 , the embodiment of the present application provides an evaluation method based on semantic similarity analysis, comprising:

[0060] Step 101, obtaining the investigation table of the environment and system information of the evaluation system.

[0061] Through the investigation of the environment and system information of the evaluation system, the system investigation table is prepared according to the investigation table template, and the investigation table is imported into the system.

[0062] Step 102, extracting the first description sentence related to the test item in the investigation table.

[0063] Step 103, determining the topic corresponding to the first description sentence.

[0064] According to the network security protection implementation guide, the evaluation item of the evaluation project historical document and other data, the feature words contained in the evaluation item are analyzed through machine learning, and the feature words in the standard description sentence of the evaluation item are combined. Each evaluation item is taken as a topic, and the feature words in the standard description sentence of the topic are used to construct the topic library and the word library.

[0065] After the construction of the topic library and the word library, the LDA model algorithm is used for topic similarity analysis. The LDA (Latent Dirichlet Allocation, Latent Dirichlet Allocation) model is a topic model, which has been applied in the fields of text processing and information retrieval. It can be used to reveal the potential topics in large-scale text corpus, and each document can be assigned to one or more topics, so as to realize the summarization of the document content and find the hidden topic structure. The basic idea of LDA model is to represent the text as the distribution of words, and each document can be described as a mixture of potential topics according to the probability distribution of word appearance.

[0066] In this step, first, the LDA model algorithm is used to analyze the topic to which the first description sentence belongs, and compared with the full topic of the evaluation item in the database. If it does not belong to the topic related to the evaluation item, the target first description sentence is deleted and recorded.

[0067] Among them, as Figure 3As shown, before training the LDA model, the target number of topics is obtained through multiple experiments. The text data set is subjected to word segmentation and stop word filtering operations. The historical data is used to train the LDA model to generate document topic distribution and word topic distribution. The trained LDA model is denoised and the model is adjusted and the topic is predicted.

[0068] Then, for the explicit first description statement, the implicit topic of the first description statement is extracted using the LDA model, and is associated with the evaluation item.

[0069] Step 104, obtaining the standard description statement of the topic.

[0070] Step 105, similarity analysis is performed on the first description statement corresponding to the topic and the standard description statement of the topic, and the test item result is obtained.

[0071] In this step, the BM25 algorithm is used to parse the first description statement, and the word q i is generated.

[0072] The relevance score of the word q i to the first description statement is determined.

[0073] The relevance score of the word q i to the first description statement is weighted and summed to obtain the relevance of the first description statement to the standard description statement.

[0074] The standard description of each topic has three results: consistent, inconsistent, and partially consistent. When performing similarity analysis, the evaluation result is one of them. According to the results of model analysis, the evaluation conclusion of the evaluation item is given. This operation process is repeated until all topics to be compared complete similarity analysis. The relevance Score(Q,d) of the first description statement to the standard description statement is calculated according to the following formula:

[0075]

[0076] Where Q is the first description statement; d is the standard description statement; n is the number of words in the first description statement; q i is the i-th word after parsing the first description statement; IDF(q i ) is the weight of the word q i ; N is the number of standard description statements in the index; n(q i ) is the number of standard description statements containing the word q i ; k1 is the first adjustment factor; b is the second adjustment factor; f i is the frequency of the word q i in the standard description statement; dl is the length of the standard description statement; avgdl is the average length of all standard description statements.

[0077] Step 106: Generate an evaluation report based on the test item results and the contents of the survey form.

[0078] Combined with the contents of the system survey form, a preliminary evaluation report is generated according to the requirements of the report template. After the evaluator reviews and confirms the report content, a formal evaluation report is generated.

[0079] The present invention uses historical data for learning and training to realize an automated evaluation model to improve the effectiveness and quality of compliance evaluation services such as security protection and cryptography, and solve the problem that service quality and efficiency are heavily dependent on the individual ability of the evaluator and cannot support the rapid expansion of the business.

[0080] Based on the same inventive concept, an embodiment of the present invention also provides an evaluation system based on semantic similarity analysis. Since the principle of solving the problem by this system is similar to that of the evaluation method based on semantic similarity analysis, the implementation of this system can refer to the implementation of the evaluation method based on semantic similarity analysis, and the repeated parts will not be repeated.

[0081] In another embodiment, the evaluation system based on semantic similarity analysis provided by the embodiment of the present invention is as follows: Figure 4 As shown, including:

[0082] The first acquisition module 10 is used to obtain a survey form on the environment and system information of the evaluation system.

[0083] The first extraction module 20 is configured to extract a first description sentence related to a test item in the survey form.

[0084] The determination module 30 is configured to determine a topic corresponding to the first description statement.

[0085] The second acquisition module 40 is used to acquire a standard description sentence of the topic.

[0086] The similarity analysis module 50 is used to perform similarity analysis on the first description sentence corresponding to the topic and the standard description sentence of the topic to obtain a test item result.

[0087] The evaluation report generating module 60 is used to generate an evaluation report based on the test item results and the contents of the survey form.

[0088] Exemplarily, the determining module includes:

[0089] An analysis unit is configured to analyze the topic to which the first description statement belongs using an LDA model algorithm, and compare the topic with all topics of the evaluation items in the database. If the topic does not belong to a topic related to the evaluation item, the target first description statement is deleted and recorded.

[0090] The extraction unit is configured to extract an implicit topic of the first description sentence from the first description sentence using an LDA model and associate the implicit topic with the evaluation item.

[0091] The analysis unit includes, for example:

[0092] The acquisition device is configured to acquire a target number of topics before training the LDA model.

[0093] The text data set processing device is configured to perform vocabulary segmentation and stop word filtering on the text data set.

[0094] The model training device is configured to train the LDA model using historical data to generate a document topic distribution and a word topic distribution.

[0095] The model processing device is configured to denoise and model parameterize the trained LDA model and perform topic prediction.

[0096] The similarity analysis module includes, for example:

[0097] The word parsing unit is configured to parse the first description sentence using a BM25 algorithm to generate a word q i .

[0098] The determination unit is configured to determine a relevance score of the word q i to the first description sentence.

[0099] The weighted summation unit is configured to perform weighted summation on the relevance score of the word q i to the first description sentence to obtain a relevance of the first description sentence to the standard description sentence.

[0100] The weighted summation unit includes, for example:

[0101] The calculation device is configured to calculate the relevance Score(Q, d) of the first description sentence to the standard description sentence according to the following formula:

[0102]

[0103] wherein Q is the first description sentence; d is the standard description sentence; n is the number of words in the first description sentence; q i is the i-th word after parsing the first description sentence; IDF(q i ) is the weight of the word q i ; N is the number of standard description sentences in the index; n(q i ) is the number of standard description sentences containing the word q i ; k1 is a first adjustment factor; b is a second adjustment factor; f i is the word q iThe frequency of occurrence of the standard description sentence; dl is the length of the standard description sentence; avgdl is the average length of all standard description sentences.

[0104] The more specific working processes of the above-mentioned modules can refer to the corresponding contents disclosed in the foregoing embodiments, and will not be described here in detail.

[0105] In another embodiment, the present application provides a computer device comprising a processor and a memory; wherein the processor implements the steps of the above-mentioned evaluation method based on semantic similarity analysis when executing the computer program stored in the memory.

[0106] The more specific processes of the above-mentioned method can refer to the corresponding contents disclosed in the foregoing embodiments, and will not be described here in detail.

[0107] In another embodiment, the present application provides a computer readable storage medium for storing a computer program; the computer program is executed by a processor to implement the steps of the above-mentioned evaluation method based on semantic similarity analysis.

[0108] The more specific processes of the above-mentioned method can refer to the corresponding contents disclosed in the foregoing embodiments, and will not be described here in detail.

[0109] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system, device and storage medium disclosed in the embodiments, since they correspond to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0110] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in the embodiments or some parts of the embodiments.

[0111] The present application has been described in detail in the above in combination with specific implementation manners and exemplary examples, but these descriptions cannot be understood as limitations of the present application. Those skilled in the art understand that the technical solutions and implementation manners of the present application can be variously replaced, modified or improved without departing from the spirit and scope of the present application, and these all fall within the scope of the present application. The protection scope of the present application is subject to the appended claims.

Claims

1. An evaluation method based on semantic similarity analysis, characterized in that: include: Obtain a survey form for the environment and system information of the evaluation system; Extract the first descriptive statement related to the test item in the survey form; Determine the topic corresponding to the first description statement; specifically, use the LDA model algorithm to analyze the topic to which the first description statement belongs, and compare it with all the topics of the evaluation items in the database. If it does not belong to the topic related to the evaluation item, delete the target first description statement and record it; for the clear first description statement, use the LDA model to extract the implicit topic of the first description statement and associate it with the evaluation item; Get the standard description sentence of the subject; The first description sentence corresponding to the topic and the standard description sentence of the topic are analyzed for similarity to obtain the test result. Specifically, the BM25 algorithm is used to parse the first description sentence and generate the word ; Determining word The relevance score with the first description sentence; Perform weighted summation on the relevance scores of the first description statement to obtain the relevance of the first description statement to the standard description statement; Generate an evaluation report based on the test item results and the contents of the survey form; The LDA model algorithm is used to analyze the topic to which the first description sentence belongs, including: Before training the LDA model, obtain the target number of topics; Perform word segmentation and stop word filtering on the text dataset; Use historical data to train the LDA model to generate document topic distribution and word topic distribution; De-noise and adjust the model parameters of the trained LDA model and perform topic prediction; The word The relevance scores of the first description statement are weighted and summed to obtain the relevance of the first description statement with the standard description statement, including: The correlation between the first description statement and the standard description statement is calculated according to the following formula: Score ( Q , d ): ； in, Q is the first description statement; d is a standard description statement; n is the number of words in the first description sentence; After parsing the first description statement i word; For words The weight of N is the number of standard description statements in the index; Contains words The number of standard description statements; is the first regulatory factor; b is the second regulatory factor; For words Frequency of occurrence in standard description sentences; is the length of the standard description sentence; is the average length of all standard description sentences.

2. An evaluation system based on semantic similarity analysis, characterized in that: include: The first acquisition module is used to obtain a survey form for the environment and system information of the evaluation system; A first extraction module is used to extract a first description sentence related to the test item in the survey form; a determination module, configured to determine a topic corresponding to the first description statement; The second acquisition module is used to obtain a standard description sentence of the subject; A similarity analysis module is used to perform similarity analysis on the first description sentence corresponding to the topic and the standard description sentence of the topic to obtain a test item result; Evaluation report generation module, used to generate an evaluation report based on the test item results and the contents of the survey form; The determination module includes: an analysis unit for analyzing the topic to which the first description sentence belongs using an LDA model algorithm, and comparing it with all the topics of the evaluation items in the database; if the first description sentence does not belong to a topic related to the evaluation item, deleting the target first description sentence and recording it; an extraction unit for extracting the implicit topic of the first description sentence using an LDA model for the explicit first description sentence, and associating it with the evaluation item; The analysis unit comprises: An acquisition device, used to acquire a target number of topics before training the LDA model; A text data set processing device, used for performing vocabulary segmentation and stop word filtering operations on the text data set; A model training device for training the LDA model using historical data to generate document topic distribution and word topic distribution; Model processing device, used to denoise and adjust the model parameters of the trained LDA model and perform topic prediction; The similarity analysis module includes: The word parsing unit is used to parse the first description sentence using the BM25 algorithm to generate word ; Determination unit, used to determine the word Relevance score to the first descriptive statement; Weighted summation unit, used to sum words Perform weighted summation on the relevance scores of the first description statement to obtain the relevance of the first description statement to the standard description statement; The weighted summation unit includes: A calculation device for calculating the correlation between the first description statement and the standard description statement according to the following formula Score ( Q , d ): ； in, Q is the first description statement; d is a standard description statement; n is the number of words in the first description sentence; After parsing the first description statement i word; For words The weight of N is the number of standard description statements in the index; Contains words The number of standard description statements; is the first regulatory factor; b is the second regulatory factor; For words Frequency of occurrence in standard description sentences; is the length of the standard description sentence; is the average length of all standard description sentences.

Images