A method for optimizing system scoring coverage

Abstract

The present invention crawls the standard text related to the online information security of the industry standard as the original data through the crawler frame webmagic, performs structured extraction on the industry standard file that needs to be updated through preset text matching rules, obtains structured standard data, establishes a field and risk weight table, obtains the updated industry standard text, and mines the mapping relationship between the differences by evaluating the differences between the old and new standard texts and the differences between the old and new scoring standard texts. The updated industry standard content is identified through the difference evaluation system, and the dynamic update of the sorted list that needs to be reviewed and adjusted is realized, which can make the system scoring standard cover more comprehensive and reduce the negative effects caused by incomplete system evaluation; it can match the national standards in related aspects in a targeted manner, so that the artificially established scoring standards can be combined with the latest national standards, so that the scoring standards cover more national standards, making them more standardized and more feasible.

Description

[Technical field]

[0001] The present invention relates to the technical field of deep learning algorithm application, and in particular to a system scoring coverage tuning method.

Technical Background

[0002] Computer security is a content that needs to be continuously optimized within important national functional departments. There are tens of millions of computer configurations. For each configuration, we need to test its security, conduct manual verification and operation configuration to see if it is safe enough. For example, if the login password requires numbers, special characters, and letters. If there is a public security system, and the administrator configures its password to only require pure numbers, then it is unsafe. At this time, the tester will give such a system a low score to prove that it is unsafe. System scoring is an indispensable aspect of large and small systems in the modern information society. It can use a set of standards to score various aspects of the system, so as to timely discover system defects and possible problems, and facilitate managers to repair them in time. However, at this stage, the system scores of various aspects are often set by humans, and there may be important issues that have not been covered by some evaluations of the system, which is easy to cause unknown negative consequences. For the operating systems of various aspects of society, the state will establish a series of complete standards to regulate them. Using national standards to regulate system operation can greatly reduce the unknown consequences of system operation, but it is believed that the details and key points involved in the set system may not be completely consistent with the national standards. For example, the old rules may not be able to adapt to the new national standards. This requires people to constantly formulate rules based on national standards, which may lead to incomplete coverage or the need to constantly change the standards. Therefore, it is necessary to optimize the coverage of the system scoring. [Summary of the invention]

[0003] The purpose of the present invention is to solve the defects of the above-mentioned prior art and provide a system scoring coverage tuning method.

[0004] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0005] A method for optimizing system scoring coverage, comprising:

[0006] Use the crawler framework webmagic to crawl industry-standard online information security-related standard text data as raw data;

[0007] The first directory and the second directory are obtained according to the standard text data, and the edit distance of each row of the first directory and the second directory is traversed by the python-Levenshtein toolkit, and the ratio of the number of subdirectories with the edit distance less than 2 to the number of subdirectories of the first directory is used as the directory overlap ratio; and the suspected update standard is obtained according to the directory overlap ratio;

[0008] The industry standard file data that needs to be updated is structuredly extracted through preset text matching rules to obtain structured standard data;

[0009] Acquiring the fault cause contained in the structured standard data by matching the structured standard data with a fault cause keyword;

[0010] By using a machine learning method, different weights are assigned to the average risk level of the failure time and the average risk level of the economic loss caused by the failure, and the comprehensive risk level of the system failure based on the industry characteristics is calculated, and finally a comprehensive risk level table of the industry system failure is output;

[0011] The network environment that needs to be tested is evaluated according to the structured standard data of the standard text to obtain the evaluation score p1. The reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2.

[0012] Determine whether the updated standard text is accurate based on the consistency between p1 and p2. If they are consistent, the updated standard text is accurate; otherwise, the updated standard text is inaccurate.

[0013] If the update standard text is inaccurate, the field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table, and the difference point is matched from two perspectives: the failure time and the economic loss caused by the failure, and the average risk level of the difference point from two perspectives: the failure time and the economic loss caused by the failure are obtained, and the average risk level of the difference point is sorted by a sorting algorithm according to the average risk level to obtain the average risk level sequence table of the difference point;

[0014] The reviewers conduct manual review based on the average risk level ranking table of the differences.

[0015] Preferably, the standard text data related to industry standard online information security is crawled as original data through the crawler framework webmagic, including: extracting target links according to seed links and putting them into the queue to be crawled, parsing and downloading the standard text data from the page, webmagics uses Jsoup component to parse HTML pages, and stores the extracted standard text data in text file format and stores it in the database.

[0016] Preferably, the step of obtaining the first directory and the second directory based on the standard text data; and obtaining the directory overlap ratio by traversing the edit distance of each row of the first directory and the second directory through the python-Levenshtein toolkit includes:

[0017] According to the standard text data, a piece of the standard text data is obtained as the first standard text data, and other standard text data except the first standard text data are traversed. During the traversal, one piece of the standard text data in the remaining other standard text data is used as the second standard text data, and the directory containing the word "requirement" in the directory of the first standard text data is used as the first directory, and the directory containing the word "requirement" in the directory of the second standard text data is used as the second directory, and the edit distance of each row of directories in the first directory and the second directory is traversed by the python-Levenshtein toolkit, and the ratio of the number of sub-directories with an edit distance less than 2 to the number of sub-directories of the first directory is calculated as the directory overlap ratio. When the directory overlap ratio exceeds a preset value, the second standard text data is used as the suspected update standard of the first standard text data.

[0018] Preferably, the fault causes contained in the structured standard data are obtained by matching the structured standard data with the fault cause keywords; and different weights are assigned to the average risk level of the fault time, the average risk level of the fault time, and the average risk level of the economic loss caused by the fault by a machine learning method, and the comprehensive risk level of the system fault based on the industry characteristics is calculated, and finally a comprehensive risk level table of the industry system fault is output, including:

[0019] Collect status information of the system under normal operation, the status information includes: system response status, network equipment status, database status, establish a mapping relationship table between fault causes and fault states, obtain the system preset fault state by inputting a preset fault cause into the system, establish a mapping relationship between the preset fault cause and the system preset fault state, obtain the current system fault state according to the status information, calculate the similarity between the current system fault state and the preset fault state name in the system preset fault state by the string edit distance method, and when the matching degree between the current system fault state and the preset fault state name in the system preset fault state is higher than the set similarity pair threshold, obtain the the preset fault state of the system corresponding to the current system fault state, obtaining the fault cause of the system fault according to the mapping relationship between the preset fault cause and the preset fault state of the system, obtaining the situation of the fault in the system, the fault situation including the fault time, the economic loss caused by the fault, and the reputation loss caused by the fault, the fault situation is stored in the history table of the database, matching the structured standard data with the keywords of the fault cause, when the matching degree between the keywords of the fault cause and the structured standard data is higher than the specific threshold, it is considered that the structured standard data contains the fault cause, and the structured standard data is stored in the history table of the database;

[0020] According to the history table of the system database, the economic losses caused by the system failure are sorted from large to small using the Learning to Rank sorting algorithm to obtain a sequence table of economic losses caused by the failure, and 5 failure time thresholds are preset according to the system failure time sequence table. The failure time is divided into 5 intervals according to the failure time threshold. System failures in the same interval are of the same risk level, with a total of 5 risk levels. The failure time risk level of the system failure is obtained according to the interval in which the system failure is located, and finally a failure time risk level table is obtained. According to the sequence table of economic losses caused by the system failure, 5 economic loss thresholds are preset. The economic losses are divided into 5 intervals according to the economic loss thresholds. System failures in the same interval are regarded as the same risk level. There are 5 risk levels in total. The economic loss risk level of the system failure is obtained according to the interval where the system failure is located, and finally a risk level table of economic losses caused by the failure is obtained; according to the failure time and the economic loss caused by the failure, the risk levels of various systems in the industry for various failure causes are averaged to obtain the average risk level of the failure time of each failure cause in the industry system and the average risk level of the economic loss caused by the failure; according to the fields corresponding to the failure causes in the history table, the average risk level of each field in the industry is obtained, and finally it is output in a table, which is an average risk level table of the industry according to the failure time and the economic loss caused by the failure.

[0021] Preferably, the automatic testing system automatically evaluates the network environment that needs to be tested according to the structured standard data of the standard text to obtain the evaluation score p1, and the assessor manually evaluates the network environment that needs to be tested to obtain the evaluation score p2, including:

[0022] The network environment that needs to be tested is automatically evaluated according to the structured standard data of the updated standard text obtained to obtain the evaluation score p1; the reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2; the network environment that needs to be tested refers to the information system that needs to be evaluated for the network security environment; the automatic testing system is an automatic testing tool with preset software, which automatically evaluates the network environment through the industry standard field name or the corresponding scoring standard content in the structured standard of the standard text to obtain the evaluation score; the reviewer manually evaluates the network environment that needs to be tested, including: the structured standard data of the updated standard text obtained and the difference between the new and old scoring standard texts are extracted, and the mapping relationship between the two differences is pushed to the reviewer, so as to assist the reviewer to manually evaluate the network environment; compare the results between p1 and p2 to see if they are consistent, and process the updated standard text according to the comparison results; if they are consistent, it is judged that the updated standard text is accurate, and the updated standard text and the corresponding structured standard data are stored in the database; if the results are inconsistent, it is judged by default that the updated standard text field is updated incorrectly, resulting in the score difference.

[0023] Preferably, if the update standard text is inaccurate, the field information of each difference point is obtained, the field information of the difference point is matched with the corresponding field in the average risk level table, and the difference point is matched from two perspectives: the failure time and the economic loss caused by the failure. The average risk level of the difference point from two perspectives of the failure time and the economic loss caused by the failure is obtained, and the difference point is sorted by a sorting algorithm according to the average risk level to obtain the average risk level sequence table of the difference point; the reviewer performs manual review according to the average risk level sequence table of the difference point, including:

[0024] During the scoring process, the text segments that lead to different scores are regarded as the difference points. The difference between the standard text that needs manual review and adjustment and the national standard text is determined through the difference points of the updated standard text. The number of standard texts that need manual review and adjustment is counted and the first round of sorting is performed from large to small according to the number of standard texts that need manual review and adjustment. The field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table. The difference point is matched from two perspectives: the failure time and the economic loss caused by the failure, and the average risk level of the difference point from the two perspectives of the failure time and the economic loss caused by the failure is obtained. According to the average The risk levels are sorted using a sorting algorithm to obtain a sequence table of the average risk levels of the difference points from the two perspectives of failure time and economic losses caused by the failure. The sequence table of the average risk levels sorts the difference points from the two perspectives of failure time and failure cause. Reviewers can give priority to reviewing and adjusting the difference points with higher risk levels from the two perspectives. According to the characteristics of the industry in which the standard text needs to be manually reviewed and adjusted, different weights are assigned to the average risk level of the failure time and the average risk level of the economic losses caused by the failure to obtain the comprehensive risk level of each difference point. According to the comprehensive risk level, the machine learning sorting algorithm is used to sort and obtain a sequence table of the comprehensive risk level of each difference point.

[0025] According to the field information of each difference point, the risk level under the standard that needs to be reviewed and adjusted is matched, and the risk level of the national standard is matched, and the risk level change of each difference point under the standard that needs to be reviewed and adjusted and the national standard is compared, and the risk level of each difference point under the standard that needs to be reviewed and adjusted is subtracted from the risk level under the national standard to obtain the risk level difference of each difference point, and obtain the risk level difference change of each difference point;

[0026] Sum the risk level differences of all difference points of each standard that requires manual review and adjustment to obtain the sum of the risk level differences of each standard that requires manual review and adjustment; sort the standards that require manual review and adjustment in descending order according to the size of the sum of the risk level differences; determine the priority of the standard that requires manual review and adjustment according to the sorting result; the larger the sum of the risk level differences, the higher the priority; based on the average value of the risk level differences, divide the sum of the risk level differences by the number of difference points of the standard that requires manual review and adjustment to obtain the average risk level difference of the standard that requires review and adjustment; sort the standards that require manual review and adjustment in descending order according to the size of the average risk level difference; and determine the average priority of the standard that requires manual review and adjustment according to the sorting result;

[0027] The reviewers conduct the review based on the ranking results of the priority and average priority of the standards that need to be reviewed and adjusted on the review system. On the review system, the number of clicks on each standard that needs to be reviewed and adjusted is obtained, and the standards are ranked from large to small according to the number of clicks. The subjective priority of the standards that need to be reviewed and adjusted during manual review is determined based on the ranking results.

[0028] [The technical solution provided by the embodiment of the present invention may have the following beneficial effects]:

[0029] The present invention evaluates the coverage of artificially established system scoring standards and reminds to supplement the standards, which can make the system scoring standards more comprehensive and reduce the negative effects caused by incomplete system evaluation; it can match relevant national standards in a targeted manner, so that the artificially established scoring standards can be combined with the latest national standards, so that the scoring standards cover more national standards, making them more standardized and more feasible.

Brief Description of the Drawings

[0030] Figure 1 It is a flow chart of a system scoring coverage tuning method provided in an embodiment of the present invention. [Specific implementation method]

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] S1: crawling industry standard online information security related standard text data as raw data through the crawler frame webmagic, including: extracting target links according to seed links and putting them into the queue to be crawled, parsing and downloading the standard text data from the page, webmagics using Jsoup component to parse HTML pages, and storing the extracted standard text data in text file format or storing it in a database.

[0033] S2: deriving a first directory and a second directory according to standard text data; traversing the edit distance of each row of the first directory and the second directory through the python-Levenshtein toolkit to obtain a directory overlap ratio, including: obtaining a piece of the standard text data as the first standard text data according to the standard text data, traversing other standard text data, and taking one of the remaining standard text data as the second standard text data during the traversal, taking the directory containing the word "requirement" in the directory of the first standard text data as the first directory, and taking the directory containing the word "requirement" in the directory of the second standard text data as the second directory, and traversing the edit distance of each row of the directory in the first directory and the second directory through the python-Levenshtein toolkit, calculating the ratio of the number of sub-directories with an edit distance less than 2 to the number of sub-directories of the first directory as the directory overlap ratio, and when the directory overlap ratio exceeds a preset value, taking the second standard text data as the suspected update standard of the first standard text data.

[0034] For example: First, the first standard text data crawled by the crawler are: "Financial Industry Cybersecurity Level Protection Evaluation Guide" and the second standard text data "Financial Industry Cybersecurity Level Protection Implementation Guide Part 2: Basic Requirements" two industry standard files. Assume that the corresponding old industry standard is "Financial Industry Cybersecurity Level Protection Evaluation Guide"; for each industry standard file, extract the text information whose name contains the keyword "requirements" in the second-level directory of the file; and use the python-Levenshtein toolkit to traverse the edit distance of each row of the first directory and the second directory, and calculate the sub-directories with an edit distance less than 2. For example, the text information extracted from "Financial Industry Cybersecurity Level Protection Evaluation Guide" includes: 6 Second-level evaluation requirements, 7 Third-level evaluation requirements, 8 Fourth-level evaluation requirements, all text information under the three second-level directories; and the text information extracted from "Financial Industry Cybersecurity Level Protection Implementation Guide Part 2: Basic Requirements" includes: 7 Second-level security requirements, 8 Third-level security requirements, 9 Fourth-level security requirements, all text information under the three second-level directories. According to the "Second Level Requirements", there are four points below. For the "Guidelines for the Implementation of Cybersecurity Level Protection in the Financial Industry Part 2: Basic Requirements", they are: General Security Requirements, Cloud Computing Security Extension Requirements, Mobile Internet Security Extension Requirements, and Internet of Things Security Extension Requirements; and for the "Financial Industry Cybersecurity Level Protection Assessment Guide", they are: General Security Assessment Requirements, Cloud Computing Security Assessment Extension Requirements, Mobile Internet Security Assessment Extension Requirements, and Internet of Things Security Assessment Extension Requirements; According to the catalog under "General Security Requirements", it includes: secure physical environment, secure communication network, secure area boundary, etc.; According to the secure physical environment, the physical location text information of the two industry standards is as follows: 7.1.1.1 Physical Location Selection This requirement includes: a) The computer room site should be selected in a building with earthquake, wind and rain resistance. b) The computer room site should avoid being located on the top floor or basement of the building, otherwise waterproof and moisture-proof measures should be strengthened. 6.1.1.1 Physical location selection assessment unit (L2-PES1-01) This assessment unit includes the following requirements: a) Assessment indicators: The computer room site should be selected in a building that is earthquake-proof, wind-proof and rain-proof. b) Assessment objects: Record form documents and computer rooms. c) Assessment implementation includes the following: 1) It should be verified whether the building has the building earthquake-resistant fortification approval document. 2) It should be verified whether there is rainwater leakage in the computer room. 3) It should be verified whether the doors and windows of the computer room are free of severe dust caused by wind. 1) It should be verified whether the roof, walls, doors, windows and floors are not damaged or cracked. d) Unit judgment: If 1) to 4) are all affirmative, it meets the requirements of this assessment unit indicator, otherwise it does not meet or partially meets the requirements of this assessment unit indicator.Assessment unit (L2-PES1-02) This assessment unit includes the following requirements: a) Assessment indicators: The computer room should not be located on the top floor or basement of the building, otherwise waterproofing and moisture-proofing measures should be strengthened. b) Assessment object: Computer room. c) Assessment implementation: It should be verified whether the computer room is not located on the top floor or basement of the building. If not, it should be verified whether waterproofing and moisture-proofing measures have been taken in the computer room. d) Unit judgment: If the above assessment implementation content is positive, it meets the requirements of this assessment unit indicator, otherwise it does not meet the requirements of this assessment unit indicator. The requirements and standards in the "Financial Industry Cybersecurity Level Protection Assessment Guide" include all the contents in the "Financial Industry Cybersecurity Level Protection Implementation Guide Part 2: Basic Requirements", and the "Financial Industry Cybersecurity Level Protection Assessment Guide" includes all the requirements of the "Financial Industry Cybersecurity Level Protection Implementation Guide Part 2: Basic Requirements", and the catalog overlap ratio exceeds the preset value of 80%. As a result, the system determines that the two industry standards are too similar. Therefore, they cannot be distinguished. If no distinction is made, the system will parse and extract text information based on the "Guidelines for the Implementation of Cybersecurity Level Protection in the Financial Industry Part 2: Basic Requirements" to update industry standard operations, which will directly lead to update errors.

[0035] S3: Preset text matching rules to perform structured extraction on industry standard files that need to be updated to obtain structured standard data, including:

[0036] The structured standard unified preset rules are expressed as: {[industry standard field name], [corresponding scoring standard content]}, the non-network security type is the type that cannot be automatically tested for security scoring, and the automatically testable network security type is the type that can be automatically tested for security scoring by the test system. The structured extraction is specifically implemented as follows:

[0037] Based on the industry standard text of "Information Security Technology Based on IPv6 High-Performance Network Audit System Product Security Technical Requirements", the field information features are extracted from the text information using syntax analysis technology; wherein, the field information features include field type and field content corresponding to the field type. For this industry standard document, the extracted text information is all text information in its first-level title that contains the keyword "requirements". Filter out other unnecessary first-level title text information. For example, the first-level titles are: "Security Function Requirements", "Environmental Adaptability Requirements", "Performance Requirements", and "Security Assurance Requirements". For each second-level title, extract the title of the corresponding last-level sub-topic as one of the keys in the feature value, and the requirements corresponding to its text information are used as the value corresponding to this key. In the "Security Function Requirements" module, it will be found that the standard requirement information in this industry standard document includes both non-network security types and automatically testable network security types. Among them, the five modules of "Information Collection", "Data Reduction", "Management and Control Requirements", "Security Management", and "Data Storage", the extracted text information is all non-network security types after vectorization. The remaining four modules, "Audit Record Statistics", "Audit Record Analysis and Processing", "Identification and Identification", and "Audit Log", all have text information that can be automatically tested for network security after vectorization. The details are as follows: For the non-network security type, "Data Storage", according to its last level title, the extracted keys are: "Storage Media", "Data Deletion", "Backup and Recovery", etc. The data format after text information standardization is as follows: {'Data deletion': [a) Should be able to record basic information of deletion behavior, including time, date, operator, and description of deletion content; b) Be able to set policies to automatically delete data that exceeds the retention period], 'Data deletion scoring criteria': [a) Failure to record basic information of deletion behavior, including time, date, operator, and description of deletion content will result in a deduction of 1 point; b) Failure to set policies to automatically delete data that exceeds the retention period will result in a deduction of 1 point]}{'Backup and recovery': [IPv6 network audit products should provide audit record backup and recovery functions], 'Backup and recovery scoring criteria': [IPv6 network audit products do not provide audit record backup and recovery functions, deduction of 1 point]} For the network security type that can be automatically tested, "Audit record analysis and processing", according to its last level title, the extracted keys are: "Correlation analysis", "Abnormal analysis", "Response warning", etc.The data format after text information standardization is as follows: {'Abnormal analysis': [a) Ability to predefine abnormal events, and trigger an alarm when the number of occurrences in a certain time period or the frequency domain reaches a certain threshold or a certain flow reaches a threshold, b) Ability to define abnormal behavior based on the results of the correlation analysis function, and trigger an alarm for abnormal behavior of the system, c) Other abnormal situations], 'Abnormal analysis scoring criteria': [a) Ability to predefine abnormal events, and trigger an alarm when the number of occurrences in a certain time period or the frequency domain reaches a certain threshold or a certain flow reaches a threshold, deduct 1 point, b) Ability to define abnormal behavior based on the results of the correlation analysis function, and trigger an alarm for abnormal behavior of the system, deduct 1 point, c) Other abnormal situations, deduct 1 point]}{'Correlation analysis': [a) Basic information correlation, including based on time, event, source IP address, source port address, destination IP address, service type, network protocol and other information collection, b) statistical association, use data mining algorithms and other methods to statistically analyze the association between different network events], 'Association analysis scoring criteria': [a) No basic information association, including information collection based on time, event, source IP address, source port address, destination IP address, service type, network protocol, etc., deduct 0.5 points, b) No statistical association, use data mining algorithms and other methods to statistically analyze the association between different network events, deduct 0.5 points]}{'Response warning': [a) The product should support policy settings to trigger alarms, b) It should be able to record alarms, and the content should include: date, time, event subject, event topic, event level, event description, number of alarms, event results, c) The alarm method should at least support email alarms and SNMP Trap alarm, audio and video alarm, SMS alarm, etc.], 'Response warning scoring criteria': [a) The product cannot support policy settings to trigger alarms, deduct 1 point, b) Failure to record alarms, the content should include: date, time, event subject, event topic, event level, event description, number of alarms, event results, deduct 1 point, c) The alarm method does not support at least one of the following methods: email alarm, SNMP trap alarm, audio and video alarm, SMS alarm, etc., deduct 1 point]}.

[0038] S4: The fault causes contained in the structured standard data are obtained by matching the structured standard data with the fault cause keywords; and different weights are assigned to the average risk level of the fault time, the average risk level of the fault time, and the average risk level of the economic loss caused by the fault by the machine learning method, and the comprehensive risk level of the system fault based on the industry characteristics is calculated, and finally the comprehensive risk level table of the industry system fault is output, including:

[0039] Collect status information of the system under normal operation, the status information includes: system response status, network equipment status, database status, establish a mapping relationship table between fault causes and fault states, obtain the system preset fault state by inputting a preset fault cause into the system, establish a mapping relationship between the preset fault cause and the system preset fault state, obtain the current system fault state according to the status information, calculate the similarity between the current system fault state and the preset fault state name in the system preset fault state by the string edit distance method, and when the matching degree between the current system fault state and the preset fault state name in the system preset fault state is higher than the set similarity pair threshold, obtain the the preset fault state of the system corresponding to the current system fault state, obtaining the fault cause of the system fault according to the mapping relationship between the preset fault cause and the preset fault state of the system, obtaining the situation of the fault in the system, the fault situation including the fault time, the economic loss caused by the fault, and the reputation loss caused by the fault, the fault situation is stored in the history table of the database, matching the structured standard data with the keywords of the fault cause, when the matching degree between the keywords of the fault cause and the structured standard data is higher than the specific threshold, it is considered that the structured standard data contains the fault cause, and the structured standard data is stored in the history table of the database;

[0040] According to the history table of the system database, the economic losses caused by the system failure are sorted from large to small using the Learning to Rank sorting algorithm to obtain a sequence table of economic losses caused by the failure, and 5 failure time thresholds are preset according to the system failure time sequence table. The failure time is divided into 5 intervals according to the failure time threshold. System failures in the same interval are of the same risk level, with a total of 5 risk levels. The failure time risk level of the system failure is obtained according to the interval in which the system failure is located, and finally a failure time risk level table is obtained. According to the sequence table of economic losses caused by the system failure, 5 economic loss thresholds are preset. The economic losses are divided into 5 intervals according to the economic loss thresholds. System failures in the same interval are regarded as the same risk level. There are 5 risk levels in total. The economic loss risk level of the system failure is obtained according to the interval where the system failure is located, and finally a risk level table of economic losses caused by the failure is obtained; according to the failure time and the economic loss caused by the failure, the risk levels of various systems in the industry for various failure causes are averaged to obtain the average risk level of the failure time of each failure cause in the industry system and the average risk level of the economic loss caused by the failure; according to the fields corresponding to the failure causes in the history table, the average risk level of each field in the industry is obtained, and finally it is output in a table, which is an average risk level table of the industry according to the failure time and the economic loss caused by the failure.

[0041] If the failure cause that most affects the failure time of the system is data poisoning, the corresponding field is data integrity loss. According to the history table of the system database, the Learning to Rank sorting algorithm is used to sort the economic losses caused by the system failure from large to small to obtain the sequence table of economic losses caused by the failure. According to the system failure time sequence table, 5 failure time thresholds are preset. According to the failure time threshold, the failure time is divided into 5 intervals. System failures in the same interval are of the same risk level, with a total of 5 risk levels. The failure time risk level of the system failure is obtained according to the interval in which the system failure is located, and finally a failure time risk level table is obtained. According to the sequence table of economic losses caused by system failures, 5 economic loss thresholds are preset. According to the economic loss threshold, the economic loss is divided into 5 intervals. System failures in the same interval are regarded as having the same risk level, with a total of 5 risk levels. The economic loss risk level of the system failure is obtained according to the interval in which the system failure is located, and finally a risk level table of economic losses caused by failures is obtained;

[0042] Considering the failure time and the economic loss caused by the failure, the risk level of each failure cause in each system of the industry is averaged to obtain the average risk level of the failure time of each failure cause in the industry system and the average risk level of the economic loss caused by the failure. According to the field corresponding to the failure cause in the history table, the average risk level of each field in the industry is obtained, and finally output in the form of a table, which is two average risk level tables of the industry considering the failure time and the economic loss caused by the failure;

[0043] The comprehensive risk level table includes the cause of the failure, the field and the comprehensive risk level. For example, in the financial services industry, the financial services industry focuses on the economic loss factor. For example, the importance of the failure time factor is relatively low, so a higher weight is given to the economic loss caused by the failure. The industry standard text that needs to be updated is obtained, and the mapping relationship between the differences is mined by evaluating the differences between the new and old standard texts and the differences between the new and old scoring standard texts. This mapping relationship is modeled. The differences include two aspects: 1. The degree of change of the standard text at different times 2. The degree of change of the scoring standard. The difference characteristics of the standard text can also be subdivided into three aspects: numerical value, degree adverb and quantity change. For example: the numerical description in the standard text has changed: the old standard text: the safety function of the product should cover 80%; the new standard text: the safety function of the product should cover 90%. The change is that the numerical feature changes from "80%" to "90%". The degree adverb in the standard text has changed: the old standard text: partially describes the safety function of the product; the new standard text: completely describes the safety function of the product. The change is that the degree adverb feature changes from "partial" to "complete". The number in the standard text has changed: Old standard text: Developers should provide a complete functional specification description, which should meet the following requirements: a) Fully describe the safety function of the product; b) Describe the purpose and usage of all safety function interfaces; c) Identify and describe all parameters related to each safety function interface; d) Describe the safety function implementation behavior related to the safety function interface; e) Describe the direct error message caused by the processing of the safety function implementation behavior; f) Describe the traceability of the safety function requirements to the safety function interface; New standard text: Developers should provide a complete functional specification description, which should meet the following requirements: a) Fully describe the safety function of the product; b) Describe the purpose and usage of all safety function interfaces; c) Identify and describe all parameters related to each safety function interface; d) Describe the safety function implementation behavior related to the safety function interface; e) Describe the direct error message caused by the processing of the safety function implementation behavior; f) Describe the traceability of the safety function requirements to the safety function interface; g) Describe all behaviors related to the safety function interface during the safety function implementation process; The change is that the number of standards has increased from "6" to "7".

[0044] Identify and update the content of industry standards through a difference evaluation system. The difference evaluation system includes a feature recognition model, including: identify "feature" entities in a given text through an entity recognition model, such as: "Tomorrow is the Mid-Autumn Festival", and "Mid-Autumn Festival" is a "festival" entity. Industry standards and scoring standards in the field of information security The feature recognition model is an entity recognition model that can identify entity words in a given text. The text involved has a strong regularity. The entity features in the training samples are taken out, a feature word dictionary is established, the sequence is annotated, and the entity is identified using a regular matching method. Take the identification of the features in "Describe the safety function of the product should cover 90%" as an example: first, the sentence is segmented [Describe the safety function of the product should cover 90%], and then the part-of-speech tagging tool is used for tagging. The tagging rule is that if the word is in the feature word dictionary, it is marked as E, and the rest are marked according to the part of speech given in the part-of-speech tagging tool. The resulting sequence is [VN DET N ADV VE]; the model is used to perform feature recognition on the industry standard text and scoring standard text to be processed, the LSTM network is used to encode the text with identified features, the sentences encoded by the LSTM are nonlinearly changed through a multi-layer perceptron to realize semantic understanding of the features, and the annotated sequence is input into the LSTM network as a feature and a label for training, wherein the output dimension is 50, the activation function is tanh, and then a Dropout layer of 0.5 is added, the activation function is a fully connected layer with softmax, the optimizer is adam, and the monitoring indicator is accuracy. The feature recognition module recognizes the given industry standard text "the description of the safety function of the product should cover 90%" and the scoring standard text "the description of the safety function of the product should cover if it only reaches 80%, deduct 1 point; the description of the safety function of the product should cover if it only reaches 70%, deduct 2 points; less than 70%, deduct 5 points (full score 5 points)", and after recognizing the numerical features, the text understanding module is used to perform semantic understanding on the identified features to realize modeling of the mapping relationship between such industry standard text and scoring standard text. For example, there is a correlation between the change in the numerical characteristics of the industry standard and the degree of change in the scoring standard. When the numerical characteristics of the industry standard are adjusted from "80%" to "90%", the corresponding numerical characteristics of the scoring standard also change from "80%" to "90%". The characteristic of the change is that the changes in the two characteristics are consistent.

[0045] S5: The automatic testing system automatically evaluates the network environment that needs to be tested according to the structured standard data of the standard text to obtain the evaluation score p1. The reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2, including:

[0046] The network environment that needs to be tested is automatically evaluated according to the structured standard data of the updated standard text obtained to obtain the evaluation score p1; the reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2; the network environment that needs to be tested refers to the information system that needs to be evaluated for the network security environment; the automatic testing system is an automatic testing tool with preset software, which automatically evaluates the network environment through the industry standard field name or the corresponding scoring standard content in the structured standard of the standard text to obtain the evaluation score; the reviewer manually evaluates the network environment that needs to be tested, including: the structured standard data of the updated standard text obtained and the difference between the new and old scoring standard texts are extracted, and the mapping relationship between the two differences is pushed to the reviewer, so as to assist the reviewer to manually evaluate the network environment; compare the results between p1 and p2 to see if they are consistent, and process the updated standard text according to the comparison results; if they are consistent, it is judged that the updated standard text is accurate, and the updated standard text and the corresponding structured standard data are stored in the database; if the results are inconsistent, it is judged by default that the updated standard text field is updated incorrectly, resulting in the score difference.

[0047] S6: If the updated standard text is inaccurate, the field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table, and the difference point is matched from two perspectives: the failure time and the economic loss caused by the failure. The average risk level of the difference point from two perspectives is obtained, and the difference point is sorted by a sorting algorithm according to the average risk level to obtain the average risk level sequence table of the difference point; the reviewer manually reviews the average risk level sequence table of the difference point, including:

[0048] During the scoring process, the text segments that lead to different scores are regarded as the difference points. The difference between the standard text that needs manual review and adjustment and the national standard text is determined through the difference points of the updated standard text. The number of standard texts that need manual review and adjustment is counted and the first round of sorting is performed from large to small according to the number of standard texts that need manual review and adjustment. The field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table. The difference point is matched from two perspectives: the failure time and the economic loss caused by the failure, and the average risk level of the difference point from the two perspectives of the failure time and the economic loss caused by the failure is obtained. According to the average The risk levels are sorted using a sorting algorithm to obtain a sequence table of the average risk levels of the difference points from the two perspectives of failure time and economic losses caused by the failure. The sequence table of the average risk levels sorts the difference points from the two perspectives of failure time and failure cause. Reviewers can give priority to reviewing and adjusting the difference points with higher risk levels from the two perspectives. According to the characteristics of the industry in which the standard text needs to be manually reviewed and adjusted, different weights are assigned to the average risk level of the failure time and the average risk level of the economic losses caused by the failure to obtain the comprehensive risk level of each difference point. According to the comprehensive risk level, the machine learning sorting algorithm is used to sort and obtain a sequence table of the comprehensive risk level of each difference point.

[0049] According to the field information of each difference point, the risk level under the standard that needs to be reviewed and adjusted is matched, and the risk level of the national standard is matched, and the risk level change of each difference point under the standard that needs to be reviewed and adjusted and the national standard is compared, and the risk level of each difference point under the standard that needs to be reviewed and adjusted is subtracted from the risk level under the national standard to obtain the risk level difference of each difference point, and obtain the risk level difference change of each difference point;

[0050] Sum the risk level differences of all difference points of each standard that requires manual review and adjustment to obtain the sum of the risk level differences of each standard that requires manual review and adjustment; sort the standards that require manual review and adjustment in descending order according to the size of the sum of the risk level differences; determine the priority of the standard that requires manual review and adjustment according to the sorting result; the larger the sum of the risk level differences, the higher the priority; based on the average value of the risk level differences, divide the sum of the risk level differences by the number of difference points of the standard that requires manual review and adjustment to obtain the average risk level difference of the standard that requires review and adjustment; sort the standards that require manual review and adjustment in descending order according to the size of the average risk level difference; and determine the average priority of the standard that requires manual review and adjustment according to the sorting result;

[0051] The reviewers conduct the review based on the ranking results of the priority and average priority of the standards that need to be reviewed and adjusted on the review system. On the review system, the number of clicks on each standard that needs to be reviewed and adjusted is obtained, and the standards are ranked from large to small according to the number of clicks. The subjective priority of the standards that need to be reviewed and adjusted during manual review is determined based on the ranking results.

Claims

1. A method for optimizing system scoring coverage, characterized in that: include: Use the crawler framework webmagic to crawl industry-standard online information security-related standard text data as raw data; The first directory and the second directory are obtained according to the standard text data, and the edit distance of each row of the first directory and the second directory is traversed by the python-Levenshtein toolkit, and the ratio of the number of subdirectories with the edit distance less than 2 to the number of subdirectories of the first directory is taken as the directory overlap ratio; According to the overlapping ratio of the directories, the industry standard files that need to be updated are obtained, including: According to the standard text data, a piece of the standard text data is obtained as the first standard text data, and other standard text data except the first standard text data are traversed. During the traversal, one piece of the standard text data in the remaining other standard text data is used as the second standard text data, and the directory containing the two words "required" in the directory of the first standard text data is used as the first directory, and the directory containing the two words "required" in the directory of the second standard text data is used as the second directory, and the edit distance of each row of directories in the first directory and the second directory is traversed by the python-Levenshtein toolkit, and the ratio of the number of sub-directories with an edit distance less than 2 to the number of sub-directories of the first directory is calculated as the directory overlap ratio, and when the directory overlap ratio exceeds a preset value, the second standard text data is used as the industry standard file that needs to be updated; Perform structured extraction of industry standard files that need to be updated through preset text matching rules to obtain structured standard data; The fault causes contained in the structured standard data are obtained by matching the structured standard data with the fault cause keywords; different weights are assigned to the average risk level of the fault time and the average risk level of the economic loss caused by the fault through machine learning methods, and the comprehensive risk level of the fault of the industry system based on the industry characteristics is calculated, and finally the average risk level table of the fault of the industry system is output; The network environment that needs to be tested is evaluated according to the structured standard data of the industry standard file that needs to be updated, and the evaluation score p1 is obtained. The reviewer manually evaluates the network environment that needs to be tested and obtains the evaluation score p2. Determine whether the industry standard file that needs to be updated is accurate based on the consistency between p1 and p2. If they are consistent, the industry standard file that needs to be updated is accurate; otherwise, the industry standard file that needs to be updated is inaccurate. If the industry standard document to be updated is inaccurate, the field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table, and the failure time of the difference point and the economic loss caused by the failure are matched to obtain the average risk level of the difference point in terms of the failure time and the economic loss caused by the failure. The difference points are sorted according to the average risk level using a sorting algorithm to obtain a sequence table of average risk levels; the reviewers conduct manual review based on the sequence table of average risk levels of the difference points.

2. A method for optimizing system scoring coverage according to claim 1, characterized in that: The method of crawling industry standard online information security-related standard text data as raw data through the crawler framework webmagic includes: extracting target links according to seed links and putting them into a queue to be crawled, parsing and downloading the standard text data from the page, webmagic using the Jsoup component to parse the html page, and storing the extracted standard text data in a text file format and storing it in a database.

3. A system scoring coverage tuning method according to claim 1, characterized in that: The fault causes contained in the structured standard data are obtained by matching the structured standard data with the fault cause keywords; different weights are assigned to the average risk level of the fault time and the average risk level of the economic loss caused by the fault by a machine learning method, and the comprehensive risk level of the fault of the industry system based on the industry characteristics is calculated, and finally the average risk level table of the fault of the industry system is output, including: Collect the status information of the industry system under normal operation, the status information includes: industry system response status, network equipment status, database status, establish a mapping relationship table between preset fault causes and preset fault states, obtain the preset fault state by inputting the preset fault cause into the industry system, establish the mapping relationship between the preset fault cause and the preset fault state, and obtain the fault state of the current industry system according to the status information. The similarity between the fault state of the current industry system and the preset fault state name in the preset fault state is calculated by the string edit distance method. When the matching degree between the fault state of the current industry system and the preset fault state name in the preset fault state is higher than the set similarity threshold, the preset fault state of the industry system corresponding to the fault state of the current industry system is obtained. According to the mapping relationship between the preset fault cause and the preset fault state of the industry system, the fault cause of the fault of the industry system is obtained, and the situation of the fault in the industry system is obtained. The fault situation includes the fault time, the economic loss caused by the fault, and the reputation loss caused by the fault. The fault situation is saved in the history table of the database. The structured standard data is matched with the keywords of the fault cause. When the matching degree between the keywords of the fault cause and the structured standard data is higher than a specific threshold, it is considered that the structured standard data contains the fault cause, and the structured standard data is saved in the history table of the database. Based on the history table of the database, use Learning to Rank sorting algorithm, sorting the economic losses caused by the failure of the industry system from large to small, and obtaining the sequence table of economic losses caused by the failure, presetting 5 failure time thresholds according to the failure time sequence table of the industry system, dividing the failure time into 5 intervals according to the failure time thresholds, the failures of the industry system in the same interval are of the same risk level, with a total of 5 risk levels, and obtaining the failure time risk level of the failure of the industry system according to the interval in which the failure of the industry system is located, and finally obtaining the failure time risk level table, presetting 5 economic loss thresholds according to the economic loss thresholds, dividing the economic losses into 5 intervals according to the economic loss thresholds, the failures of the industry system in the same interval are regarded as of the same risk level, with a total of 5 risk levels, and obtaining the economic loss risk level of the failure of the industry system according to the interval in which the failure of the industry system is located, and finally obtaining the risk level table of economic losses caused by the failure; according to the failure time and the economic loss caused by the failure, averaging the risk level of each failure cause in each industry system, obtaining the average risk level of the failure time of each failure cause in the industry system and the average risk level of the economic loss caused by the failure, obtaining the average risk level of each field of the industry according to the field corresponding to the failure cause in the history table, and finally outputting it in the form of a table.

4. The method for optimizing system scoring coverage according to claim 1, characterized in that: Through the automatic testing system, the network environment that needs to be tested is evaluated according to the structured standard data of the standard file to obtain the evaluation score p1. The reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2, including: According to the structured standard data of the industry standard file that needs to be updated, the network environment that needs to be tested is automatically evaluated to obtain the evaluation score p1; the reviewer manually evaluates the network environment that needs to be tested to obtain the evaluation score p2; the network environment that needs to be tested refers to the information system that needs to be evaluated for the network security environment; the automatic testing system is a built-in preset software automatic testing tool, which automatically evaluates the network environment through the industry standard field name or the corresponding scoring standard content in the structured standard of the industry standard file that needs to be updated to obtain the evaluation score; the reviewer manually evaluates the network environment that needs to be tested, including: mining the mapping relationship between the structured standard data of the industry standard file that needs to be updated and the old and new scoring standard texts and pushing them to the reviewer, so as to assist the reviewer in manually evaluating the network environment; comparing p1 Whether the results of p2 are consistent, the industry standard files that need to be updated are processed according to the comparison results; if they are consistent, the industry standard files that need to be updated are judged to be accurate, and the industry standard files that need to be updated and the corresponding structured standard data are stored in the database; if the results are inconsistent, it is judged by default that the fields of the industry standard files that need to be updated are updated incorrectly, resulting in different scores.

5. A method for optimizing system scoring coverage according to claim 1, characterized in that : If the industry standard file to be updated is inaccurate, the field information of each difference point is obtained, and the field information of the difference point is matched with the corresponding field in the average risk level table, and the difference point is matched from two perspectives: the failure time and the economic loss caused by the failure. The average risk level of the difference point from two perspectives of the failure time and the economic loss caused by the failure is obtained, and the average risk level sequence table of the difference point is obtained according to the average risk level by using a sorting algorithm; The reviewer conducts manual review based on the average risk level ranking table of the differences, including: During the scoring process, the field information that leads to different scores is regarded as the difference point. The difference between the industry standard file that needs to be updated and the national industry standard file is determined, the number of industry standard files that need to be manually reviewed and adjusted is counted, and the first round of sorting is performed from large to small according to the number of industry standard files that need to be manually reviewed and adjusted; the field information of each difference point is obtained, the field information of the difference point is matched with the corresponding field in the average risk level table, and the difference point is matched from two perspectives of failure time and economic loss caused by the failure to obtain the average risk of the difference point in terms of failure time and economic loss caused by the failure. Level, sorting by sorting algorithm according to the average risk level, obtaining a sequence table of the average risk levels of the difference points from the two perspectives of failure time and economic loss caused by the failure, the sequence table of the average risk level sorts the difference points from the two perspectives of failure time and failure cause, the reviewers give priority to reviewing and adjusting the difference points with higher risk levels from the two perspectives, according to the characteristics of the industry where the industry standard documents need to be manually reviewed and adjusted, different weights are given to the average risk level of the failure time and the average risk level of the economic loss caused by the failure, and the comprehensive risk level of each difference point is obtained, and the machine learning sorting algorithm is used to sort according to the comprehensive risk level to obtain the average risk level sequence table of each difference point; According to the field information of each difference point, the risk level under the standard that needs to be reviewed and adjusted is matched, and the risk level of the national standard is matched, and the risk level change of each difference point under the standard that needs to be reviewed and adjusted and the national standard is compared, and the risk level of each difference point under the standard that needs to be reviewed and adjusted is subtracted from the risk level under the national standard to obtain the risk level difference of each difference point, and obtain the risk level difference change of each difference point; Sum the risk level differences of all difference points of each standard that needs to be reviewed and adjusted to obtain the sum of the risk level differences of each standard that needs to be reviewed and adjusted; sort the standards that need to be reviewed and adjusted in descending order according to the size of the sum of the risk level differences; determine the priority of the standards that need to be reviewed and adjusted according to the sorting result; the larger the sum of the risk level differences, the higher the priority; based on the average value of the risk level differences, divide the sum of the risk level differences by the number of difference points of the standards that need to be reviewed and adjusted to obtain the average risk level difference of the standards that need to be reviewed and adjusted; sort the standards that need to be reviewed and adjusted in descending order according to the size of the average risk level difference; and determine the average priority of the standards that need to be reviewed and adjusted according to the sorting result; Reviewers conduct reviews on the review system based on the ranking results of the priority and average priority of the standards that need to be reviewed and adjusted. On the review system, the number of clicks on each article that needs to be reviewed and adjusted is obtained, and the articles are sorted from large to small according to the number of clicks. The subjective priority of the standards that need to be reviewed and adjusted during manual review is determined based on the ranking results.

Images