Key information searching method and device

Abstract

The invention provides a key information searching method and device. The key information searching method comprises the following steps: crawling a target website to obtain webpage content; extracting text information from the webpage content; judging whether the webpage content is obtained by crawling a specific keyword or not; if yes, matching the text information with the specific keyword by using a regular matching formula to obtain a key information evaluation score corresponding to the text information; if not, key index values corresponding to a plurality of preset key indexes are recognized from the text information, and a key information evaluation score is obtained through calculation based on the initial score of each key index, the weight corresponding to each key index and a weighted value algorithm; and if the key information evaluation score is greater than the score threshold, determining that the text information is key information. In the scheme, webpage contents are obtained in batches by utilizing a crawler, possible key information is judged and scored, and the information of which the evaluation score of the key information is greater than the scoring threshold value is determined as the key information, so that the aims of improving the efficiency and continuously searching the key information are fulfilled.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for searching key information. Background Art

[0002] The world today is in an era of information explosion. The Internet is like a vast ocean of information. It is like looking for a needle in a haystack to accurately retrieve key information in the ocean of information. Among them, key information refers specifically to information that is of great value to enterprises or individuals or is prohibited from appearing on the Internet and is strictly prevented from being leaked. In order to prevent the risks brought by the leakage of key information to enterprises or individuals, it is necessary to search and discover the leakage of key information in a timely manner and take corresponding emergency measures, such as contacting the website to delete it.

[0003] In the prior art, most mainstream websites have built their own search engines, which support users to search by entering keywords and display content page by page by turning pages. However, only one keyword can be entered at a time, and the display screen can only display about 10 items. It is inefficient and unsustainable to search for key information manually by turning pages on the website. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for searching key information to achieve the purpose of improving efficiency and continuously searching for key information.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a method for searching key information, the method comprising:

[0007] Use specific keywords or fuzzy keywords in the preset keyword configuration file as search term parameters of the python crawler to crawl the target website to obtain web page content;

[0008] Extracting text information from the webpage content;

[0009] Determining whether the webpage content is obtained by crawling through the specific keyword;

[0010] If yes, the text information is matched with the specific keyword by using a regular matching formula to obtain a key information evaluation score corresponding to the text information;

[0011] If not, identify key indicator values ​​corresponding to a plurality of preset key indicators from the text information, score each of the key indicators using the preset scoring rules and each of the key indicator values, obtain an initial score for each of the key indicators, and calculate the key information evaluation score corresponding to the text information based on the initial score of each of the key indicators, the weight corresponding to each of the key indicators, and the weighted value algorithm; the weight corresponding to each of the key indicators is obtained in advance through the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two of the key indicators compared with each other;

[0012] If the key information evaluation score corresponding to the text information is greater than the scoring threshold, the text information is determined to be key information.

[0013] Preferably, the method of using a specific keyword or fuzzy keyword in a preset keyword configuration file as a search term parameter of a python crawler to crawl a target website to obtain webpage content includes:

[0014] Call the gevent library or concurrent library to build a distributed crawler system;

[0015] The specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain web page content.

[0016] Preferably, the webpage content includes: HTML message; extracting text information from the webpage content includes:

[0017] If the HTML message contains plain text information, the Beautifulsoup tool is used to extract tag data from the HTML message to obtain text information;

[0018] If the HTML message contains non-plaintext information, determining whether the non-plaintext information is an output result using a picture rendering method;

[0019] If yes, extract the image link from the HTML message;

[0020] Based on the image link and the selenium tool, a dynamic rendering image is obtained;

[0021] The text information is extracted from the dynamically rendered image using text recognition technology.

[0022] Preferably, extracting the image link from the HTML message includes:

[0023] If the non-plaintext information is output by splicing and rendering multiple pages of pictures, the canvas tag content in the HTML message is parsed to obtain the picture link format and multiple real-time links;

[0024] Based on the picture link format, multiple real-time links are spliced ​​to obtain a picture link.

[0025] Preferably, the method further comprises:

[0026] Saving the text information into a target file;

[0027] If the text information is determined to be key information, marking the text information in the target file;

[0028] Obtaining the user's review result for the target file;

[0029] Based on the audit results, the weights corresponding to the key indicators are adjusted.

[0030] A second aspect of an embodiment of the present invention discloses a key information search device, the device comprising:

[0031] A crawling unit is used to use specific keywords or fuzzy keywords in a preset keyword configuration file as search term parameters of a python crawler to crawl a target website to obtain web page content;

[0032] An extraction unit, used to extract text information from the webpage content;

[0033] A determination unit, used to determine whether the webpage content is crawled through the specific keyword;

[0034] A first scoring unit is configured to match the text information with the specific keyword using a regular matching formula to obtain a key information evaluation score corresponding to the text information if the webpage content is crawled using the specific keyword;

[0035] A second scoring unit is used to identify key indicator values ​​corresponding to a plurality of preset key indicators from the text information if the webpage content is not obtained by crawling the specific keywords, score each of the key indicators using the preset scoring rules and each of the key indicator values, obtain an initial score for each of the key indicators, and calculate the key information evaluation score corresponding to the text information based on the initial score of each of the key indicators, the weight corresponding to each of the key indicators, and the weighted value algorithm; the weight corresponding to each of the key indicators is obtained in advance by the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two of the key indicators compared with each other;

[0036] A determination unit is used to determine that the text information is key information if the key information evaluation score corresponding to the text information is greater than a scoring threshold.

[0037] Preferably, the crawling unit is specifically used for:

[0038] Call the gevent library or concurrent library to build a distributed crawler system;

[0039] The specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain web page content.

[0040] Preferably, the webpage content includes: HTML message; the extraction unit includes:

[0041] A first extraction subunit is used to extract tag data from the HTML message using a Beautifulsoup tool to obtain text information if the HTML message contains plain text information;

[0042] The second extraction subunit is used to determine whether the non-plaintext information is an output result of image rendering if the HTML message contains non-plaintext information; if so, extract the image link from the HTML message; obtain a dynamic rendering image based on the image link and the selenium tool; and extract text information from the dynamic rendering image using text recognition technology.

[0043] Preferably, the second extraction subunit for extracting the image link from the HTML message is specifically used for:

[0044] If the non-plaintext information is output by splicing and rendering multiple pages of pictures, the canvas tag content in the HTML message is parsed to obtain the picture link format and multiple real-time links;

[0045] Based on the picture link format, multiple real-time links are spliced ​​to obtain a picture link.

[0046] Preferably, the device further comprises:

[0047] An adjustment unit is used to save the text information into a target file; if the text information is determined to be key information, the text information is marked in the target file; obtain the user's review result for the target file; based on the review result, adjust the weight corresponding to each of the key indicators.

[0048] Based on a method and device for searching key information provided by the above-mentioned embodiment of the present invention, a specific keyword or fuzzy keyword in a preset keyword configuration file is used as a search term parameter of a python crawler to crawl a target website to obtain web page content; text information is extracted from the web page content; it is determined whether the web page content is crawled through the specific keyword; if so, the text information is matched with the specific keyword using a regular matching formula to obtain a key information evaluation score corresponding to the text information; if not, key indicator values ​​corresponding to multiple preset key indicators are identified from the text information, and each of the key indicators is scored using a preset scoring rule and each of the key indicator values ​​to obtain an initial score for each of the key indicators; based on the initial score of each of the key indicators, the weight corresponding to each of the key indicators and the weighted value algorithm, the key information evaluation score corresponding to the text information is calculated; the weight corresponding to each of the key indicators is obtained in advance through the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two of the key indicators compared with each other; if the key information evaluation score corresponding to the text information is greater than the scoring threshold, the text information is determined to be key information. In this solution, crawlers are used to batch acquire web page content, judge and score possible key information, and determine the information with a key information evaluation score greater than a scoring threshold as key information, thereby achieving the purpose of improving efficiency and continuous search for key information. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flowchart of a method for searching key information disclosed in an embodiment of the present invention;

[0051] Figure 2 A structural diagram of a key information search device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0054] First, the technical terms appearing in this application are explained as follows:

[0055] Key information refers specifically to information that is of great value to enterprises or individuals or that is prohibited from appearing on the Internet and must be strictly prevented from being leaked.

[0056] Taking the technology line of the banking industry as an example, key enterprise information includes the source code of the bank's production application system, demand analysis and related design documents, database documents, system architecture and network architecture solutions, implementation processes, production application configuration parameters, etc. This type of key information can also be called sensitive enterprise information. Once leaked, it will be analyzed and used by lawbreakers to attack and infringe on the bank system by exploiting loopholes, using social engineering, network penetration, etc., in order to seek illegal profits. This will cause great damage to the bank's safe operation, cause customer funds to suffer losses, and have a negative impact on the company's reputation.

[0057] Personal key information refers to information related to personal identity or activities. Taking the banking industry as an example, since a large amount of customer key information is processed and stored, once personal customer information is leaked or illegally used, it will lead to infringement of the customer's personal dignity, personal and property safety. Personal key information includes but is not limited to the customer's name, ID number, contact number and address, financial account, password and biometric information, transaction details, whereabouts, etc.

[0058] Python crawler, also known as web crawler, is a program or script that automatically crawls Internet information according to certain rules. Python is also a mainstream web crawler programming language due to its easy configuration and rich API packages and third-party packages. The Python crawler architecture mainly consists of five parts, namely scheduler, URL manager, web page downloader, web page parser, and application (crawled valuable data).

[0059] ‌‌Selenium is an open source automated testing tool for Web applications. It interacts with browser drivers (such as ChromeDriver) through the WebDriver protocol. It can be used to simulate the interaction between users and Web browsers and supports multi-platform, multi-browser and multi-language automated testing. Python can load the selenium library to handle complex interactive web pages, capture dynamically generated data, and handle AJAX and JavaScript rendering content commonly found in modern web applications.

[0060] PaddleOCR is an OCR tool library based on PaddlePaddle. It includes an ultra-lightweight Chinese OCR with a total model of only 8.6M. A single model supports Chinese and English number combination recognition, vertical text recognition, and long text recognition. It also supports multiple text detection and text recognition training algorithms, and can be loaded and used in Python.

[0061] In the prior art, most mainstream websites have built their own search engines, which support users to search by entering keywords and display content page by page by turning pages. However, only one keyword can be entered at a time, and the display screen can only display about 10 items. It is inefficient and unsustainable to search for key information manually by turning pages on the website.

[0062] Therefore, an embodiment of the present invention discloses a method and device for searching key information. In this solution, a crawler is used to batch acquire web page content, judge and score possible key information, and determine the information with a key information evaluation score greater than a scoring threshold as key information, thereby achieving the purpose of improving efficiency and continuous search for key information.

[0063] like Figure 1 FIG. 1 is a flowchart of a method for searching key information disclosed in an embodiment of the present invention, comprising the following steps:

[0064] Step S101: using specific keywords or fuzzy keywords in a preset keyword configuration file as search term parameters of a python crawler to crawl a target website to obtain web page content.

[0065] It should be noted that for specific keywords, taking the banking industry as an example, for the bank's internal program source code, related design documents, implementation processes and other confidential documents, you can extract more special words from the code and documents and combine them with the code file name (or document file name) as specific keywords.

[0066] For example, for a source code website, if the key information to be searched is program source code, the source code file can be read, and string data that is particularly different from ordinary code files can be selected to create the following keyword array file:

[0067]

[0068] In one embodiment, the above-mentioned specific keywords are used as the search term parameters of the python crawler, and the file type of the source program is used as the search condition parameter of the python crawler to extract the page content of the target website. The page content includes but is not limited to: related code file name, uploader, upload date, source code link url and other information.

[0069] It is understandable that, according to the above rules, a keyword array file can be constructed similarly for the content belonging to key technical information, and a precise positioning search process can be performed on the corresponding document website.

[0070] It should be noted that fuzzy keywords are general and unspecific keywords. Since it is impossible to know exactly which ID card numbers and bank card numbers appear on Internet pages, they are written into the keyword configuration file for specific searches. For such fuzzy keywords, if the bank is investigating customer information that may have been leaked on the Internet, it cannot directly use the above-mentioned specific keyword method.

[0071] The following customer information will be the key information to be searched:

[0072] 1. Customer’s bank account number (usually 16-19 digits).

[0073] 2. Customer ID number (mainly ID card number).

[0074] 3. Customer name.

[0075] 4. Whether the frequency of occurrence of the above information 1 to 3 in a document is greater than 1.

[0076] 5. Customer contact number (based on an 11-digit mobile phone number). This item is not a required item and may not be included in the identification formula mechanism, but it can be used as a reference.

[0077] Among them, special attention is paid to documents on the Internet that contain key personal customer information such as bank card numbers and ID numbers. By applying regular matching formulas to the page content, you can search for continuous 16 to 19-digit strings, because this may be a leaked bank account. In addition, you can also effectively search for strings with a length of 18 digits. If a reasonable year, month, and day digits appear in digits 7-14, it can be considered an ID number. For the judgment rule of the customer's name, you can judge by identifying whether 3-4 Chinese characters appear and whether the first Chinese character is in the Hundred Family Surnames Dictionary.

[0078] Taking the above-mentioned customer information as an example, fuzzy keywords have been constructed to serve as a preliminary basis for investigation of possible bank customer information leakage on the Internet. The following are examples for reference: bank card number, card number, account holder, list, etc.

[0079] Preferably, after determining the specific keywords and the fuzzy keywords, a specific keyword library and a fuzzy keyword library are respectively established, and finally a keyword configuration file is generated based on the specific keyword library and the fuzzy keyword library.

[0080] In step S101, specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the python crawler. The python crawler sends HTTP requests through the requests library and uses the BeautifulSoup library to parse the returned web page content (i.e., HTML message), thereby realizing automatic collection of web page content.

[0081] Specifically, according to the characteristics of the target website (for example, the number of information items displayed on a page after entering the search term, the anti-crawler mechanism set up by the website server for crawlers, etc.), a special User-Agent request header and request message (including cookies after customer login verification, search keywords, page count, etc.) information are constructed, a service request is sent through the Request library, and the returned web page content (that is, HTML message) is parsed using the Beautifulsoup library, thereby realizing the automatic collection of web page content.

[0082] In one embodiment, the gevent library or concurrent library is called to build a distributed crawler system; specific keywords or fuzzy keywords in a preset keyword configuration file are used as search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain web page content.

[0083] It is understandable that the type of target website can be a source code website and an Internet document sharing website. The embodiment of the present invention can search multiple target websites and can build a distributed crawler system for multi-threaded processing by calling the gevent library or the concurrent library to crawl multiple target websites simultaneously, thereby greatly reducing the crawling time and improving the efficiency of searching for key information.

[0084] Step S102: extracting text information from the webpage content.

[0085] In step S102, there are two situations:

[0086] The first case: If the HTML message contains plain text information, use the Beautifulsoup tool to extract tag data from the HTML message to obtain text information.

[0087] Specifically, if the web page content is static content, that is, the relevant content is all plain text information, then by analyzing the returned web page content (that is, HTML message), the tags that appear in the content of interest (such as document title, link, description, text, upload author, upload date, current page number, etc.) can be located, and the tag data can be extracted through Beautifulsoup to obtain the text information.

[0088] It should be noted that static content is also text content in HTML message format. The text output by the web page can be located and obtained through specific tags in the HTML message.

[0089] The second case: if the HTML message contains non-plaintext information, it is determined whether the non-plaintext information is an output result using an image rendering method.

[0090] If so, extract the image link from the HTML message; obtain the dynamic rendering image based on the image link and the selenium tool; and extract text information from the dynamic rendering image using text recognition technology.

[0091] Specifically, if the web page content is dynamic content, that is, it is non-plaintext information, then analyze whether the target website uses image rendering (such as JavaScript) to render the web page content. If the target website outputs rendered images, the text information cannot be directly extracted. Therefore, adjust the web page access mechanism, use the selenium tool to simulate the user's browser access behavior, capture the dynamic rendering images, download them to local storage and processing, that is, use text recognition technology to extract text information from the dynamic rendering images.

[0092] In one embodiment, extracting a picture link from an HTML message includes:

[0093] If the non-plaintext information is output by splicing and rendering multiple pages of images, the canvas tag content in the HTML message is parsed to obtain the image link format and multiple real-time links; based on the image link format, multiple real-time links are spliced ​​to obtain the image link.

[0094] Specifically, if the target website uses a multi-page image display method for the output web page content, it is necessary to conduct an in-depth analysis of the returned HTML message. For example, a website uses the canvas function to return real-time document image data splicing. Then, the real-time link URL of the file image website is obtained through analysis and parsing. After parsing the file image link format and splicing the image link URL, the dynamically rendered image is downloaded to the local computer through the image download method, and text information is extracted from the dynamically rendered image using text recognition technology.

[0095] It should be noted that generally, dynamic web pages refer to target websites with backgrounds, which use Java, VB, VC and database programming in addition to HTML to display results. For example, the address of the picture output on the result page after a website search is not directly found in the return message or HTML message. It is returned through the HTML message. <canvas>The tag content and some related return parameters must be judged and spliced ​​to obtain the specific link address of the output image.

[0096] Generally, high-level websites have anti-crawler mechanisms, and will not allow the return message to be directly read and the link address of the image directly obtained from the message. Crawler developers need to analyze and study each website. In addition, the target website will be frequently upgraded, and the crawler program may also need to be adapted and modified accordingly to maintain the original crawling function.

[0097] Preferably, the specific process of extracting text information from dynamically rendered images using text recognition technology is: use the PaddleOCR model to initialize the OCR tool, traverse the dynamically rendered image content, grab the text therein, and thus identify and extract the textual information displayed on the web page. This method is particularly effective for table images.

[0098] Step S103: Determine whether the webpage content is crawled through specific keywords; if so, execute step S104; if not, execute step S105.

[0099] In step S103, it is distinguished whether the webpage content is obtained by crawling with specific keywords or crawling with fuzzy keywords, and different scoring methods are then adopted.

[0100] It should be noted that after executing step S104 or S105, step S106 is directly executed.

[0101] Step S104: using a regular matching formula, matching the text information with specific keywords to obtain a key information evaluation score corresponding to the text information.

[0102] In step S104, the regular matching formula is a powerful tool for matching and manipulating text, which is a pattern consisting of a series of characters and special characters, used to describe the text pattern to be matched. The regular matching formula can find, replace, extract and verify specific patterns in the text.

[0103] In the specific implementation of step S104, a regular matching formula is used to match specific keywords and text information, wherein the text information includes: document content and document name.

[0104] Specifically, the document content and the document name are matched respectively through the regular matching formula: the specific keyword and the document content are matched through the regular matching formula, and the specific keyword and the document name are matched through the regular matching formula.

[0105] If only one of the document content match and the document name match passes, the key information evaluation score is: KPI = 0.4.

[0106] If both are matched, the key information evaluation score KPI = 1.0.

[0107] Step S105: Identify key indicator values ​​corresponding to multiple preset key indicators from the text information, score each key indicator using the preset scoring rules and each key indicator value, and obtain an initial score for each key indicator; based on the initial score of each key indicator, the weight corresponding to each key indicator and the weighted value algorithm, calculate the key information evaluation score corresponding to the text information.

[0108] It is understandable that, for example, the fuzzy keyword can be the head of household, so the key indicator value corresponding to the key indicator "name" can be identified in the text information. For another example, the key indicator value corresponding to the key indicator "bank account number" may be identified in the text information searched using the fuzzy keywords card number, account number, etc. The specific identification method can be: search for a continuous string of 16 to 19 digits, because this may be a leaked bank account. In addition, a string of 18 digits can also be effectively searched. If a reasonable year, month, and day digits appear in 7-14 digits, it can be considered as an ID card number. For the judgment rule of the customer's name, it can be judged by identifying whether 3-4 Chinese characters appear and the first Chinese character is in the Hundred Family Surnames dictionary.

[0109] The weight corresponding to each key indicator is obtained in advance through the AHP hierarchical analysis method and a preset importance matrix. The importance matrix is ​​used to reflect the importance of any two key indicators compared with each other.

[0110] Exemplarily, the importance matrix is ​​as follows:

[0111]

[0112] The importance value reflects the importance of any two key indicators compared to each other. For example, the importance value of the name and the ID number compared to each other is 1 / 4, which means that the importance of the name is 1 / 4 of the ID number, that is, compared with the name leakage, the ID number leakage is more concerned because there are many people with the same name, but the ID number is less repeated, so it is more accurate to locate the customer.

[0113] The weight column is the weight corresponding to each key indicator obtained through the AHP hierarchical analysis method and the above importance matrix.

[0114] In one embodiment, a new importance matrix may be constructed for characteristic key indicators (such as transaction amount, transaction date, balance, etc.) based on the content characteristics of key information such as account flow.

[0115] It should be noted that the AHP is a qualitative and quantitative analysis decision-making method, which is widely used in system engineering and decision analysis. It decomposes the elements related to decision-making into levels such as goals, criteria, and plans. By constructing a judgment matrix and performing consistency tests, the weights of each plan are finally determined to select the optimal plan. The weights corresponding to each key indicator obtained through the AHP hierarchical analysis method and the above-mentioned importance matrix are as follows:

[0116] #This is an example program;

[0117] #comparison_matrix is ​​the importance matrix of indicators, which needs to be scored subjectively;

[0118] #The first row, second column to the nth row, nth column correspond to the factors that need to be considered, denoted as C1, C2, C3,...;

[0119] #The i-th row and j-th column represent the importance of Ci to Cj, and the scores are 1, 3, 5, 7, and 9. 1 represents that Ci and Cj are equally important, and 9 represents that Ci is extremely more important than Cj. If Ci is not as important as Cj, then 1 / 3, 1 / 5, 1 / 7, etc. correspond to the importance of Cj to Ci. Position (i, j) and position (j, i) must be reciprocals of each other because they represent the same two key indicators for comparison;

[0120] #After the pairwise comparison is completed, just input the matrix;

[0121] comparison_matrix = np.array([

[0122] [1, 8, 6, 5],

[0123] [1 / 8, 1, 2, 1],

[0124] [1 / 6, 1 / 2, 1, 2],

[0125] [1 / 5, 1 / 5, 1 / 2, 1] ])

[0127] weights = ahp_weight_calculation(comparison_matrix)

[0128] print("Weight calculated by AHP:")

[0129] for criterion, weight in enumerate(weights):

[0130] print(f'{criterion + 1}: {weight:.4f}')

[0131] Finally, we get:

[0132] The weight C1 of the bank account is 0.6204, the weight C2 of the ID number is 0.2002, the weight C3 of the name is 0.1016, indicating that the importance of C3 is lower than that of C2, and the weight C4 of the frequency of occurrence is 0.0778.

[0133] In step S105, each key indicator is scored using the preset scoring rules and each key indicator value, and the specific implementation process of obtaining the initial score of each key indicator is as follows:

[0134] First, assume that P1: bank account matching score (0-10), P2: ID number matching score (0-10), P3: name matching score (0-10), P4: frequency of repetition of any two items of bank account, ID number, and name (0-10).

[0135] Then, the default value of P1 to P4 is 0 points, and the initial score is assigned if the following situations occur:

[0136] P1:

[0137]

[0138] P2:

[0139]

[0140] P3:

[0141]

[0142] P4:

[0143] If any two key indicators of account number + ID card + name appear in two or more records, 10 points will be given, 2 points will be given for one record, and 0 points will be given if they do not appear.

[0144] In step S105, the formula corresponding to the weighted value algorithm is as follows:

[0145] KPI=∑ Px*cx

[0146] Among them, KPI represents the key information evaluation score; P represents the initial score; c represents the weight; x is a positive integer, and the value range of x is 1 to the number of key indicators.

[0147] Taking the four key indicators listed above as an example, the value range of x is 1 to 4; P1 to P4 represent the initial scores corresponding to the above key indicators; c1 to c4 represent the weights corresponding to the four key indicators respectively, then: KPI=(P1*c1+P2*c2+P3*c3+P4*c4) / 10.

[0148] It should be noted that in the above example of initial scoring, the scoring range of the matching score is enlarged by 10 times for the convenience of calculation. Normally, the initial score should range from 0 to 1, so the overall score is then divided by 10 to restore it. If the matching score is between (0-1), it does not need to be divided by 10.

[0149] Step S106: If the key information evaluation score corresponding to the text information is greater than the scoring threshold, the text information is determined to be key information.

[0150] In step S106, text information with a key information evaluation score KPI>0.5 is key information.

[0151] In one embodiment, the text information is saved in a target file; if the text information is determined to be key information, the text information is marked in the target file; the user's review result for the target file is obtained; based on the review result, the weight corresponding to each key indicator is adjusted.

[0152] Specifically, the text information (including keywords, document title, document link, document text content, upload author, upload date, matching degree, etc.) of "suspected leakage" (key information evaluation score KPI <0.5) and "highly suspected leakage" (key information evaluation score KPI>0.5) is output, organized and stored in the result Excel file, and the key information is marked for subsequent manual review. Based on the review results, the corresponding weights of each key indicator are optimized to more accurately locate the key information in the Internet crawler search results.

[0153] Based on the method for searching key information disclosed in the above-mentioned embodiment of the present invention, in this scheme, a crawler is used to batch obtain web page content, judge and score possible key information, and determine the information with a key information evaluation score greater than a scoring threshold as key information, thereby achieving the purpose of improving efficiency and continuous search for key information.

[0154] Corresponding to the method for searching key information disclosed in the above embodiment of the present invention, Figure 2 As shown, it is a structural diagram of a key information search device disclosed in an embodiment of the present invention, and the device includes: a crawling unit 201, an extraction unit 202, a judgment unit 203, a first scoring unit 204, a second scoring unit 205 and a determination unit 206.

[0155] The crawling unit 201 is used to use specific keywords or fuzzy keywords in a preset keyword configuration file as search term parameters of the python crawler to crawl the target website to obtain web page content.

[0156] In one embodiment, the crawling unit 201 is specifically used for:

[0157] Call the gevent library or concurrent library to build a distributed crawler system;

[0158] The specific keywords or fuzzy keywords in the preset keyword configuration file are used as the search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain the web page content.

[0159] The extraction unit 202 is used to extract text information from the webpage content.

[0160] In one embodiment, the webpage content includes: HTML message; the extraction unit includes:

[0161] The first extraction subunit is used to extract tag data from the HTML message using the Beautifulsoup tool to obtain text information if the HTML message contains plain text information;

[0162] The second extraction subunit is used to determine whether the non-plaintext information is the output result of image rendering if the HTML message contains non-plaintext information; if so, extract the image link from the HTML message; obtain the dynamic rendering image based on the image link and the selenium tool; use text recognition technology to extract text information from the dynamic rendering image.

[0163] In one embodiment, the second extraction subunit for extracting the image link from the HTML message is specifically used to:

[0164] If the non-plaintext information is output by splicing and rendering multiple pages of images, the canvas tag content in the HTML message is parsed to obtain the image link format and multiple real-time links;

[0165] Based on the image link format, multiple real-time links are spliced ​​to obtain the image link.

[0166] The judgment unit 203 is used to judge whether the webpage content is obtained by crawling through specific keywords.

[0167] The first scoring unit 204 is used to match the text information with the specific keyword using a regular matching formula if the webpage content is crawled through a specific keyword, so as to obtain a key information evaluation score corresponding to the text information.

[0168] The second scoring unit 205 is used to identify key indicator values ​​corresponding to multiple preset key indicators from the text information if the web page content is not obtained by crawling through specific keywords, and score each key indicator using the preset scoring rules and each key indicator value to obtain an initial score for each key indicator. Based on the initial score of each key indicator, the weight corresponding to each key indicator and the weighted value algorithm, the key information evaluation score corresponding to the text information is calculated; the weight corresponding to each key indicator is obtained in advance through the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two key indicators compared with each other.

[0169] The determination unit 206 is configured to determine that the text information is key information if the key information evaluation score corresponding to the text information is greater than a scoring threshold.

[0170] In one embodiment, the device further comprises:

[0171] The adjustment unit is used to save the text information into the target file; if the text information is determined to be key information, the text information is marked in the target file; obtain the user's review result for the target file; based on the review result, adjust the weight corresponding to each key indicator.

[0172] Based on the key information search device disclosed in the above-mentioned embodiment of the present invention, in this scheme, a crawler is used to batch obtain web page content, judge and score possible key information, and determine the information with a key information evaluation score greater than a scoring threshold as key information, thereby achieving the purpose of improving efficiency and continuous search for key information.

[0173] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0174] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0175] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / canvas>

Claims

1. A method for searching key information, characterized in that: The method comprises: Use specific keywords or fuzzy keywords in the preset keyword configuration file as search term parameters of the python crawler to crawl the target website to obtain web page content; Extracting text information from the webpage content; Determining whether the webpage content is obtained by crawling through the specific keyword; If yes, the text information is matched with the specific keyword by using a regular matching formula to obtain a key information evaluation score corresponding to the text information; If not, identify key indicator values ​​corresponding to a plurality of preset key indicators from the text information, score each of the key indicators using the preset scoring rules and each of the key indicator values, obtain an initial score for each of the key indicators, and calculate the key information evaluation score corresponding to the text information based on the initial score of each of the key indicators, the weight corresponding to each of the key indicators, and the weighted value algorithm; the weight corresponding to each of the key indicators is obtained in advance through the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two of the key indicators compared with each other; If the key information evaluation score corresponding to the text information is greater than the scoring threshold, the text information is determined to be key information.

2. The method according to claim 1, characterized in that The specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the python crawler to crawl the target website to obtain web page content, including: Call the gevent library or concurrent library to build a distributed crawler system; The specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain web page content.

3. The method according to claim 1, characterized in that The webpage content includes: HTML message; extracting text information from the webpage content includes: If the HTML message contains plain text information, the Beautifulsoup tool is used to extract tag data from the HTML message to obtain text information; If the HTML message contains non-plaintext information, determining whether the non-plaintext information is an output result using a picture rendering method; If yes, extract the image link from the HTML message; Based on the image link and the selenium tool, a dynamic rendering image is obtained; The text information is extracted from the dynamically rendered image using text recognition technology.

4. The method according to claim 3, characterized in that The step of extracting the image link from the HTML message includes: If the non-plaintext information is output by splicing and rendering multiple pages of pictures, the canvas tag content in the HTML message is parsed to obtain the picture link format and multiple real-time links; Based on the picture link format, multiple real-time links are spliced ​​to obtain a picture link.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Saving the text information into a target file; If the text information is determined to be key information, marking the text information in the target file; Obtaining the user's review result for the target file; Based on the audit results, the weights corresponding to the key indicators are adjusted.

6. A key information search device, characterized in that: The device comprises: A crawling unit is used to use specific keywords or fuzzy keywords in a preset keyword configuration file as search term parameters of a python crawler to crawl a target website to obtain web page content; An extraction unit, used to extract text information from the webpage content; A determination unit, used to determine whether the webpage content is crawled through the specific keyword; A first scoring unit is configured to match the text information with the specific keyword using a regular matching formula to obtain a key information evaluation score corresponding to the text information if the webpage content is crawled using the specific keyword; A second scoring unit is used to identify key indicator values ​​corresponding to a plurality of preset key indicators from the text information if the webpage content is not obtained by crawling the specific keywords, score each of the key indicators using the preset scoring rules and each of the key indicator values, obtain an initial score for each of the key indicators, and calculate the key information evaluation score corresponding to the text information based on the initial score of each of the key indicators, the weight corresponding to each of the key indicators, and the weighted value algorithm; the weight corresponding to each of the key indicators is obtained in advance by the AHP hierarchical analysis method and a preset importance matrix, and the importance matrix is ​​used to reflect the importance of any two of the key indicators compared with each other; A determination unit is used to determine that the text information is key information if the key information evaluation score corresponding to the text information is greater than a scoring threshold.

7. The device according to claim 6, characterized in that The crawling unit is specifically used for: Call the gevent library or concurrent library to build a distributed crawler system; The specific keywords or fuzzy keywords in the preset keyword configuration file are used as search term parameters of the distributed crawler system, so that the distributed crawler system crawls the target website in a multi-threaded manner to obtain web page content.

8. The device according to claim 6, characterized in that The webpage content includes: HTML message; the extraction unit includes: A first extraction subunit is used to extract tag data from the HTML message using a Beautifulsoup tool to obtain text information if the HTML message contains plain text information; The second extraction subunit is used to determine whether the non-plaintext information is an output result of image rendering if the HTML message contains non-plaintext information; if so, extract the image link from the HTML message; obtain a dynamic rendering image based on the image link and the selenium tool; and extract text information from the dynamic rendering image using text recognition technology.

9. The device according to claim 8, characterized in that The second extraction subunit for extracting the image link from the HTML message is specifically used for: If the non-plaintext information is output by splicing and rendering multiple pages of pictures, the canvas tag content in the HTML message is parsed to obtain the picture link format and multiple real-time links; Based on the picture link format, multiple real-time links are spliced ​​to obtain a picture link.

10. The device according to any one of claims 6 to 9, characterized in that: The device also includes: An adjustment unit is used to save the text information into a target file; if the text information is determined to be key information, the text information is marked in the target file; obtain the user's review result for the target file; based on the review result, adjust the weight corresponding to each of the key indicators.

Images