A method and system for automated collection and extraction of information on innovation activities of enterprises

Abstract

The application provides a kind of automation collection and extraction enterprise innovation activity information method and system, by S1 based on TF-IDF keyword extraction algorithm, the feature word library of innovation activity information is established;S2 scans the webpage structure of target website, is positioned to target webpage and obtains webpage source code;S3 is divided into block according to HTML label, obtains the text block to be identified;S4. According to the feature word library, the correlation of the text block to be identified and the target information is calculated, and the innovation activity content is obtained according to the correlation;S5. Innovation activity content is grabbed using crawler tool;S6. The quality evaluation and value classification of the target content grabbed are carried out;S7 the high value target content is stored in enterprise information base, and the low value target content is stored in standby contrast learning base, realizes the efficient and accurate grabbing of enterprise innovation activity information, and improves the utilization rate of these information.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise data crawling, and in particular to a method and system for automatically collecting and extracting enterprise innovation activity information. Background Art

[0002] Web data scraping, also known as web crawler technology, is a critical tool widely used in fields such as big data analysis, information retrieval, and market research. Enterprise data scraping involves using web crawler technology to extract target information from a company's official website for subsequent use.

[0003] There are many methods for capturing enterprise data on the market. The traditional method is to write programs to traverse the content on the target web page according to preset rules or algorithms and capture the required data from it. However, most of them directly capture the entire page or simply use keywords to capture. This results in complex and mixed content. The captured data needs to undergo complex data cleaning, otherwise it is difficult to use it effectively or directly.

[0004] Furthermore, target information, such as company profiles and product information, is typically not found on the homepage of a company's official website, but rather on secondary pages. Therefore, crawling requires first accessing the official website and then using the navigation bar to navigate to the target webpage. Furthermore, the target webpage often contains a significant amount of irrelevant content, such as navigational and copyright text. Therefore, how to crawl concise, high-quality, and effective company target information has become a pressing technical challenge.

[0005] Innovation activities are becoming more and more active, but information related to innovation activities has not been collected and organized, and is mixed in with a large amount of other information. As a result, governments at all levels, various entrepreneurial parks, and various innovation entities are unable to grasp and accurately understand innovation activity information in a timely manner, thus affecting the implementation of various innovation activities and obtaining timely and effective support. Summary of the Invention

[0006] The present invention solves the problem in the prior art that the content of keyword capture of enterprise innovation activity information is complicated, of varying quality, complex to clean, and difficult to effectively or directly utilize. It proposes a method and system for automatically collecting and extracting enterprise innovation activity information, which achieves the conciseness of keyword capture content, high quality, simple cleaning, and effective and direct utilization.

[0007] The present invention provides the following specific solutions:

[0008] A method for automatically collecting and extracting enterprise innovation activity information, characterized by comprising:

[0009] S1. Establish a feature vocabulary for innovation activity information: This refers to target information such as company profile information, product and service information, technological innovation information, scenario innovation, implementation case studies, honorary awards, and government projects. Using the TF-IDF keyword extraction algorithm, extract several keywords for each target information from a sample set. These keywords are then weighted to generate multiple feature vocabulary for each target information in the innovation activity information.

[0010] S2 target page positioning: scanning the target website's web page structure, using the keyword of each feature of the vocabulary to locate the target page in the web page structure, obtain the target page source code;

[0011] S3. Blocking of web page text content: Remove noise from the web page source code, block it according to HTML tags, calculate the text aggregation of each block of content based on text and symbol density algorithm, and obtain the text block to be identified;

[0012] S4. Information Identification: Based on each of the feature lexicons, calculate the relevance between the text block to be identified and the target information, and filter the target content corresponding to each feature lexicon based on the relevance, i.e., the target innovation activity content;

[0013] S5. Quality Assessment: Perform quality assessment and value classification on the captured target content, and manually review the quality assessment results; the value classification includes high value and low value;

[0014] S6. Storing in a database: storing the target content of high value in an enterprise information database, storing the target content of low value in a backup reference learning database, and finally obtaining the target content of the multiple feature word databases to form a complete enterprise information database.

[0015] Preferably, the TF-IDF-based keyword extraction algorithm in S1 extracts several keywords for each target information from a set of samples, and performs weight processing on the keywords to obtain a feature word library for each target information, including:

[0016] S11. A keyword extraction algorithm based on TF-IDF is used, using an open source word segmentation component in Python, to extract keywords of the target information from a set of standard samples of the target information;

[0017] S12. Process and sort the keywords, retaining the top 50 keywords with the highest number of occurrences;

[0018] S13. After manual sorting, the keywords that are relevant to the target information are weighted up, and the irrelevant keywords are weighted down, to obtain a feature word library of the target information.

[0019] Preferably, the step of locating a target webpage in the webpage structure and obtaining the webpage source code of the target webpage in S2 includes:

[0020] S21. Establish the navigation title thesaurus of the target page;

[0021] S22. Capture all navigation titles on the web page structure, compare the navigation title with the navigation title vocabulary, identify the navigation where the target page is located, and extract the corresponding web link;

[0022] S23. Obtain the webpage source code of the target webpage through the webpage link.

[0023] Preferably, the step of removing noise from the webpage source code and dividing the source code into blocks according to HTML tags in S3 includes:

[0024] S31. After removing the noise from the web page source code, a DOM tree of the web page structure is obtained. The noise includes the header, footer, navigator, and copyright.

[0025] S32 traverses all subtags under the body tag in the DOM tree, establishes a map structure to store the subtag metadata and all text under the subtag; if there is a second subtag under the subtag, extract the second subtag content in the same way, and so on, until all tags are extracted;

[0026] S33. All texts under each of the sub-tags constitute a text block.

[0027] Preferably, the step of calculating the text aggregation of each block of content based on a text and symbol density algorithm in S3 to obtain a text block to be identified includes:

[0028] S34. Calculate the text aggregation of each text block using a text and symbol density algorithm to obtain a text aggregation value Score1 for each text block;

[0029] S35. Set a text aggregation threshold, select all text blocks whose aggregation value Score1 is greater than the text aggregation threshold, and obtain the text blocks to be identified.

[0030] Preferably, the text and symbol density algorithm in S34 includes:

[0031] The formula for calculating text aggregation value is:

[0032] Score1=log(sd)*tdi*log10(pnumi+2)*log(sbdi) (1)

[0033] Among them, tdi is the text density, sd is the standard deviation of tdi, pnumi is the number of p labels of the node, and sbdi is the text symbol density;

[0034] The above SBDI (text symbol density) calculation formula is:

[0035] sbdi=(ti-lti) / (sbi+1) (2)

[0036] Where sbi is the number of symbols in the text, ti is the number of strings in node i, that is, the length of the text in node i; lti is the number of strings with links in node i, that is, the length of the text with links in node i;

[0037] The above tdi (text density) calculation formula is:

[0038] tdi=(ti-lti) / (tgi-ltgi) (3)

[0039] Among them, ti is the number of strings in node i, that is, the length of the text of node i; lti is the number of linked strings in node i, that is, the length of the text with links in node i; tgi is the number of tags in node i, and ltgi is the number of linked tags in node i.

[0040] Preferably, the step of calculating the relevance between the text block to be identified and the target information based on the feature word library in S4 to obtain the target content includes:

[0041] S41. Calculate the relevance Score2 of each text block to be identified and the feature vocabulary using the feature vocabulary;

[0042] S42. Set a relevance threshold, select all text blocks whose relevance Score2 is greater than the relevance threshold, and obtain the target content.

[0043] Preferably, the step of calculating the relevance Score2 of each of the to-be-recognized text blocks and the feature word library using the feature word library in S41 includes:

[0044] The correlation calculation formula is:

[0045] Score2=μ(x)n*normalized_weigth (4)

[0046] Wherein, n is the number of times the keyword appears in the text block to be identified, and normalized_weight is the normalized weight, that is, the new weight obtained after normalizing the weights of all keywords in the feature vocabulary using the python numpy library.

[0047] Preferably, the quality evaluation and value classification of the captured target content in S5 includes:

[0048] The target content is judged in descending order according to the relevance Score2, and whether the text block contains the full name or the abbreviation of the enterprise. If it does, the text block is evaluated as high value, and the remaining text blocks are evaluated as low value, and the judgment is completed; if it does not, the next text block is judged until all text blocks are judged.

[0049] Preferably, the manual review of the quality evaluation results in S5 includes:

[0050] During the manual review process, if there is a deviation in the quality evaluation, the feature vocabulary of each target information in the enterprise innovation activity information described in S1 will be recreated, and an artificial logical target result will be generated. The artificial logical target result will be used for machine learning to optimize the crawling algorithm.

[0051] The present invention also provides an automated collection and extraction system for enterprise innovation activity information formed by the above method, characterized by comprising: a feature word library creation module, a web page structure scanning module, a text segmentation module, an information recognition module, a data capture module, a quality evaluation module and a data storage module;

[0052] The feature word library creation module extracts a number of keywords from a standard sample of innovation activity information based on the TF-IDF keyword extraction algorithm to obtain a feature word library of the innovation activity information; the innovation activity information refers to target information such as the company's profile information, product and service information, technological innovation information, scenario innovation, implementation case studies, honorary awards, and government projects;

[0053] The webpage structure scanning module scans the webpage structure of the target website, locates the target webpage in the webpage structure using the keywords in each of the characteristic lexicons, and obtains the webpage source code of the target webpage, including using a crawler tool to crawl the target information;

[0054] The text segmentation module removes noise from the webpage source code, segments the text content into blocks according to HTML tags, and calculates the text aggregation degree of each block of content based on a text and symbol density algorithm to obtain the text block to be identified;

[0055] The information recognition module calculates the relevance of the text block to be recognized based on each characteristic word library of the innovation activity information, and obtains the target content corresponding to each characteristic word library, that is, the target innovation activity content;

[0056] The quality evaluation module performs quality evaluation and value classification on the captured target content and manually reviews the quality evaluation results;

[0057] The data storage module stores the high-value target content in an enterprise information library and stores the low-value target content in a standby comparison learning library.

[0058] Beneficial effects of the present invention:

[0059] The present invention provides a method and system for automatically collecting and extracting enterprise innovation activity information. The method comprises the following steps: S1 uses a TF-IDF keyword extraction algorithm to extract keywords of each target information in the innovation activity information and performs weight processing, accurately reflects the characteristics of the target information, and establishes an accurate feature word library for each target information. S2 uses the keywords in the feature word library to locate the target web page and obtain the web page source code. S3 divides the web page content into blocks according to HTML tags, reasonably splits the web page DOM tree structure, calculates the text aggregation degree of each block of content based on a text and symbol density algorithm, and obtains a text block to be identified. Since the innovation activity information has a high text aggregation, the content presenting the innovation activity information more accurately can be obtained by calculating the text aggregation degree. S4 calculates the relevance between the text block and the target information based on the feature word library, and obtains the corresponding target content, i.e., the target innovation activity content. S5 performs quality evaluation and value classification on the captured target content, and manually reviews the quality evaluation results. S6 stores the target content in a database. Scanning the web page structure of the target website can accurately identify the target web page, and filter out large text blocks based on text and symbol density algorithms, effectively removing the interference of irrelevant content; by establishing a feature vocabulary of enterprise innovation activity information, and using the relevance algorithm to calculate the correlation between text blocks and feature vocabulary, the required crawled content can be accurately identified, thereby improving the quality of the crawled content; quality evaluation further confirms the accuracy of the crawled content, and achieves the conciseness, high quality and effectiveness of the target information. The innovation activity information has a variety of target information, and multiple feature vocabulary libraries have been established to fully and comprehensively reflect the innovation activities.

[0060] In particular, the captured content is evaluated as high-value or low-value, and the high-value content is stored in the enterprise information database, while the low-value content is stored in the backup control learning library as an audit control group. After subsequent manual processing, new artificial logical target results are generated for machine learning comparison, thereby continuously improving the capture quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a method for automatically collecting and extracting enterprise innovation activity information provided by the present invention.

[0062] Figure 2 This is a flow chart of a method for automatically collecting and extracting enterprise innovation activity information provided by the present invention.

[0063] Figure 3This is an architectural diagram of a system for automatically collecting and extracting enterprise innovation activity information provided by the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0065] Example 1:

[0066] like Figure 1 As shown, a method for automatically collecting and extracting enterprise innovation activity information includes:

[0067] S1. Establish a feature vocabulary for innovation activity information: This refers to target information such as a company's profile, product and service information, technological innovation information, scenario innovation, implementation cases, honors and awards, and government projects. Using the TF-IDF keyword extraction algorithm, extract several keywords for each target information from a sample. These keywords are then weighted to generate multiple feature vocabulary for each target information in the innovation activity information.

[0068] S2 target page positioning: scanning the target website's web page structure, using the keyword of each feature of the vocabulary to locate the target page in the web page structure, obtain the target page source code;

[0069] S3. Blocking of web page text content: Remove noise from the web page source code, block it according to HTML tags, calculate the text aggregation of each block of content based on text and symbol density algorithm, and obtain the text block to be identified;

[0070] S4. Information Identification: Based on each of the feature lexicons, calculate the relevance between the text block to be identified and the target information, and filter the target content corresponding to each feature lexicon based on the relevance, i.e., the target innovation activity content;

[0071] S5. Quality Assessment: Perform quality assessment and value classification on the captured target content, and manually review the quality assessment results; the value classification includes high value and low value;

[0072] S6. Storing in a database: storing the target content of high value in an enterprise information database, storing the target content of low value in a backup reference learning database, and finally obtaining the target content of the multiple feature word databases to form a complete enterprise information database.

[0073] By establishing a characteristic vocabulary of enterprise innovation activity information and scanning the web page structure of the target website, we can accurately identify the target web page, filter out large text blocks based on text and symbol density algorithms, calculate the relevance of text blocks and the vocabulary, accurately identify the required content to be captured, and further confirm the accuracy of the captured content through quality evaluation, thus achieving efficient and accurate capture of enterprise innovation activity information.

[0074] Specifically, the TF-IDF-based keyword extraction algorithm described in S1 extracts several keywords for each target information from a set of samples, and performs weight processing on the keywords to obtain a feature word library for each target information, including:

[0075] S11. A keyword extraction algorithm based on TF-IDF is used, using an open source word segmentation component in Python, to extract keywords of the target information from a set of standard samples of the target information;

[0076] S12. Process and sort the keywords, retaining the top 50 keywords with the highest number of occurrences;

[0077] S13. After manual sorting, the keywords that are relevant to the target information are weighted up, and the irrelevant keywords are weighted down, to obtain a feature word library of the target information.

[0078] In some embodiments, a feature vocabulary of each target information in the innovation activity information is established. Taking the company profile information as an example of the target information, the keywords involved in the target information in the innovation activity information are extracted from the company profile information of 3,000 standard samples. The word segmentation uses a keyword extraction algorithm based on TF-IDF and is completed using the python open source word segmentation component; the data after the word segmentation of the sample company profile is processed, and the top 50 keywords with the highest number of occurrences are counted. After manual sorting, the most representative keywords are weighted. Of course, in addition to the profile information, other innovation activity information including product service information, technological innovation information, scenario innovation, implementation cases and other target information are also processed, and some irrelevant words are downgraded.

[0079] In some embodiments, as Figure 2As shown in the figure, since there is a high probability that there will be no company profile information on the homepage of the company's official website, many official websites put the company profile information on the secondary page. You need to enter the official website first and then enter the company profile page through the navigation bar. Therefore, first, we need to establish a behavioral feature library, enumerate the navigation titles that may be used for company profiles and product data, such as About Us, Company Profile, Product Information, Product Data, etc., and establish a set of feature word libraries for company profile navigation texts. Company product information and other innovation activity information are also processed; secondly, use Python Selenium technology to capture all the navigation texts on the company's official website, compare them with the feature word library, identify the navigation where the company profile is located, and then extract the corresponding web page link; finally, use the Python Selenium library to obtain the web page source code of the corresponding web page.

[0080] Specifically, the process of removing noise from the webpage source code and dividing the source code into blocks according to HTML tags in S3 includes:

[0081] S31. After removing the noise from the web page source code, a DOM tree of the web page structure is obtained. The noise includes the header, footer, navigator, and copyright.

[0082] S32 traverses all subtags under the body tag in the DOM tree, establishes a map structure to store the subtag metadata and all text under the subtag; if there is a second subtag under the subtag, extract the second subtag content in the same way, and so on, until all tags are extracted;

[0083] S33. All texts under each of the sub-tags constitute a text block.

[0084] Specifically, the text aggregation degree of each block of content is calculated based on the text and symbol density algorithm described in S3 to obtain the text block to be identified, including:

[0085] S34. Calculate the text aggregation of each text block using a text and symbol density algorithm to obtain a text aggregation value Score1 for each text block;

[0086] S35. Set a text aggregation threshold, select all text blocks whose aggregation value Score1 is greater than the text aggregation threshold, and obtain the text blocks to be identified.

[0087] Specifically, the text and symbol density algorithm in S34 includes:

[0088] The formula for calculating text aggregation value is:

[0089] Score1=log(sd)*tdi*log10(pnumi+2)*log(sbdi) (1)

[0090] Among them, tdi is the text density, sd is the standard deviation of tdi, pnumi is the number of p labels of the node, and sbdi is the text symbol density;

[0091] The above SBDI (text symbol density) calculation formula is:

[0092] sbdi=(ti-lti) / (sbi+1) (2)

[0093] Where sbi is the number of symbols in the text, ti is the number of strings in node i, that is, the length of the text in node i; lti is the number of strings with links in node i, that is, the length of the text with links in node i;

[0094] The above tdi (text density) calculation formula is:

[0095] tdi=(ti-lti) / (tgi-ltgi) (3)

[0096] Among them, ti is the number of strings of node i, that is, the length of the text of node i; lti is the number of linked strings of node i, that is, the length of the text with links of node i; tgi is the number of tags of node i, and ltgi is the number of linked tags of node i.

[0097] Specifically, the step S4 of calculating the relevance between the text block to be identified and the target information based on the feature word library to obtain the target content includes:

[0098] S41. Calculate the relevance Score2 of each text block to be identified and the feature vocabulary using the feature vocabulary;

[0099] S42. Set a relevance threshold, select all text blocks whose relevance Score2 is greater than the relevance threshold, and obtain the target content.

[0100] Specifically, the calculation of the relevance Score2 of each of the to-be-recognized text blocks and the feature vocabulary using the feature vocabulary in S41 includes:

[0101] The correlation calculation formula is:

[0102] Score2=∑n*normalized_weigth (4)

[0103] Wherein, n is the number of times the keyword appears in the text block to be identified, and normalized_weight is the normalized weight, that is, the new weight obtained after normalizing the weights of all keywords in the feature vocabulary using the python numpy library.

[0104] Specifically, the quality evaluation and value classification of the captured target content in S5 includes:

[0105] The target content is judged in descending order according to the relevance Score2, and whether the text block contains the full name or the abbreviation of the enterprise. If it does, the text block is evaluated as high value, and the remaining text blocks are evaluated as low value, and the judgment is completed; if it does not, the next text block is judged until all text blocks are judged.

[0106] Specifically, the manual review of the quality evaluation results described in S5 includes:

[0107] During the manual review process, if there is a deviation in the quality evaluation, the feature word library of each target information in the enterprise innovation activity information described in S1 is recreated, and an artificial logical target result is generated. The artificial logical target result is used for machine learning to optimize the crawling algorithm. In some embodiments, when crawling the enterprise profile, after using the Python Selenium library to obtain the webpage source code of the corresponding webpage:

[0108] First, the body of the web page must be identified. Since corporate websites contain a large amount of content that is irrelevant to the company profile, such as navigation, copyright, and other irrelevant text, the body of the web page needs to be divided into blocks. The specific method is: remove noise from the web page source code, generally including the header, footer, navigator, copyright, and other content; construct a DOM tree for the web page after removing noise, traverse all the first-level subtags under the body tag, and establish a map structure to store the tag metadata and all the text under the tag. If there are subtags under the tag, extract the subtag content in the same way, and so on, until all tags are extracted and all the text blocks under the web page are obtained.

[0109] Secondly, the text aggregation of each block of content is calculated based on the text and symbol density algorithm. The aggregation is a float value score1, which means that the larger the score1, the greater the text aggregation of the text block, and the more likely it is a long text, which means that the text block is valuable. A threshold is set for the text aggregation to filter out all text blocks with a value greater than the threshold.

[0110] Again, using the characteristic vocabulary of each target information of the enterprise innovation activity information created above, the relevance score2 of all the text blocks obtained in the previous step and the enterprise profile is calculated according to the relevance calculation formula. Similarly, the larger the value of score2, the higher the relevance to the enterprise innovation activity. A threshold is set for the relevance, and all text blocks with a value greater than the threshold are filtered out to obtain the target content to be captured.

[0111] Specifically, the quality evaluation and value classification of the captured target content in S6 above includes:

[0112] The target content is judged in descending order according to the relevance Score2, and whether the text block contains the full name or the abbreviation of the enterprise. If it does, the text block is evaluated as high value, and the remaining text blocks are evaluated as low value, and the judgment is completed; if it does not, the next text block is judged until all text blocks are judged.

[0113] In some embodiments, the target information of innovation activity information of Company A captured by the method described in this application is as follows: Text 1, relevance value 98; Text 2, relevance value 95; Text 3, relevance value 92; Text 4, relevance value 90; Text 5, relevance value 89. From Text 1 to Text 5, it is judged in turn whether the text content contains A (company name) or a (company abbreviation). During the judgment process, when it is found that the text content of Text 3 contains A, it is finally judged that Text 3 is high-value, and Texts 1, 2, 4, and 5 are low-value. This completes the quality evaluation and value classification of the captured content.

[0114] Specifically, the manual review of the quality evaluation results described in S6 above includes:

[0115] During the manual review process, if there is a deviation in the quality evaluation, the characteristic vocabulary of the enterprise profile described in S1 or the characteristic vocabulary of the target information of other innovation activity information such as product data will be recreated, and an artificial logical target result will be generated. The artificial logical target result will be used for machine learning to optimize the crawling algorithm.

[0116] Example 2:

[0117] like Figure 3 As shown, the present invention provides an automated collection and extraction system for enterprise innovation activity information, including: a feature word library creation module, a web page structure scanning module, a text segmentation module, an information recognition module, a data capture module, a quality evaluation module and a data storage module;

[0118] The feature word library creation module extracts a number of keywords from a standard sample of innovation activity information based on the TF-IDF keyword extraction algorithm to obtain a feature word library of the innovation activity information; the innovation activity information refers to target information such as the company's profile information, product and service information, technological innovation information, scenario innovation, implementation cases, honorary awards, and government projects;

[0119] The webpage structure scanning module scans the webpage structure of the target website, locates the target webpage in the webpage structure using the keywords in each of the characteristic lexicons, and obtains the webpage source code of the target webpage, including using a crawler tool to crawl the target information;

[0120] The text segmentation module removes noise from the webpage source code, segments the text content into blocks according to HTML tags, and calculates the text aggregation degree of each block of content based on a text and symbol density algorithm to obtain the text block to be identified;

[0121] The information recognition module calculates the relevance of the text block to be recognized based on each characteristic word library of the innovation activity information, and obtains the target content corresponding to each characteristic word library, that is, the target innovation activity content;

[0122] The quality evaluation module performs quality evaluation and value classification on the captured target content and manually reviews the quality evaluation results;

[0123] The data storage module stores the high-value target content in an enterprise information library and stores the low-value target content in a standby comparison learning library.

[0124] It should be noted that the specific embodiments described above can enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although this specification has described the present invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the present invention can still be modified or replaced with equivalents. In short, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be included in the scope of protection of the patent for the present invention.

Claims

1. A method for automatically collecting and extracting enterprise innovation activity information, characterized in that: include: S1. Establish a feature vocabulary for innovation activity information: This innovation activity information refers to company profile information, product and service information, technological innovation information, scenario innovation, implementation case studies, honors and awards, and government project target information. Using the TF-IDF keyword extraction algorithm, extract several keywords for each target information from a sample set. These keywords are then weighted to generate multiple feature vocabulary for each target information in the innovation activity information. S2 target page positioning: scanning the target website's page structure, using the keyword of each feature of the vocabulary to locate the target page in the page structure, obtain the target page source code; S3. Blocking of web page content: Remove noise from the web page source code, block based on HTML tags, calculate the text aggregation of each block of content based on text and symbol density algorithm, and obtain the text block to be identified; S4 Information Identification: Based on each of the feature lexicon, calculate the relevance of the text block to be identified with the target information, based on the relevance screening to obtain the target content corresponding to each feature lexicon that is, the target content of the innovation activity; S5. Quality Assessment: Perform quality assessment and value classification on the captured target content, and manually review the quality assessment results; the value classification includes high value and low value; S6. Storing in a database: storing the high-value target content in the enterprise information database, and storing the low-value target content in the backup reference learning database, ultimately obtaining the target content of the multiple feature lexicons to form a complete enterprise information database.

2. The method for automatically collecting and extracting enterprise innovation activity information according to claim 1, characterized in that: The TF-IDF-based keyword extraction algorithm described in S1 extracts several keywords from a set of samples of each target information, and performs weight processing on the keywords to obtain a feature vocabulary of each target information, including: S11. A keyword extraction algorithm based on TF-IDF, using an open source word segmentation component in Python, extracts keywords from a set of standard samples of the target information; S12. Process and sort the keywords, retaining the top 50 keywords with the highest number of occurrences; S13. After manual sorting, the keywords that are relevant to the target information are weighted up, and the irrelevant keywords are weighted down, thereby obtaining a feature word library of the target information.

3. A method for automatically collecting and extracting enterprise innovation activity information according to claim 1 or 2, characterized in that: The step S2 of locating a target web page in the web page structure and obtaining the web page source code of the target web page includes: S21 establishes the target page navigation title thesaurus; S22 captures all navigation titles on the page structure, compares the navigation title with the navigation title vocabulary, identifies the target page where the navigation, extracts the corresponding web link; S23. Obtain the webpage source code of the target webpage through the webpage link.

4. The method for automatically collecting and extracting enterprise innovation activity information according to claim 3, characterized in that: The process of removing noise from the webpage source code and dividing the source code into blocks according to HTML tags in S3 includes: S31. The web page source code is noise-removed to obtain a web page structure DOM tree, the noise includes the header, footer, navigator navigation and copyright copyright; S32 traverses all subtags under the body tag in the DOM tree, establishes a map structure to store the subtag metadata and all text under the subtag; if there is a second subtag under the subtag, the second subtag content is extracted in the same way, and so on, until all tags are extracted; S33. All texts under each of the sub-tags constitute a text block.

5. The method for automatically collecting and extracting enterprise innovation activity information according to claim 4, characterized in that: The text aggregation degree of each block of content is calculated based on the text and symbol density algorithm described in S3 to obtain the text block to be identified, including: S34 uses text and symbol density algorithm to calculate the text aggregation of each text block, to obtain the text aggregation value Score1 of each text block; S35. Set a text aggregation threshold, select all text blocks whose aggregation value Score1 is greater than the text aggregation threshold, and obtain the text blocks to be identified.

6. The method for automatically collecting and extracting enterprise innovation activity information according to claim 5, characterized in that: The text and symbol density algorithm described in S34 includes: The formula for calculating text aggregation value is: Score1 = log(sd)*tdi*log10(pnumi + 2)*log(sbdi)(1) Among them, tdi is the text density, sd is the standard deviation of tdi, pnumi is the number of p labels of the node, and sbdi is the text symbol density; The above sbdi calculation formula is: sbdi = (ti - lti) / (sbi + 1) (2) Where sbi is the number of symbols in the text, ti is the number of strings in node i, that is, the length of the text in node i; lti is the number of strings with links in node i, that is, the length of the text with links in node i; The above tdi is the text density, and its calculation formula is: tdi = (ti - lti) / (tgi - ltgi) (3) Among them, ti is the number of strings of node i, that is, the length of the text of node i; lti is the number of linked strings of node i, that is, the length of the text with links of node i; tgi is the number of tags of node i, and ltgi is the number of linked tags of node i.

7. The method for automatically collecting and extracting enterprise innovation activity information according to claim 6, characterized in that: The step S4 of calculating the relevance between the text block to be identified and the target information based on the feature word library to obtain the target content includes: S41 using the feature vocabulary were calculated for each of the text blocks to be identified and the relevance of the feature vocabulary Score2; S42. Set a relevance threshold, select all text blocks whose relevance Score2 is greater than the relevance threshold, and obtain the target content.

8. The method for automatically collecting and extracting enterprise innovation activity information according to claim 7, characterized in that: The calculation of the relevance Score2 of each of the to-be-recognized text blocks and the feature word library using the feature word library in S41 includes: The correlation calculation formula is: （4） Wherein, n is the number of times the keyword appears in the text block to be identified, and normalized_weight is the normalized weight, i.e., the new weight obtained after normalizing the weights of all keywords in the feature vocabulary using the Python numpy library.

9. The method for automatically collecting and extracting enterprise innovation activity information according to claim 8, characterized in that: The quality evaluation and value classification of the captured target content as described in S5 include: The target content is judged in descending order according to the relevance Score2, and whether the text block contains the full name or the abbreviation of the enterprise. If it does, the text block is evaluated as high value, and the remaining text blocks are evaluated as low value, and the judgment is completed; if it does not, the next text block is judged until all text blocks are judged.

10. The method for automatically collecting and extracting enterprise innovation activity information according to claim 9, characterized in that: Manual review of quality assessment results as described in S5, including: During the manual review process, if there is a deviation in the quality evaluation, the feature vocabulary of each target information in the enterprise innovation activity information described in S1 will be recreated, and an artificial logical target result will be generated. The artificial logical target result will be used for machine learning to optimize the crawling algorithm.

11. An automated collection and extraction system for enterprise innovation activity information formed by the method according to any one of claims 1 to 10, characterized in that: include: Feature word library creation module, web page structure scanning module, text segmentation module, information recognition module, data capture module, quality evaluation module and data storage module; The feature word library creation module extracts a number of keywords from a standard sample of innovation activity information based on the TF-IDF keyword extraction algorithm to obtain a feature word library of the innovation activity information; the innovation activity information refers to the company's profile information, product and service information, technological innovation information, scenario innovation, implementation case studies, honorary awards, and government project target information; The webpage structure scanning module scans the webpage structure of the target website, locates the target webpage in the webpage structure using the keywords in each of the characteristic lexicons, and obtains the webpage source code of the target webpage, including using a crawler tool to crawl the target information; The text segmentation module removes noise from the webpage source code, segments the text content into blocks according to HTML tags, and calculates the text aggregation degree of each block of content based on a text and symbol density algorithm to obtain the text block to be identified; The information recognition module calculates the relevance of the text block to be recognized based on each characteristic word library of the innovation activity information, and obtains the target content corresponding to each characteristic word library, that is, the target innovation activity content; The quality evaluation module performs quality evaluation and value classification on the captured target content and manually reviews the quality evaluation results; The data storage module stores the high-value target content in the enterprise information library and stores the low-value target content in the standby comparison learning library, and finally obtains the target content of the multiple feature word libraries to form a complete enterprise information library.

Images