Automatic extraction and recommendation method and system of science and technology policy information based on web crawler

Through the method of obtaining and filtering policy information data based on network crawlers, combining collaborative knowledge graphs and graph convolution networks, automatic extraction and personalized recommendation of scientific and technological policy information is realized, and the problems of low efficiency and high cost in the existing technology are solved. Improve the retrieval efficiency of science and technology policy information and the personalized push effect.

CN118245671BActive Publication Date: 2024-11-08广东省华南技术转移中心有限公司
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410288786.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-14
Publication Date
2024-11-08
Estimated Expiration
2044-03-14

AI Technical Summary

Technical Problem

The existing technology is difficult to efficiently and automatically extract and recommend scientific and technological policy information, resulting in low efficiency and high cost, and the logic of policy provisions and the complexity of professional terms increase reading difficulty.

Method used

Using a method based on web crawler to obtain policy information web page data, extract and store science and technology policy information through pre-filtering and collaborative cloud platforms, build a science and technology policy keyword database, use user browsing records to build a collaborative knowledge graph, and build a graph convolution network based on graph Recommended model to realize the representation of user preference information and personalized push of scientific and technological policy information.

Benefits of technology

It realizes the quasi-real-time aggregation and personalized recommendation of science and technology policy information, improves the retrieval efficiency, reduces the dependence and cost of manual processing, and solves the problems of policy information dispersion and text structure complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118245671B_ABST
    Figure CN118245671B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for automatically extracting and recommending science and technology policy information based on a web crawler, including: using a web crawler method to obtain policy information web page data related to science and technology, and using a collaborative cloud platform to extract and store science and technology policy information from the pre-processed policy information web page data; determining the user's target science and technology policy keywords, and performing data crawling in the science and technology policy information gathered on the cloud platform to generate an initial push data set for science and technology policy information; using user browsing records to construct a collaborative knowledge graph to characterize the user's potential preference information; and constructing a science and technology policy recommendation model to output the user's predicted probability of science and technology policy information, and to perform subsequent push of science and technology policy information. The present invention integrates science and technology policy information, solves the problems caused by the dispersion of science and technology policies and the complexity of text structure and semantics for policy recommendation, improves the efficiency of science and technology policy retrieval, and realizes personalized push of science and technology policy information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a method and system for automatically extracting and recommending science and technology policy information based on a web crawler. Background Art

[0002] Using big data technology to build a policy service cloud platform can effectively solve the problem of scattered policy data, and is a great help in effectively analyzing government policy information and giving full play to the policy-oriented role. With the powerful support of big data-related technologies, it is possible to integrate and classify massive amounts of policy data, extract information, collect resources, and accurately match them. Policies no longer exist in isolation, but form countless related combinations and systems, and the huge value behind them will be revealed.

[0003] There are many policy articles, which are strictly logical and standardized, and contain many professional terms. Each article has extremely rich content, which brings great challenges to the reading comprehension of the research and application groups. Most of the current information systems can only collect and input information, and rely heavily on manual processing. The number of new policy items is currently in the hundreds every working day. If the traditional method is used, after the policy data is entered, manual information extraction and integrity check are required, which is inefficient and costly. Therefore, how to efficiently extract and recommend science and technology policy information is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method and system for automatically extracting and recommending science and technology policy information based on a web crawler.

[0005] The first aspect of the present invention provides a method for automatically extracting and recommending science and technology policy information based on a web crawler, comprising:

[0006] Using web crawler methods to obtain policy information web page data related to science and technology, pre-filtering the policy information web page data, and collaborating with the cloud platform to extract and store science and technology policy information from the pre-processed policy information web page data, thereby achieving quasi-real-time aggregation of science and technology policies;

[0007] Build a science and technology policy keyword library, determine the user's target science and technology policy keywords, use the target science and technology policy keywords to crawl data from the science and technology policy information gathered on the cloud platform, and generate an initial push data set of science and technology policy information;

[0008] Obtaining the user's browsing interaction with the initial pushed data set, and using the user's browsing history to construct a collaborative knowledge graph to represent the user's potential preference information;

[0009] A science and technology policy recommendation model is constructed based on a graph convolutional network, and representation learning of users and science and technology policies is realized according to the potential preference information. The predicted probability of users for science and technology policy information is output, and subsequent push of science and technology policy information is performed according to the predicted probability.

[0010] In this solution, a web crawler method is used to obtain policy information web page data related to science and technology, and the policy information web page data is pre-filtered, specifically:

[0011] Obtain website portraits of different websites according to the front-end and back-end data interaction methods, data encapsulation methods and security mechanisms of different websites, preset a web crawler method library, and use the website portraits to obtain the best web crawler method in the web crawler method library;

[0012] Using the optimal web crawler method to periodically obtain original unstructured web pages, determining the matching degree between the label of the web page content and the preset science and technology policy label in the unstructured web page data, and performing pre-filtering according to the matching degree to obtain policy information web page data related to science and technology;

[0013] Learning the content tag hierarchy of the policy release website, establishing a tree structure of different web page templates according to the content tag hierarchy and storing them in a preset cloud platform, reading the web page template of the policy consultation web page data, and extracting science and technology policy information by calling the tree structure through the cloud platform;

[0014] Define the categories of science and technology policies and the description information of different categories of science and technology policy information, extract keywords from the extracted science and technology policy information, obtain keyword vectors and corresponding semantic information, and use machine learning methods to obtain the Manhattan distance between keywords and description information to classify science and technology policy information;

[0015] The extracted science and technology policy information is matched with the corresponding science and technology policy categories and stored in the database in the cloud platform.

[0016] In this scheme, the target science and technology policy keywords are used to crawl data from the science and technology policy information gathered on the cloud platform to generate an initial push data set of science and technology policy information, specifically:

[0017] Filter entity word vectors according to different categories of science and technology policy description information to build a science and technology policy keyword library, use the science and technology policy keyword library to train a word segmentation model, obtain the user's current timestamp retrieval requirements, and import the text corresponding to the retrieval requirements into the word segmentation model;

[0018] The word segmentation model is used to obtain the probability of each character in the text corresponding to the search requirement belonging to the entity word label, and characters that do not meet the probability threshold range are filtered out to obtain the remaining word vectors and short sentence vectors;

[0019] Use the word vector and the short sentence vector to perform word segmentation to obtain the demand word segmentation result, and use the similarity calculation to obtain the target science and technology policy keyword in the science and technology policy keyword library;

[0020] A coarse-grained data search is performed on the science and technology policy information gathered on the cloud platform through the target science and technology policy keywords, and science and technology policy information that meets the search criteria is captured according to a preset number to generate an initial push data set.

[0021] In this solution, the user's browsing history is used to construct a collaborative knowledge graph to represent the user's potential preference information, specifically:

[0022] Determine the entities corresponding to different categories of science and technology policies and the relationships between entities in the science and technology policy keyword library, and construct triples through entities and relationships to generate a knowledge graph corresponding to the science and technology policy key system;

[0023] According to the browsing interaction data of the user with the initial push data set within a preset time, the user browsing record is used to determine whether the user has an interaction record with the science and technology policy information in the initial push data set, and a short-term interaction matrix is ​​constructed. A short-term collaborative knowledge graph is constructed through the knowledge graph and the short-term interaction matrix;

[0024] Filter the user's historical interaction data according to the user's identity information, terminal address information, IP address and physical address, build a long-term interaction matrix according to the historical interaction data, and build a long-term collaborative knowledge graph through the long-term interaction matrix and knowledge graph;

[0025] In the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, the TransR method is used to perform feature mapping of the entities in the triple in the relational space, and a relational attention mechanism is introduced in the relational space to aggregate the entities and neighboring entities into a new entity representation;

[0026] Through multi-layer propagation, the embedded representation of science and technology policy entities is obtained, and the short-term science and technology policy embedded sequence and the long-term science and technology policy embedded sequence are generated. The short-term science and technology policy embedded sequence and the long-term science and technology policy embedded sequence are learned using the GUR unit.

[0027] The hidden state output by the GUR unit represents the user's short-term preference representation and long-term preference representation, and the user's potential preference information is obtained by combining the weight information.

[0028] In this scheme, the time interval between two adjacent science and technology policy interaction projects is calculated in the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence;

[0029] The time interval is encoded to obtain the time correlation between each pair of science and technology policy projects, and the encoded time interval is combined with the self-attention mechanism to obtain the weight information of the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence.

[0030] In this scheme, the representation learning of users and science and technology policies is realized according to the potential preference information, and the predicted probability of users for science and technology policy information is output, specifically:

[0031] A science and technology policy recommendation model is built based on graph convolutional networks. The science and technology policy entities directly connected to users are obtained and marked in the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, and the similarity between the relationship corresponding to the mark and the potential preference information is compared.

[0032] Normalizing the similarity, extracting science and technology policy entities whose normalized similarity scores are greater than a preset score threshold as neighbor nodes, aggregating the user's embedded representation and neighbor nodes, and obtaining an aggregated user vector representation;

[0033] Using the embedded representation of the science and technology policy entity and the aggregated user vector representation to perform a cross operation, a cross matrix is ​​constructed to obtain the science and technology policy vector representation, and the prediction probability of the user vector representation to the science and technology policy vector representation is calculated according to the activation function;

[0034] According to the predicted probability, a preset number of science and technology policy information is filtered as push data, the science and technology policy category of the push data is obtained, similar push data is obtained based on the science and technology policy category, and the push data and similar push data are used to establish a push data set for subsequent push.

[0035] The second aspect of the present invention also provides a system for automatically extracting and recommending science and technology policy information based on a web crawler, the system comprising: a memory, a processor, a timed centralized procurement control module, a data crawler adaptation module, a resource management module, a collaborative recommendation learning module, and a real-time recommendation module, wherein the memory and the processor store and execute a method program for automatically extracting and recommending science and technology policy information based on a web crawler:

[0036] The timed centralized acquisition control module sets a timer to start the task and presets the crawling task time;

[0037] The data crawler adaptation module selects the best web crawler method based on the website portrait to obtain policy information web page data related to science and technology;

[0038] The resource management module pre-filters the crawled policy information webpage data, extracts and stores the science and technology policy information using the cloud platform, classifies the science and technology policy information, and uses the user's target science and technology policy keywords to crawl data to generate an initial push data set for the science and technology policy information;

[0039] The collaborative recommendation learning module uses user browsing history to build short-term collaborative knowledge graphs and long-term collaborative knowledge graphs to learn and represent users' potential preference information;

[0040] The real-time recommendation module builds a science and technology policy recommendation model to obtain the user's predicted probability of science and technology policy information, and selects a preset number of science and technology policy information for subsequent push.

[0041] The present invention discloses a method and system for automatically extracting and recommending science and technology policy information based on a web crawler, including: using a web crawler method to obtain policy information web page data related to science and technology, and coordinating a cloud platform to extract and store science and technology policy information from the pre-processed policy information web page data; determining the user's target science and technology policy keywords, and performing data crawling in the science and technology policy information gathered on the cloud platform to generate an initial push data set for science and technology policy information; using user browsing records to construct a collaborative knowledge graph to characterize the user's potential preference information; constructing a science and technology policy recommendation model to output the user's predicted probability of science and technology policy information, and then push the science and technology policy information. The present invention integrates science and technology policy information, solves the problems caused by the dispersion of science and technology policies and the complexity of text structure and semantics for policy recommendation, improves the efficiency of science and technology policy retrieval, and realizes personalized push of science and technology policy information. A method for realizing data collection based on a modeled data network crawler method, a method for coordinating cloud storage resource application for cost control, and then a method for collecting key information through feature words, and a method for coordinating recommendations with user browsing records to realize a method for accurate recommendation of science and technology policies. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of the method for automatically extracting and recommending science and technology policy information based on a web crawler of the present invention is shown;

[0043] Figure 2 A flowchart of the present invention for constructing a collaborative knowledge graph to represent potential preference information of a user is shown;

[0044] Figure 3 A flow chart showing the construction of a science and technology policy recommendation model according to the present invention is shown;

[0045] Figure 4 The block diagram of the automatic extraction and recommendation system of science and technology policy information based on web crawlers of the present invention is shown. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0048] Figure 1 The flowchart of the method for automatically extracting and recommending science and technology policy information based on a web crawler of the present invention is shown.

[0049] like Figure 1 As shown, the first aspect of the present invention provides a method for automatically extracting and recommending science and technology policy information based on a web crawler, comprising:

[0050] S102, using a web crawler method to obtain policy information web page data related to science and technology, pre-filtering the policy information web page data, and collaborating with the cloud platform to extract and store science and technology policy information from the pre-processed policy information web page data, thereby realizing quasi-real-time aggregation of science and technology policies;

[0051] S104, constructing a science and technology policy keyword library, determining the user's target science and technology policy keywords, using the target science and technology policy keywords to crawl data from the science and technology policy information gathered on the cloud platform, and generating an initial push data set of science and technology policy information;

[0052] S106, obtaining the user's browsing interaction with the initial pushed data set, and constructing a collaborative knowledge graph using the user's browsing history to represent the user's potential preference information;

[0053] S108, constructing a science and technology policy recommendation model based on a graph convolutional network, realizing representation learning of users and science and technology policies according to the potential preference information, outputting the predicted probability of users for science and technology policy information, and subsequently pushing science and technology policy information according to the predicted probability.

[0054] It should be noted that different websites have different ways of interacting with the front-end and back-end data, different forms of data encapsulation, and even some websites have set up security mechanisms. Data acquisition needs to take into full consideration various situations. The problems that need to be handled by the data capture script can be divided into the following categories: (1) Field mapping. Although most official website policy data fields are similar, the naming methods are diverse. The fields need to be remapped according to the existing dictionary, mainly including the release time, policy title, and policy original text. (2) Interface acquisition. Websites that use this method can directly parse by crawling interface data, but most websites have verification of the number of interface visits, intervals, keys, tokens and other information. Postman needs to be used to fully simulate and replay the interface characteristics. (3) Page interception. Because the front-end uses JavaScript to encapsulate the interface data twice or does not use the front-end and back-end separation method, the interface data cannot be directly obtained. The page structure needs to be parsed. For the page list that uses JavaScript for secondary encapsulation, regular expressions need to be written to parse each detail page. This type of website has many changes, and scripts need to be written dynamically according to actual conditions.

[0055] According to the front-end and back-end data interaction methods, data encapsulation methods and security mechanisms of different websites, website portraits of different websites are obtained, and a web crawler method library is preset, including methods such as general web crawlers, focused web crawlers, incremental web crawlers and deep web crawlers. The best web crawler method is obtained from the web crawler method library using the website portrait; the original unstructured web pages are obtained regularly using the best web crawler method, and the matching degree between the label of the web page content and the preset science and technology policy label is judged in the unstructured web page data, and the policy information web page data related to science and technology is obtained by pre-filtering according to the matching degree; the content label hierarchy of the policy release website is learned, and a tree structure of different web page templates is established according to the content label hierarchy structure and stored in the preset cloud platform. Generally, the same type of web pages under a certain initial URL use the same content label hierarchy structure, that is, the same web page template. The web page template of the policy consultation web page data is read, and the science and technology policy information is extracted by calling the tree structure through the cloud platform.

[0056] The science and technology policies are divided into 15 categories: comprehensive, scientific research institutions, reform of science and technology plans, management of science and technology funds and finance, basic research and scientific research bases, enterprise technological progress and high-tech industrialization, rural science and technology and social development, scientific and technological talents, science and technology intermediary services, science and technology conditions and standards, science and technology finance and taxation, scientific and technological achievements and intellectual property rights, popularization of science and technology, science and technology awards, and international science and technology cooperation; the science and technology policy categories and the descriptive information of different categories of science and technology policy information are defined, the extracted science and technology policy information is subjected to keyword extraction, the keyword vector and the corresponding semantic information are obtained, and the machine learning method is used to obtain the Manhattan distance of the keywords and the descriptive information for classification of the science and technology policy information; the extracted science and technology policy information is matched with the corresponding science and technology policy categories and stored in the database in the cloud platform.

[0057] It should be noted that entity word vectors are screened according to different categories of science and technology policy description information to construct a science and technology policy keyword library, and the science and technology policy keyword library is used in combination with deep learning methods such as LSTM networks to train a word segmentation model to obtain the user's retrieval needs of the current timestamp, and the text corresponding to the retrieval needs is imported into the word segmentation model to identify keyword entities; the probability of each character in the text corresponding to the retrieval need belonging to the entity word label is obtained through the word segmentation model, and the characters that do not meet the probability threshold interval are screened and eliminated to obtain the remaining word vectors and short sentence vectors; the word vectors and short sentence vectors are used to perform word segmentation to obtain the demand word segmentation results, and the demand word segmentation results are used in the science and technology policy keyword library by similarity calculation to obtain the target science and technology policy keywords; coarse-grained data retrieval is performed in the science and technology policy information gathered on the cloud platform through the target science and technology policy keywords, and science and technology policy information that meets the retrieval criteria is captured according to a preset number to generate an initial push data set.

[0058] Figure 2 A flow chart of the present invention for constructing a collaborative knowledge graph to characterize a user's potential preference information is shown.

[0059] According to an embodiment of the present invention, a collaborative knowledge graph is constructed using user browsing records to represent the user's potential preference information, specifically:

[0060] S202, determining entities corresponding to different categories of science and technology policies and the relationships between entities in the science and technology policy keyword library, and generating a knowledge graph corresponding to the science and technology policy keyword system by constructing triples through entities and relationships;

[0061] S204, based on the browsing interaction data of the user with the initial push data set within a preset time, determine whether the user has an interaction record with the science and technology policy information in the initial push data set by using the user browsing record, construct a short-term interaction matrix, and construct a short-term collaborative knowledge graph through the knowledge graph and the short-term interaction matrix;

[0062] S206, filtering the user's historical interaction data according to the user's identity information, terminal address information, IP address, and physical address, constructing a long-term interaction matrix according to the historical interaction data, and constructing a long-term collaborative knowledge graph through the long-term interaction matrix and the knowledge graph;

[0063] S208, in the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, using the TransR method to perform feature mapping of the entities in the triples into the relational space, introducing a relational attention mechanism in the relational space, and aggregating the entities and neighboring entities into a new entity representation;

[0064] S210, obtaining the embedded representation of the science and technology policy entity through multi-layer propagation, generating a short-term science and technology policy embedded sequence and a long-term science and technology policy embedded sequence, and using the GUR unit to learn the short-term science and technology policy embedded sequence and the long-term science and technology policy embedded sequence;

[0065] S212, the user's short-term preference representation and long-term preference representation are represented by the hidden state output by the GUR unit, and the user's potential preference information is obtained in combination with the weight information.

[0066] It should be noted that in the science and technology policy keyword library, the corresponding entities of different categories of science and technology policies and the relationships between entities are determined, and the triples (h, r, t) are constructed to generate the knowledge graph corresponding to the science and technology policy keyword system. h∈E, r∈R, t∈E are represented as the head entity, relationship and tail entity respectively, and E and R represent the set of entities and relationships. Introducing the relational attention mechanism in the relational space f(h,r,t)=(W r e t ) T sigmoid(W r e h +e r ), e h ,e r ,e t is the initial representation of the head entity, relation and tail entity, W r is the transformation matrix corresponding to the relationship. In the collaborative knowledge graph, users and technology policies are regarded as new entities, and whether there are browsing interaction records is regarded as a new relationship. The gated recurrent unit GRU is used to model the short-term and long-term preferences of users, and the long-term technology policy embedding sequence is modeled to capture the short-term and long-term preferences of users. At the same time, considering how technology policies stimulate users' long-term and short-term preferences, the final technology policy is recommended to users based on different contributions in the long term and short term, thereby improving the accuracy of recommendations.

[0067] In the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence, the time interval between two adjacent science and technology policy interaction items is calculated, and the time interval information is used as auxiliary information to further model the user's dynamic preference; the time interval is encoded to obtain the time correlation between each pair of science and technology policy items, and the weight information of the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence is obtained by combining the encoded time interval with the self-attention mechanism. The encoded time interval T i Expressed as: T i =σ(Wlog(t i -t i-1 )+b), σ is the activation function, t i ,t i-1 is the timestamp of the i-th and i-1-th science and technology policy interaction projects, W is the transformation matrix, b is the bias, and the self-attention mechanism is used to calculate the different weights of long-term preferences and short-term preferences to complete the fusion of long-term and short-term preferences. The self-attention mechanism outputs the weight information q as follows: x i x i-1 are the i-th and i-1-th science and technology policy interaction projects, Represents the matrix size, used to scale attention, and T is the matrix transpose.

[0068] Figure 3 A flow chart of the present invention for constructing a science and technology policy recommendation model is shown.

[0069] According to an embodiment of the present invention, representation learning of users and science and technology policies is implemented based on the potential preference information, and the predicted probability of users for science and technology policy information is output, specifically:

[0070] S302, constructing a science and technology policy recommendation model based on a graph convolutional network, obtaining science and technology policy entities directly connected to the user in the short-term collaborative knowledge graph and the long-term collaborative knowledge graph for labeling, and comparing the relationship corresponding to the label with the similarity of the potential preference information;

[0071] S304, normalizing the similarity, extracting science and technology policy entities whose normalized similarity scores are greater than a preset score threshold as neighbor nodes, aggregating the user's embedded representation and neighbor nodes, and obtaining an aggregated user vector representation;

[0072] S306, performing a cross operation using the embedded representation of the science and technology policy entity and the aggregated user vector representation, constructing a cross matrix to obtain the science and technology policy vector representation, and calculating the prediction probability of the user vector representation to the science and technology policy vector representation according to the activation function;

[0073] S308, filtering a preset number of science and technology policy information as push data according to the predicted probability, obtaining the science and technology policy category of the push data, obtaining similar push data based on the science and technology policy category, and establishing a push data set with the push data and similar push data for subsequent push.

[0074] It should be noted that the graph convolutional network is used to aggregate the neighborhood information of entities in the collaborative knowledge graph to enrich the semantic representation of entities. The similarity between the relationship between the science and technology policies directly connected to the user and the potential preference information is used to represent the user's attention to different relationships. The neighbor entities that the user is interested in are selected to update the user's embedded representation. The embedded representation v of the science and technology policy entity is used to represent the user vector representation e after aggregation. u Perform crossover operation v·e u , we get the updated science and technology policy vector representation e from the cross matrix v According to the activation function, the inner product of the user vector representation and the technology policy vector representation is calculated as the predicted probability y = σ (e u T e v ), σ is the activation function, and T is the matrix transpose. Based on the science and technology policy category, data with a similarity greater than a preset similarity threshold is obtained as similar push data, and the push data and similar push data are used to establish a push data set for subsequent push.

[0075] According to an embodiment of the present invention, when a user is not clear about a specific science and technology policy, a fuzzy description of existing science and technology projects and demand policies is made according to a preset format, the fuzzy description text is segmented using a word segmentation model, the frequency and distribution of each word vector in the word segmentation result are extracted to determine keywords, science and technology policy associations are made according to the keywords, interaction data of the user's association with the science and technology policy is obtained, and the affiliation of different science and technology policies is continuously updated according to the interaction data. In addition, similar users of the current user are obtained from historical recommendation information according to the keywords, and historical interaction data of similar users and historical recommendation results are obtained. The associated science and technology policies and corresponding affiliations are updated through the historical interaction data, a recommendation list is generated according to the affiliation, and associated science and technology policies are pre-recommended according to the recommendation list. A tree structure is generated according to the temporal correlation and coupling correlation of science and technology policies to facilitate users' subsequent retrieval and review of science and technology policies.

[0076] Figure 4 The block diagram of the automatic extraction and recommendation system of science and technology policy information based on web crawlers of the present invention is shown.

[0077] The second aspect of the present invention also provides a system 4 for automatically extracting and recommending science and technology policy information based on a web crawler, the system comprising: a memory 41, a processor 42, a timed centralized procurement control module 43, a data crawler adaptation module 44, a resource management module 45, a collaborative recommendation learning module 46, and a real-time recommendation module 47, wherein the memory and the processor store and execute a method program for automatically extracting and recommending science and technology policy information based on a web crawler:

[0078] The timed centralized acquisition control module sets a timer to start the task and presets the crawling task time;

[0079] The data crawler adaptation module selects the best web crawler method based on the website portrait to obtain policy information web page data related to science and technology;

[0080] The resource management module pre-filters the crawled policy information webpage data, extracts and stores the science and technology policy information using the cloud platform, classifies the science and technology policy information, and uses the user's target science and technology policy keywords to crawl data to generate an initial push data set for the science and technology policy information;

[0081] The collaborative recommendation learning module uses user browsing history to build short-term collaborative knowledge graphs and long-term collaborative knowledge graphs to learn and represent users' potential preference information;

[0082] The real-time recommendation module builds a science and technology policy recommendation model to obtain the user's predicted probability of science and technology policy information, and selects a preset number of science and technology policy information for subsequent push.

[0083] The third aspect of the present invention also provides a computer-readable storage medium, which includes a method program for automatically extracting and recommending science and technology policy information based on a web crawler. When the method program for automatically extracting and recommending science and technology policy information based on a web crawler is executed by a processor, the steps of the method for automatically extracting and recommending science and technology policy information based on a web crawler as described in any of the above items are implemented.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0085] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0086] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0087] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0088] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0089] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for automatically extracting and recommending science and technology policy information based on a web crawler, characterized in that: The following steps are involved: Using web crawler methods to obtain policy information web page data related to science and technology, pre-filtering the policy information web page data, and collaborating with the cloud platform to extract and store science and technology policy information from the pre-processed policy information web page data, thereby achieving quasi-real-time aggregation of science and technology policies; Build a science and technology policy keyword library, determine the user's target science and technology policy keywords, use the target science and technology policy keywords to crawl data from the science and technology policy information gathered on the cloud platform, and generate an initial push data set of science and technology policy information; Obtaining the user's browsing interaction with the initial pushed data set, and using the user's browsing history to construct a collaborative knowledge graph to represent the user's potential preference information; A science and technology policy recommendation model is constructed based on a graph convolutional network, and representation learning between users and science and technology policies is realized according to the potential preference information, and the predicted probability of the user for science and technology policy information is output, and subsequent science and technology policy information is pushed according to the predicted probability; The collaborative knowledge graph is constructed using user browsing records to represent the user's potential preference information, specifically: Determine the entities corresponding to different categories of science and technology policies and the relationships between them in the science and technology policy keyword database, and construct triples through entities and relationships to generate the knowledge graph corresponding to science and technology policy keywords; According to the browsing interaction data of the user with the initial push data set within a preset time, the user browsing record is used to determine whether the user has an interaction record with the science and technology policy information in the initial push data set, and a short-term interaction matrix is ​​constructed. A short-term collaborative knowledge graph is constructed through the knowledge graph and the short-term interaction matrix; Filter the user's historical interaction data according to the user's identity information, terminal address information, IP address and physical address, build a long-term interaction matrix according to the historical interaction data, and build a long-term collaborative knowledge graph through the long-term interaction matrix and knowledge graph; In the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, the TransR method is used to perform feature mapping of the entities in the triple in the relational space, and a relational attention mechanism is introduced in the relational space to aggregate the entities and neighboring entities into a new entity representation; Through multi-layer propagation, the embedded representation of science and technology policy entities is obtained, and the short-term science and technology policy embedded sequence and the long-term science and technology policy embedded sequence are generated. The short-term science and technology policy embedded sequence and the long-term science and technology policy embedded sequence are learned using the GUR unit. The hidden state output by the GUR unit represents the user's short-term and long-term preferences, and the user's potential preference information is obtained by combining the weight information; Calculate the time interval between two adjacent science and technology policy interaction projects in the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence; The time interval is encoded to obtain the time correlation between each pair of science and technology policy projects, and the weight information of the short-term science and technology policy embedding sequence and the long-term science and technology policy embedding sequence is obtained by combining the encoded time interval with the self-attention mechanism. It is expressed as: , is the activation function, is the timestamp of the i-th and i-1-th science and technology policy interaction projects, is the transformation matrix, The self-attention mechanism is used to calculate the different weights of long-term preference and short-term preference to complete the fusion of long-term and short-term preference. The self-attention mechanism outputs weight information for: , are the i-th and i-1-th science and technology policy interaction projects, Transpose the matrix.

2. According to claim 1, a method for automatically extracting and recommending science and technology policy information based on a web crawler is characterized in that: The web crawler method is used to obtain policy information web page data related to science and technology, and the policy information web page data is pre-filtered, specifically: Obtain website portraits of different websites according to the front-end and back-end data interaction methods, data encapsulation methods and security mechanisms of different websites, preset a web crawler method library, and use the website portraits to obtain the best web crawler method in the web crawler method library; Using the optimal web crawler method to periodically obtain original unstructured web pages, determine the matching degree between the label of the web page content and the preset science and technology policy label in the unstructured web page data, and perform pre-filtering based on the matching degree to obtain policy information web page data related to science and technology; Learning the content tag hierarchy of the policy release website, establishing a tree structure of different web page templates according to the content tag hierarchy and storing them in a preset cloud platform, reading the web page template of the policy consultation web page data, and extracting science and technology policy information by calling the tree structure through the cloud platform; Define the categories of science and technology policies and the description information of different categories of science and technology policy information, extract keywords from the extracted science and technology policy information, obtain keyword vectors and corresponding semantic information, and use machine learning methods to obtain the Manhattan distance between keywords and description information to classify science and technology policy information; The extracted science and technology policy information is matched with the corresponding science and technology policy categories and stored in the database in the cloud platform.

3. According to the method of automatic extraction and recommendation of science and technology policy information based on web crawlers in claim 1, it is characterized in that: The target science and technology policy keywords are used to crawl data from the science and technology policy information gathered on the cloud platform to generate an initial push data set of science and technology policy information, specifically: Filter entity word vectors according to different categories of science and technology policy description information to build a science and technology policy keyword library, use the science and technology policy keyword library to train a word segmentation model, obtain the user's current timestamp retrieval requirements, and import the text corresponding to the retrieval requirements into the word segmentation model; The word segmentation model is used to obtain the probability of each character in the text corresponding to the search requirement belonging to the entity word label, and characters that do not meet the probability threshold range are filtered out to obtain the remaining word vectors and short sentence vectors; Use the word vector and the short sentence vector to perform word segmentation to obtain the demand word segmentation result, and use the similarity calculation to obtain the target science and technology policy keyword in the science and technology policy keyword library; A coarse-grained data search is performed on the science and technology policy information gathered on the cloud platform through the target science and technology policy keywords, and science and technology policy information that meets the search criteria is captured according to a preset number to generate an initial push data set.

4. According to the method of automatic extraction and recommendation of science and technology policy information based on web crawlers in claim 1, it is characterized in that: According to the potential preference information, the representation learning of users and science and technology policies is realized, and the predicted probability of users for science and technology policy information is output, specifically: A science and technology policy recommendation model is built based on a graph convolutional network. The science and technology policy entities directly connected to the user are obtained and marked in the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, and the similarity between the relationship corresponding to the mark and the potential preference information is obtained. Normalizing the similarity, extracting science and technology policy entities whose normalized similarity scores are greater than a preset score threshold as neighbor nodes, aggregating the user's embedded representation and neighbor nodes, and obtaining an aggregated user vector representation; Using the embedded representation of the science and technology policy entity and the aggregated user vector representation to perform a cross operation, a cross matrix is ​​constructed to obtain the science and technology policy vector representation, and the prediction probability of the user vector representation to the science and technology policy vector representation is calculated according to the activation function; According to the predicted probability, a preset number of science and technology policy information is filtered as push data, the science and technology policy category of the push data is obtained, similar push data is obtained based on the science and technology policy category, and the push data and similar push data are used to establish a push data set for subsequent push.

5. A system for automatically extracting and recommending science and technology policy information based on web crawlers, characterized in that: The method for automatically extracting and recommending science and technology policy information based on a web crawler as described in any one of claims 1 to 4 is implemented. The system includes: a memory, a processor, a timed centralized procurement control module, a data crawler adaptation module, a resource management module, a collaborative recommendation learning module, and a real-time recommendation module. The program for the method for automatically extracting and recommending science and technology policy information based on a web crawler is stored and executed in the memory and the processor: The timed centralized acquisition control module sets a timer to start the task and presets the crawling task time; The data crawler adaptation module selects the best web crawler method based on the website portrait to obtain policy information web page data related to science and technology; The resource management module pre-filters the crawled policy information webpage data, extracts and stores the science and technology policy information using the cloud platform, classifies the science and technology policy information, and uses the user's target science and technology policy keywords to crawl data to generate an initial push data set for the science and technology policy information; The collaborative recommendation learning module uses user browsing records to construct short-term collaborative knowledge graphs and long-term collaborative knowledge graphs to learn and represent user potential preference information; The real-time recommendation module constructs a science and technology policy recommendation model to obtain the user's predicted probability of science and technology policy information, and selects a preset number of science and technology policy information for subsequent push.

6. The system for automatically extracting and recommending science and technology policy information based on web crawlers according to claim 5 is characterized in that: Generate the initial push data set of science and technology policy information, specifically: Filter entity word vectors according to different categories of science and technology policy description information to build a science and technology policy keyword library, use the science and technology policy keyword library to train a word segmentation model, obtain the user's current timestamp retrieval requirements, and import the text corresponding to the retrieval requirements into the word segmentation model; The word segmentation model is used to obtain the probability of each character in the text corresponding to the search requirement belonging to the entity word label, and characters that do not meet the probability threshold range are filtered out to obtain the remaining word vectors and short sentence vectors; Use the word vector and the short sentence vector to perform word segmentation to obtain the demand word segmentation result, and use the similarity calculation to obtain the target science and technology policy keyword in the science and technology policy keyword library; A coarse-grained data search is performed on the science and technology policy information gathered on the cloud platform through the target science and technology policy keywords, and science and technology policy information that meets the search criteria is captured according to a preset number to generate an initial push data set.

7. The system for automatically extracting and recommending science and technology policy information based on web crawlers according to claim 5 is characterized in that: Construct a science and technology policy recommendation model to obtain the user's predicted probability of science and technology policy information, specifically: A science and technology policy recommendation model is built based on a graph convolutional network. The science and technology policy entities directly connected to the user are obtained and marked in the short-term collaborative knowledge graph and the long-term collaborative knowledge graph, and the similarity between the relationship corresponding to the mark and the potential preference information is obtained. Normalizing the similarity, extracting science and technology policy entities whose normalized similarity scores are greater than a preset score threshold as neighbor nodes, aggregating the user's embedded representation and neighbor nodes, and obtaining an aggregated user vector representation; Using the embedded representation of the science and technology policy entity and the aggregated user vector representation to perform a cross operation, a cross matrix is ​​constructed to obtain the science and technology policy vector representation, and the prediction probability of the user vector representation to the science and technology policy vector representation is calculated according to the activation function; According to the predicted probability, a preset number of science and technology policy information is filtered as push data, the science and technology policy category of the push data is obtained, similar push data is obtained based on the science and technology policy category, and the push data and similar push data are used to establish a push data set for subsequent push.

Citation Information

Patent Citations

  • Sequence recommendation method and device and computer readable storage medium

    CN111522962A

  • Self-supervised learning collaborative sequence recommendation method and device based on time interval and medium

    CN114896515A

  • Deep knowledge tracing with transformers

    US20210390873A1