A method and apparatus for data processing based on knowledge graphs

Abstract

The embodiment of the present invention discloses a data processing method and device based on a knowledge graph, as well as a storage medium and an electronic device, wherein the method includes: realizing efficient management and precise service of policy information through automated collection, structured processing, intelligent classification, ontology model construction and personalized matching and push, the method improves the efficiency of data collection and processing, enhances the accuracy of text classification, optimizes data storage and query, improves user experience, and enhances the practicality and adaptability of the system through a flexible task response mechanism, thereby improving the quality and efficiency of policy information processing.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a data processing method and device based on a knowledge graph, as well as a storage medium and an electronic device. Background Art

[0002] In the existing technology, technical means such as knowledge graph and deep learning are usually used to conduct semantic mining and structured analysis of science and technology policy texts. Knowledge graph technology converts unstructured text data into structured semantic knowledge networks by extracting knowledge entities and their relationships in the text, allowing users to understand policy content and their relationships more intuitively and quickly. Deep learning technology, especially deep neural network models, provides strong support for automatically identifying and extracting entity relationships in texts, making the construction of knowledge graphs more professional and refined.

[0003] Although some progress has been made in existing research, there are still many challenges in efficiently collecting policy data from massive network resources, accurately extracting and parsing key information of policy texts, and presenting them to users in an intuitive and easy-to-understand way. Therefore, developing a data processing method based on knowledge graphs has important practical significance and application value for improving the efficiency of extracting and processing policy information. Summary of the invention

[0004] In order to make up for the shortcomings of the prior art, the present invention provides a data processing method based on knowledge graph, which realizes efficient management and precise service of policy information through automated collection, structured processing, intelligent classification, ontology model construction and personalized matching and push. This method improves the efficiency of data collection and processing, enhances the accuracy of text classification, optimizes data storage and query, improves user experience, and enhances the practicality and adaptability of the system through a flexible task response mechanism, thereby improving the quality and efficiency of policy information processing.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method based on a knowledge graph, comprising: S1, deploying a cloud data extraction program in a cloud environment, wherein the cloud data extraction program is configured to obtain data from at least one specified policy data source, parse the data into a predetermined structured format, and upload the data in the structured format to a cloud storage;

[0006] S2. Use the pre-trained Transformer model with sentence-level attention layer to encode the policy text, evaluate the importance of each sentence in the entire document, and classify the policy type.

[0007] S3, the user calls and parses the data in the cloud storage, searches the relationship table containing the policy information from the cloud storage according to the information field of the policy data, extracts data from the relationship table, maps the extracted data to the above policy information field based on the phrase relationship tree structure, and stores it in the local database;

[0008] S4. Read the fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triples required for the policy knowledge graph, and store the knowledge graph.

[0009] S5. When users search for policy information on the front end, the policy data is matched according to the keyword field and the industry to which the user's company belongs. A best matching score is calculated for each policy data according to the preset matching algorithm, and the data is sorted according to the score to push the most relevant policy data to the user.

[0010] In one embodiment, in the above step S2, a Transformer model of a sentence-level attention layer is added, and its structure includes SentenceEncoder: a Transformer-based encoder for encoding each sentence, SentenceAttentionLayer: using a linear transformation to map the representation of the sentence to a scalar value representing the importance score of the sentence, and using softmax normalization to evaluate the importance of the sentence, DocumentRepresentation: combining sentence-level attention weights to construct a representation of the entire document, and ClassificationLayer: classifying the document representation.

[0011] In one embodiment, in step S3, the step of constructing the phrase relationship tree includes initializing the root node: creating a root node representing the entire text, dividing the text into multiple clauses, each clause containing a field and its corresponding value; creating a child node for each clause and making it a child node of the root node; creating child nodes for the fields and values ​​in each clause respectively, and making them child nodes of the corresponding child nodes.

[0012] In one embodiment, in step S4, by reading the relationship tree field in the local data, the policy entities, attributes and relationships are defined to construct a policy ontology model; the entities include seven entity types: science and technology policies, categories, institutions, related policies, regions, provinces, and cities; the attributes are defined as classifying funding, regulatory penalties, and honorary qualifications as entity attributes of the policy; the relationships are defined as including reference relationships between policies, as well as relationships of categories, releases, and affiliations; after the above processing steps, the entity, attribute and relationship triples required for the policy knowledge graph are obtained.

[0013] In one implementation manner, in step S5, the matching score is calculated as follows:

[0014]

[0015] Among them, the keyword matching weight is , the industry matching weight is , the time matching weight is , the regional matching weight is , TDIDFkw word frequency-inverse document frequency value is used to measure the importance of keywords in policy titles and contents; IndustrySim industry matching measures the similarity between the industry to which the user's enterprise belongs and the industry to which the policy applies; TimeDiff time matching measures the proximity between the policy release time and the current time; RegionMatch regional matching measures the similarity between the area that the user is concerned about and the area where the policy is released.

[0016] In a second aspect, the present invention provides a data processing device based on a knowledge graph, comprising: a data acquisition module, used to deploy a cloud data extraction program in a cloud environment, wherein the cloud data extraction program is configured to obtain data from at least one designated policy data source, parse the data into a predetermined structured format, and upload the data in the structured format to cloud storage.

[0017] A classification module is used to encode the policy text using a pre-trained Transformer model with a sentence-level attention layer, evaluate the importance of each sentence in the entire document, and classify the policy type;

[0018] A processing module, used to call and parse the data in the cloud storage, search the relationship table containing the policy information from the cloud storage according to the information field of the policy data, extract data from the relationship table, map the extracted data to the above-mentioned policy information field based on the phrase relationship tree structure, and store it in the local database;

[0019] The knowledge graph construction module is used to read fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triplets required for the policy knowledge graph, and store the knowledge graph.

[0020] Push module: When users search for policy information on the front end, the policy data is matched according to the keyword field and the industry to which the user's company belongs. The best matching score is calculated for each policy data according to the preset matching algorithm, and the data is sorted according to the score to push the most relevant policy data to the user.

[0021] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0023] Through the above technical scheme, compared with the prior art, the present invention has the following beneficial effects: The present invention provides a data processing method based on knowledge graph, which realizes efficient management and precise service of policy information through automated collection, structured processing, intelligent classification, ontology model construction and personalized matching and push. The method improves the efficiency of data collection and processing, enhances the accuracy of text classification, optimizes data storage and query, improves user experience, and enhances the practicality and adaptability of the system through a flexible task response mechanism, improves the quality and efficiency of policy information processing, thereby helping enterprises and users to better understand and utilize policy information.

[0024] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0026] Figure 1 It is a flowchart of a method provided by an exemplary embodiment of the present invention.

[0027] Figure 2 It is a schematic diagram of the structure of a device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0028] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0029] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0030] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0031] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0032] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0033] To facilitate understanding of this embodiment, a data processing method based on a knowledge graph disclosed in an embodiment of the present invention is first described in detail, as shown in the attached figure. Figure 1 As shown, a data processing method based on knowledge graph includes the following steps:

[0034] S1. Deploy a cloud data extraction program in a cloud environment, wherein the cloud data extraction program is configured to obtain data from at least one designated policy data source, parse the data into a predetermined structured format, and upload the data in the structured format to cloud storage.

[0035] Administrative normative documents generally have a validity period, which is generally three to five years and shall not exceed five years. Based on this, and taking into account the timeliness and availability of policies, the present invention takes the published policies as an example and determines the search time range as 2019-2023. Relying on various official platforms, the present invention uses industry and information technology, development and reform, finance, science and technology, commerce, environmental protection, taxation, market supervision, etc. as core search terms to search for relevant policy information issued within 5 years.

[0036] Furthermore, the implementation of obtaining data includes the following steps: creating a virtual machine instance in a cloud environment and deploying the cloud data extraction program; configuring the cloud data extraction program to enable it to identify and obtain data in a specified policy data source, such as various types of policy data, including policy documents, industry regulations, market supervision information, etc., using web page parsing technology to parse the data in the data source and extract the required information; converting the extracted information into a predetermined structured format, such as CSV or JSON; uploading the data in the structured format to a cloud storage service;

[0037] Furthermore, the implementation of the cloud data extraction program includes the following steps: using Python to write a data extraction script, integrating requests and BeautifulSoup libraries to crawl and parse web page data; using the boto3 library to upload structured data to a cloud storage service, such as AWSS3. The data extraction cycle can be set, and the cloud data extraction program is executed to obtain the latest policy data and update the data in the cloud storage;

[0038] Furthermore, the policy data in the cloud storage service is preprocessed by means of a document representation method. The text preprocessing includes: removing stop words, punctuation marks, numbers and other non-text content, performing stemming or lemmatization, and converting the text to lowercase, etc., and performing data cleaning, classification and labeling. A combination of regular expressions and manual review is used to clean up redundant data, complete missing values ​​or delete items where missing values ​​are located, and obtain relevant policy data. Cloud storage provides an API interface, allowing users to query the policy data in the cloud storage through the API.

[0039] S2. Use the pre-trained Transformer model with sentence-level attention layer to encode the policy text, evaluate the importance of each sentence in the entire document, and classify the policy type.

[0040] Furthermore, in the above step S2, a Transformer model of a sentence-level attention layer is added, whose structure includes SentenceEncoder: a Transformer-based encoder for encoding each sentence, SentenceAttentionLayer: a linear transformation is used to map the representation of the sentence to a scalar value representing the importance score of the sentence, and softmax normalization is used to evaluate the importance of the sentence, DocumentRepresentation: a representation of the entire document is constructed by combining sentence-level attention weights, and ClassificationLayer: a document representation is classified.

[0041] Furthermore, specifically for the SentenceEncoder module, its function is to receive a single sentence as input, and output the representation of each sentence, use the Transformer encoder to encode each sentence, and extract the feature representation of each word. The implementation example is as follows: receive the word sequence in the sentence, map each word to a dense vector space through token embedding, use the pre-trained word embedding model or the word embedding layer of the Transformer model to convert each word into a high-dimensional vector; add position information to the word at each position through positional encoding; enable the word at each position to interact with the words at other positions through the multi-head self-attention mechanism, so as to capture the dependencies within the sentence; further transform the word representation through the feedforward neural network to increase the expressive power of the model; ensure that the outputs of the multi-head self-attention layer and the feedforward neural network layer are in the same scale range through layer normalization; finally, add the input directly to the outputs of the multi-head self-attention layer and the feedforward neural network layer through residual connection. The SentenceEncoder module is implemented through the above steps, and then each sentence is efficiently encoded, and the feature representation of each word is extracted to provide high-quality sentence representation for the subsequent SentenceAttentionLayer.

[0042] Furthermore, specifically for the SentenceAttentionLayer module, its function is to evaluate the importance of each sentence in the entire document. This module receives the sentence representation processed by SentenceEncoder and calculates the importance weight of each sentence. The specific implementation steps are as follows: First, receive the sentence representation processed by SentenceEncoder, which contains the feature information of each word; generate an unnormalized weight for each sentence through a linear transformation (LinearTransformation), which maps the representation of each sentence to a scalar value; then, normalize the generated unnormalized weight through the softmax function to obtain the weight distribution of each sentence. The weight distribution reflects the relative importance of each sentence in the entire document. The above steps realize the quantification of the importance of each sentence.

[0043] Furthermore, specifically for the DocumentRepresentation module, its function is to construct a representation of the entire document in combination with the attention weights at the sentence level. Among them, document representation refers to the process of converting natural language text into a numerical form or a structured form that can be processed by a computer. Document representation uses mathematical methods to analyze and process text data, which is an important step in natural language processing (NLP). This module receives the sentence representations and their corresponding weights calculated from SentenceAttentionLayer, and performs weighted averaging on the sentence representations according to these weights to generate a comprehensive representation of the document. The specific implementation steps are as follows: First, receive the sentence representations and their corresponding weights, which are calculated by SentenceAttentionLayer based on the importance of each sentence; second, multiply each sentence representation by its corresponding weight and then perform a sum operation to obtain a comprehensive representation of the document.

[0044] Furthermore, specifically for the ClassificationLayer module, its function is to classify the document representation to predict the category to which the document belongs. This module receives the comprehensive document representation generated by the DocumentRepresentation module, maps it to the category space, and generates the probability distribution of each category. The specific implementation steps are as follows: First, receive the comprehensive document representation generated by the DocumentRepresentation module, and map the document representation to the category space through a fully connected layer (FullyConnectedLayer). The fully connected layer is responsible for converting the document representation into unnormalized scores for each category; then, the generated unnormalized scores are normalized by the softmax function to obtain the probability distribution of each category. The probability distribution reflects the possibility that the document belongs to each category. This module realizes the classification prediction of documents through the above steps and generates the probability distribution of each category, thereby providing users with accurate document classification results and improving the performance of the overall model.

[0045] S3, the user calls and parses the data in the cloud storage, searches the relationship table containing the policy information from the cloud storage according to the information field of the policy data, extracts data from the relationship table, maps the extracted data to the above policy information field based on the phrase relationship tree structure, and stores it in the local database;

[0046] Further, the user searches for a table or list containing policy information from the cloud storage according to information fields of the policy data, such as: unique number of the policy data, issuing region (province, city, district), issuing unit, policy title, policy category, and issuing time, extracts data of each row from the table or list, maps data of each row to the above policy information fields based on the syntactic features extracted from the grammatical structure of the document (Phrase Relationship Tree) and the grammatical relationship tree (PhraseStructureTrees), and stores the data in the local database;

[0047] Furthermore, with respect to the policy information of the present invention, the steps of constructing the phrase relationship tree are specifically as follows:

[0048] Step 1. Text preprocessing, removing unnecessary whitespace characters and splitting the text into separate lines.

[0049] Step 2. Identify key fields and identify each key field and its corresponding value. Identify the following fields: policy number, issuing region, issuing unit, policy title, policy category, issuing time, etc.

[0050] Step 3. Build a phrase relationship tree. Building a phrase relationship tree can better understand the structure of the text and extract key information from it. The steps of building a phrase relationship tree include creating a root node to represent the entire text, dividing the text into multiple clauses, each clause contains a field and its corresponding value; creating a child node for each clause and making it a child node of the root node; creating child nodes for the fields and values ​​in each clause and making them child nodes of the corresponding child nodes. The following is a specific example of a phrase relationship tree:

[0051] PolicyText2

[0052] └──Policy Number:2023-0013

[0053] └──Region:XX City 4

[0054] └──Unit:XX City Agency 5

[0055] └──Title: Notice on strengthening urban greening construction 6

[0056] └──Category: Urban Planning 7

[0057] └──PublishTime:2023-01-15

[0058] After the phrase relationship tree is built, each information field and its corresponding value can be extracted according to the field. Finally, the extracted information is stored in a key-value database. For example, Redis can be used locally as a key-value database. We can use the phrase relationship tree to parse and extract key information from the policy text and store it in the key-value database for subsequent retrieval and processing.

[0059] S4. Read the fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triples required for the policy knowledge graph, and store the knowledge graph.

[0060] Furthermore, in step S4, by reading the relationship tree field in the local data, the policy entities, attributes and relationships are defined and extracted to construct a policy ontology model. Among them, ontology is a formal description of concepts, entities and the relationships between them in a defined field, which is expressed in a structured manner. Entities are things or concepts in the real world, generally appearing in the form of proper nouns. Entity recognition is the process of identifying entity character fragments from character sequences. Entity definition is an important node in the knowledge graph. In the policy ontology, entities refer to metadata such as policy documents, policy categories, and policy issuing agencies. They are also related policies mentioned in policy documents, or abstract policy concepts. In the present invention, entities include seven entity types: science and technology policies, categories, agencies, related policies, regions, provinces, and cities.

[0061] Furthermore, in the policy ontology, the attribute definition is specifically to classify the funding, regulatory punishment, and honorary qualification clauses as the entity attributes of the policy. The purpose is to help small and medium-sized enterprises make good use of policy dividends and avoid risks and policy forbidden areas. For example, the policy text is divided into types according to the sentence level, specifically into funding, regulatory punishment, and honorary qualification types; if negative words such as "strictly prohibited" and "rectification" appear in the sentence, or if there are obvious punitive expressions, it is classified as a regulatory punishment type; on the contrary, if the content contains more positive words, such as "subsidy", "support", "encouragement", etc., it is classified as a funding type; if it involves keywords such as "qualification" and "qualification", it is classified as an honorary qualification type.

[0062] Furthermore, entity relationship refers to the association relationship between two entities, including grammatical relationship, semantic relationship, etc. Relation extraction is to determine the relationship between target entity pairs based on context information. According to the available context, relationship extraction is divided into sentence-level relationship extraction and document-level relationship extraction. Among them, sentence-level relationship extraction requires that entity pairs appear in the same sentence, while entity pairs extracted at the document level can be distributed in multiple sentences. The entity relationship definition in the present invention includes policy relationships such as "publish", "reference", "include", "belong to", "invest", "apply to", and "research and development"; after the above processing steps, the entity, attribute and relationship triples required for the policy knowledge graph are obtained.

[0063] In order to construct the policy knowledge graph, a relationship mapping table is created through the entity type and relationship type definitions in the above policy text. Then, triples (Subject, Predicate, Object) are extracted from the policy text and converted into the format required by PropertyGraphModel, that is, the triples are converted into nodes and edges, and necessary attributes are added to each node and edge, thus completing the construction of the policy knowledge graph.

[0064] Optionally, you can use the trained deep learning model to perform entity recognition and relationship extraction on new text data. Next, build a graph data structure, select each word as a node, define the edges between nodes to represent word order relationships or relationships between entities, and define features for each edge, such as part-of-speech tags, relationship tags, etc. Store the converted node and edge data as CSV files, where the node file contains attributes. Finally, select the graph data node ID, label, and attribute, the edge file contains the source node ID, target node ID, relationship label, and library (such as JanusGrap or ArangoDB), and use the corresponding tools or scripts to batch import the CSV files into the database to complete the construction of the policy knowledge graph.

[0065] Furthermore, the machine learning model is trained with labeled data sets to automatically identify and extract entities, attributes, and relationships. Machine learning methods that can be used include: using labeled data to train classifiers or sequence labeling models such as conditional random fields (CRFs) to identify entity boundaries and types, as well as relationships between entities. The above models can directly learn features from the original text and perform extraction tasks.

[0066] In addition, the extraction of policy entities, attributes and relationships can also be performed in other ways. In the present invention, the following methods can be used for extraction:

[0067] Optionally, entities, attributes, and relations are identified and extracted based on predefined rules or patterns. Rules are designed based on specific vocabulary, grammatical structure, or contextual information. For example, pattern matching can be used to identify key phrases or sentences in policy text that contain entity, attribute, or relation information.

[0068] Optionally, use existing natural language processing tools and libraries (such as spaCy, NLTK, StanfordNLP, etc.) for tasks such as entity recognition (NER) and relation extraction (RE).

[0069] S5. When users search for policy information on the front end, the policy data is matched according to the keyword field and the industry to which the user's company belongs. A best matching score is calculated for each policy data according to the preset matching algorithm, and the data is sorted according to the score to push the most relevant policy data to the user.

[0070] When the user calls it on the front end, the keyword is the search term provided by the user, which is used to match the policy title, content and other text fields. The industry to which the user's company belongs is the industry classification of the user's company, which is used to match the industry to which the policy applies. Policy data includes information such as policy title, content, release date, industry, and release region.

[0071] Further, specifically, a best matching score is calculated for each policy data according to a preset matching algorithm. For each policy data, based on the frequency and position of the keywords in the policy title and content, the TF-IDF values ​​(term frequency-inverse document frequency) of the keywords in the title and content are counted, and the sum is calculated to obtain the keyword matching score;

[0072] The present invention uses the TF-IDF (TermFrequency-InverseDocumentFrequency) value to quantify the importance of keywords. Based on the keyword matching score dimension, the example calculation formula is as follows:

[0073]

[0074] Among them, TFi is the word frequency of keyword i in the document, IDFi is the inverse document frequency of keyword i, which is the reciprocal of the number of documents containing keyword i among all documents, and is used to measure the prevalence and distinctiveness of keywords.

[0075] Further, calculation based on the industry matching dimension: based on the similarity between the user's enterprise and the policy applicable industry, compare the matching degree between the industry to which the user's enterprise belongs and the policy applicable industry. If they are completely consistent, full marks will be awarded, otherwise the similarity score will be calculated according to the industry classification system. In the present invention, it is defined that if the industry to which the user's enterprise belongs is completely consistent with the policy applicable industry, the matching degree is 1; if there is partial consistency or a certain degree of correlation, the matching degree is between 0 and 1; if there is no consistency, the matching degree is 0. For example, the hierarchy or semantic similarity in the industry classification system can be used for calculation. When using the hierarchy, the similarity is calculated based on the distance of the industry classification tree; when using semantic similarity, a pre-trained word embedding model (such as Word2Vec or BERT) is used to calculate the similarity between industry terms.

[0076] An example of how the calculations might be done using a hierarchy is as follows:

[0077]

[0078] Among them, d is the shortest distance between the industry to which the user enterprise belongs and the industry to which the policy applies in the classification tree.

[0079] Further, based on the time matching dimension calculation: calculate the time difference between the policy release date and the current date, and convert it into a score. The most recently released policy has the highest score, and the score decreases over time. In the present invention, it is defined that the most recently released policy has the highest score, and the score gradually decreases as the release time passes.

[0080] The calculation method is as follows: Use an exponential decay function to calculate the time matching degree:

[0081]

[0082] Among them, t is the time difference between the current time and the policy release time (in days), and α is the attenuation coefficient, which can be adjusted according to actual needs.

[0083] Further, based on the regional matching dimension calculation: refer to the industry matching calculation method, if the user's concerned area is consistent with the policy release area, full marks will be awarded; otherwise, the score will be based on the degree of geographical proximity; if the user's concerned area is consistent with the policy release area, the matching degree is 1; if there is partial consistency or a certain degree of correlation, the matching degree is between 0 and 1; if there is no consistency, the matching degree is 0. The similarity is calculated by using the distance calculation in the geographic information system (GIS) or the hierarchical relationship of administrative regions. The distance calculation method is used, and the example is as follows:

[0084]

[0085] Among them, v is the geographical distance between the user's attention area and the policy release area, and β is the distance influence coefficient, which can be adjusted according to actual needs.

[0086] Assign a weight to each of the above dimensions. The weight can be adjusted and optimized according to the actual situation. The keyword matching weight is , the industry matching weight is , the time matching weight is , the regional matching weight is For each policy data, the scores of the four dimensions are obtained according to the matching calculation function defined above, and the final matching score is calculated using the weighted summation method. The matching score is specifically:

[0087]

[0088] Among them, the keyword matching weight is , the industry matching weight is , the time matching weight is , the regional matching weight is ; TFIDF (TermFrequency-InverseDocumentFrequency) word frequency-inverse document frequency value is used to measure the importance of keywords in policy titles and contents; IndustrySim industry matching measures the similarity between the industry to which the user's company belongs and the industry to which the policy applies; TimeDiff measures the proximity between the policy release time and the current time; RegionMatch regional matching measures the similarity between the user's focus area and the policy release area.

[0089] Using the above algorithm, we can calculate the policy data matching score example: Assume that the TF-IDF value of the keywords in the policy title and content is 0.8, the similarity between the industry to which the user's company belongs and the industry to which the policy applies is 0.7, the policy release time is 30 days away from the current time, the decay coefficient ɑ=0.05, the similarity between the user's focus area and the policy release area is 0.9, and the weights are set as follows: =0.4, =0.3, =0.2, =0.1; TFIDFkw=0.8, IndustrySim=0.7, TimeDiff=e-0.05*30≈0.223, RegionMatch=0.9.

[0090] The final matching score calculation process is as follows:

[0091] Score=0.4*0.8+0.3*0.7+0.2*0.223+0.1*0.9=0.6646

[0092] Through calculation, an optimal matching score is obtained for each policy, and the policy data is sorted in descending order according to the calculated matching score, and finally the top N most relevant policy data are pushed to the user.

[0093] The present invention realizes efficient management and precise service of policy information through automated collection, structured processing, intelligent classification, ontology model construction and personalized matching and push, which can help users and enterprises obtain policy information, major supported key areas and subsidy standards involved in the policy, seize opportunities, comply with policy orientation and realize precise matching between supply and demand.

[0094] Second, as attached Figure 2 As shown, an embodiment of the present invention also provides a data processing device based on a knowledge graph, including: a data acquisition module 201, used to deploy a cloud data extraction program in a cloud environment, the cloud data extraction program is configured to obtain data from at least one specified policy data source, and parse the data into a predetermined structured format, and upload the data in the structured format to cloud storage.

[0095] A classification module 202, for encoding the policy text using a pre-trained Transformer model with a sentence-level attention layer added, evaluating the importance of each sentence in the entire document, and classifying the policy type;

[0096] Processing module 203, used for users to call and parse data in cloud storage, find a relation table containing policy information from cloud storage according to the information field of policy data, extract data from the relation table, map the extracted data to the above policy information field based on the phrase relationship tree structure, and store it in the local database;

[0097] The knowledge graph construction module 204 is used to read the fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triples required for the policy knowledge graph, and store the knowledge graph.

[0098] Push module 205, when the user retrieves policy information on the front end, the policy data is matched according to the keyword field and the industry to which the user's company belongs, and a best matching score is calculated for each policy data according to a preset matching algorithm, and the data is sorted according to the score to push the most relevant policy data to the user.

[0099] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0100] The core of the invention The present invention provides a data processing system based on a knowledge graph, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program and implements the data processing method based on the knowledge graph as described above.

[0101] A second aspect of the present invention provides a method comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program and implements the knowledge graph-based data processing method as described in the aforementioned embodiments.

[0102] In addition to the above-mentioned methods and devices, the embodiments of the present invention are also a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the target detection method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0103] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limiting. The above details do not limit the present disclosure to be implemented by adopting the above specific details.

[0104] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0105] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0106] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0107] It should also be noted that in the apparatus, equipment and method of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present disclosure. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown here, but to the widest scope consistent with the principles and novel features disclosed herein.

[0108] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A data processing method based on knowledge graph, characterized in that: The following steps are involved: S1. Deploy a cloud data extraction program in a cloud environment, wherein the cloud data extraction program is configured to obtain data from at least one designated policy data source, parse the data into a predetermined structured format, and upload the structured format data to cloud storage; S2. Use a pre-trained Transformer model with a sentence-level attention mechanism to encode the policy text, evaluate the importance of each sentence in the entire document, and classify the policy type; S3. The user calls and parses the data in the cloud storage, searches for a relational table containing policy information from the cloud storage according to the information field of the policy data, extracts data from the relational table, maps the extracted data to the above-mentioned policy information field based on the phrase relationship tree structure, and stores it in a local database; S4. Read the fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triples required for the policy knowledge graph, and store the knowledge graph; S5. When the user retrieves policy information on the front end, match the policy data according to the keyword field and the industry to which the user's company belongs, calculate an optimal matching score for each policy data according to the preset matching algorithm, sort according to the score, and push the most relevant policy data to the user; wherein, in the step S2, the Transformer model with sentence-level attention mechanism added specifically includes: SentenceEncoder: an encoder based on Transformer to encode each sentence; SentenceAttentionLayer: a linear transformation is used to convert the sentence Representation is mapped to a scalar value representing the importance score of the sentence, and softmax normalization is used to evaluate the importance of the sentence; DocumentRepresentation: combines the sentence-level attention weights to construct the representation of the entire document; ClassificationLayer: classifies the document representation; wherein, in step S3, the construction steps of the phrase relationship tree include initializing the root node, creating a root node representing the entire text, dividing the text into multiple clauses, each clause containing a field and its corresponding value; creating a child node for each clause and using it as a child node of the root node; creating child nodes for the fields and values ​​in each clause respectively, and using them as child nodes of the corresponding child nodes. After the phrase relationship tree is constructed, each information field and its corresponding value are extracted according to the field, and the extracted information is stored in the local database.

2. The method according to claim 1, characterized in that: The step S1 also includes preprocessing the extracted policy data, which includes: removing stop words, punctuation marks, and numbers, performing stem extraction and word form restoration, and performing data cleaning, classification, and labeling processing, using a combination of regular expressions and manual review to clean up redundant data, complete missing values ​​or delete items where missing values ​​are located, and obtain the policy data.

3. The method according to claim 1, characterized in that: In the step S4, by reading the relationship tree field in the local data, the policy entities, attributes and relationships are defined to construct a policy ontology model; the entities include seven entity types: science and technology policies, categories, institutions, related policies, regions, provinces, and cities; the attributes are defined as classifying the funding, regulatory punishment, and honorary qualification clauses as entity attributes of the policy; the relationships are defined as including the reference relationships between policies, as well as the relationships of categories, releases, and affiliations; after the above processing steps, the entity, attribute, and relationship triples required for the policy knowledge graph are obtained.

4. The method according to claim 1, characterized in that: In step S4, structured data is read from cloud storage and loaded into a local computing environment, and pattern matching is used to identify key phrases or sentences in the policy text, thereby performing entity recognition and relationship extraction on the text data.

5. The method according to claim 1, characterized in that: In step S5, the matching score is calculated as follows: Among them, the keyword matching weight is , the industry matching weight is , the time matching weight is , the regional matching weight is , TDIDFkw word frequency-inverse document frequency value is used to measure the importance of keywords in policy titles and contents; IndustrySim industry matching measures the similarity between the industry to which the user's enterprise belongs and the industry to which the policy applies; TimeDiff time matching measures the proximity between the policy release time and the current time; RegionMatch regional matching measures the similarity between the area that the user is concerned about and the area where the policy is released.

6. A data processing device based on a knowledge graph, comprising: A data acquisition module, used to deploy a cloud data extraction program in a cloud environment, wherein the cloud data extraction program is configured to obtain data from at least one specified policy data source, parse the data into a predetermined structured format, and upload the data in the structured format to cloud storage; The classification module is used to encode the policy text using a pre-trained Transformer model with sentence-level attention mechanism, evaluate the importance of each sentence in the entire document, and classify the policy type; A processing module, used to call and parse the data in the cloud storage, search the relationship table containing the policy information from the cloud storage according to the information field of the policy data, extract data from the relationship table, map the extracted data to the above-mentioned policy information field based on the phrase relationship tree structure, and store it in the local database; The knowledge graph construction module is used to read the fields in the local relationship tree database, define and extract policy-related entities, attributes and relationships, build a policy ontology model, obtain the entity, attribute and relationship triples required for the policy knowledge graph, and store the knowledge graph; The push module, when the user retrieves policy information on the front end, matches the policy data according to the keyword field and the industry to which the user's company belongs, calculates an optimal matching score for each policy data according to the preset matching algorithm, sorts them according to the score, and pushes the most relevant policy data to the user; wherein, the Transformer model with sentence-level attention mechanism added specifically includes: SentenceEncoder: Transformer-based encoder that encodes each sentence; SentenceAttentionLayer: Uses linear transformation to map the representation of a sentence to a scalar value representing the importance score of the sentence, and uses softmax normalization to evaluate the importance of the sentence; DocumentRepresentation: Combines sentence-level attention weights to construct a representation of the entire document; ClassificationLayer: classify document representations; Among them, the steps of constructing the phrase relationship tree include initializing the root node, creating a root node representing the entire text, dividing the text into multiple clauses, each clause containing a field and its corresponding value; creating a child node for each clause and using it as a child node of the root node; creating child nodes for the fields and values ​​in each clause and using them as child nodes of the corresponding child nodes. After the phrase relationship tree is constructed, each information field and its corresponding value are extracted according to the field, and the extracted information is stored in the local database.

7. An electronic device, characterized in that: The electronic device comprises: a memory and a processor, wherein the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 5.

Images