Government affair information recommendation method and device based on knowledge graph and multi-mode fusion

Abstract

The embodiment of the invention discloses a knowledge graph and multi-modal fusion government affair information recommendation method and equipment, belongs to the technical field of information recommendation, and solves the problem that the matching degree between a recommendation result and a demand is relatively low when a user obtains government affair information. Comprising the steps of generating a user preference vector based on a user historical retrieval behavior, and performing coarse-grained retrieval on a government affair knowledge graph to obtain candidate knowledge sub-graphs; fusing the candidate knowledge sub-graphs with multi-modal data uploaded by the user to obtain a multi-modal semantic vector; performing multi-hop information aggregation processing on the multi-modal semantic vector, and performing fine-grained retrieval based on an aggregated graph node vector and a user query vector; based on a reordering mechanism, reordering the retrieval results, and determining recommended government affair knowledge sub-graphs; and based on the multi-modal data and the recommended government affair knowledge subgraph, performing visual question and answer and visual common sense reasoning processing to obtain a multi-modal reasoning result, and based on user requirements, sending the multi-modal reasoning result as recommended content to the user.

Description

Technical Field

[0001] The present application relates to the field of information recommendation technology, and in particular to a method and device for recommending government information by integrating knowledge graph and multimodality. Background Art

[0002] With the rapid development of digital government services, massive government information such as policies and regulations, service guidelines, case data, etc. is growing exponentially. The public is facing serious difficulties in information screening. The recommendation system, as the core technology of information filtering, has become a key means to optimize the government service experience and improve service efficiency.

[0003] In the existing technology, large language models are widely used to analyze users' government affairs consultations because they have certain semantic understanding and generation capabilities. However, in actual applications, large language models frequently suffer from hallucination problems. In the absence of real data support, they generate government affairs information that is inconsistent with the facts, logically incorrect, or non-existent, affecting the user's service process.

[0004] Secondly, government systems usually include various types of data such as pictures of service materials and policy briefing videos, which are very different from text information modalities. It is difficult to accurately understand multiple types of government data through large-scale language models, and thus it is difficult to accurately grasp the user's multimodal consultation intentions, resulting in a low match between recommendation results and demand. Summary of the invention

[0005] The embodiments of the present application provide a method and device for recommending government information by integrating knowledge graph and multimodality, which is used to solve the following technical problems: In the prior art, when users obtain government information, they frequently encounter hallucination problems of large language models and find it difficult to accurately understand various types of government data, resulting in a low match between the recommendation results and the needs.

[0006] The present application embodiment adopts the following technical solutions: An embodiment of the present application provides a government information recommendation method that integrates a knowledge graph and multimodality. It includes constructing a government affairs knowledge graph based on government entity information, government relationship information, and government attribute information; in response to a user's government information retrieval request, generating a user preference vector based on the user's historical retrieval behavior, and performing a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; performing semantic encoding on the candidate knowledge subgraph to generate an initial semantic vector, and extracting text feature vectors from the multimodal data uploaded by the user, and fusing the initial semantic vector and the text feature vector to obtain a multimodal semantic vector; performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained retrieval based on the aggregated graph node vector and the user query vector; based on a re-ranking mechanism, re-ranking the retrieval results to determine a recommended government affairs knowledge subgraph; based on the multimodal data and the recommended government affairs knowledge subgraph, performing visual question answering and visual common sense reasoning processing to obtain a multimodal reasoning result, and sending the multimodal reasoning result as recommended content to the user based on the user's needs.

[0007] In an embodiment of the present application, a candidate subgraph is quickly located from the knowledge graph through coarse-grained retrieval, reducing the data range and subsequent calculation amount, and improving the recommendation response speed. The candidate knowledge subgraph is semantically encoded into a vector and fused with the text feature vectors of multimodal data such as images and videos uploaded by the user to realize the associated understanding of text and image information, improve the understanding accuracy of complex government affairs problems, and avoid the one-sidedness of unimodal information. Secondly, in an embodiment of the present application, a fine-grained retrieval is performed by combining the aggregated graph node vector and the user query vector to ensure that the recommended content deeply matches the user's needs and improve the accuracy of policy interpretation and process guidance. And re-rank by comprehensively considering factors such as semantic relevance and timeliness to ensure that current and effective policies and the latest handling procedures are preferentially displayed in the recommended results, improving the intelligence and interaction efficiency of government services.

[0008] In an implementation manner of the present application, generating a user preference vector based on the user's historical retrieval behavior and performing a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph specifically includes: constructing a user retrieval interaction matrix based on the user's historical retrieval behavior, and determining the association strength between the user and the government affairs knowledge entity through the user retrieval interaction matrix, and generating an initial preference vector based on the association strength; assigning a time decay factor to the government affairs knowledge entity according to the time sequence of the user's historical retrieval, and performing weighted aggregation on different types of government affairs knowledge entities based on the user's historical retrieval type; calculating and adjusting the initial preference vector through the time decay factor and weighted aggregation to obtain a user preference vector; determining seed nodes based on the calculation scores corresponding to the user preference vector; and constructing a candidate knowledge subgraph with the seed nodes as the center through the edge relationship of the knowledge graph.

[0009] In an implementation manner of the present application, semantic encoding is performed on the candidate knowledge subgraph to generate an initial semantic vector, and text feature vector extraction is performed on the multimodal data uploaded by the user, specifically including: encoding the entity text and relationship text in the candidate knowledge subgraph respectively through a pre-trained language model to generate an initial semantic vector; classifying the multimodal data uploaded by the user to determine audio data and image data; converting the audio data into text content; and taking screenshots of key frames for the image data and extracting text content from the key frames; inputting the text content into the pre-trained language model to obtain a text feature vector.

[0010] In an implementation manner of the present application, multi-hop information aggregation processing is performed on the multimodal semantic vector, and fine-grained retrieval is performed based on the aggregated graph node vector and the user query vector, specifically including: using the multimodal semantic vector as the graph node feature; determining the feature propagation between each graph node feature through a graph convolutional network and aggregating multi-hop neighbor information through multiple layers of iteration; during the aggregation process, adding positional encoding to distinguish the hierarchical relationship of each node in the candidate knowledge subgraph and integrating the topological structure features corresponding to the distinguished candidate knowledge subgraph into the node vector to obtain a graph node vector; calculating the cosine similarity between the user query vector and the graph node vector, and performing fine-grained retrieval in the candidate knowledge subgraph based on the similarity result.

[0011] In an implementation manner of the present application, based on a re-ranking mechanism, the retrieval results are re-ranked to determine a recommended government affairs knowledge subgraph, specifically including: determining an initial semantic score based on cosine similarity; determining the number of hops between a node and the query core entity, and determining a structural association score based on the number of hops; determining the time difference between the release time of the node information and the current retrieval time, and obtaining a time factor score based on the time difference; determining a user behavior score based on the user's historical retrieval data; where the user's historical retrieval data at least includes historical click data, historical favorite data, and historical comment data; re-ranking the retrieval results based on preset weight parameters, the initial semantic score, the structural association score, the time factor score, and the user behavior score to determine a recommended government affairs knowledge subgraph.

[0012] In an implementation manner of the present application, based on preset weight parameters, the initial semantic score, the structural association score, the time factor score, and the user behavior score, the retrieval results are re-ranked, specifically including: based on the function: ; obtaining the comprehensive scores corresponding to each core entity in the candidate knowledge subgraph respectively; sorting each core entity in descending order based on the comprehensive scores to determine a recommended government affairs knowledge subgraph; where is the comprehensive score function; is the dynamic adjustment coefficient; is the first preset weight coefficient; is the second preset weight coefficient; is the third preset weight coefficient; is the fourth preset weight coefficient; is the initial semantic score; is the structural association degree score; is the time factor score; is the user behavior score; is the non - linear adjustment threshold.

[0013] In an implementation manner of the present application, based on the comprehensive score, each core entity is sorted in descending order to determine the recommended government affairs knowledge sub - graph, specifically including: based on the retrieval results sorted in descending order, the first K nodes and the associated sub - graphs corresponding to the first K nodes are selected as the candidate set; based on the function: ; the candidate set is screened by secondary indicators to obtain the final ranking, and based on the final ranking, the recommended government affairs knowledge sub - graph is obtained; where is the secondary screening function; is the balance coefficient; is the secondary indicator.

[0014] In an implementation manner of the present application, based on the multi - modal data and the recommended government affairs knowledge sub - graph, visual question - answering and visual common - sense reasoning processing are performed to obtain multi - modal reasoning results, specifically including: based on the text feature vector, the current retrieval task is divided into a question - answering basic task and a common - sense reasoning task; in the case of belonging to the question - answering basic task, the multi - modal data and the recommended government affairs knowledge sub - graph are input into a preset question - answering model, and through the preset question - answering model, the question information corresponding to the multi - modal data is determined, and based on the question information, a search is performed in the recommended government affairs knowledge sub - graph to obtain the recommended information; the recommended information is cross - verified with the structured information in the recommended government affairs knowledge sub - graph, and in the case of passing the verification, it is sent to the user.

[0015] In an implementation manner of the present application, after dividing the current retrieval task into a question - answering basic task and a common - sense reasoning task based on the text feature vector, the method further includes: in the case of belonging to the common - sense reasoning task, the multi - modal data and the recommended government affairs knowledge sub - graph are input into a preset common - sense reasoning model, and through the preset common - sense reasoning model, the question information corresponding to the multi - modal data is determined, and based on the question information, relevant government affairs common - sense knowledge is called in the knowledge base, so as to perform reasoning based on the relevant government affairs common - sense knowledge and the recommended government affairs knowledge sub - graph to obtain the recommended information.

[0016] An embodiment of the present application provides a government affairs information recommendation device that integrates a knowledge graph and multi-modalities, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: construct a government affairs knowledge graph based on government affairs entity information, government affairs relationship information, and government affairs attribute information; in response to a user's government affairs information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge sub-graph; perform semantic encoding on the selected knowledge sub-graph to generate an initial semantic vector, and extract a text feature vector from the multi-modal data uploaded by the user, and fuse the initial semantic vector and the text feature vector to obtain a multi-modal semantic vector; perform multi-hop information aggregation processing on the multi-modal semantic vector, and perform a fine-grained retrieval based on the aggregated graph node vector and the user query vector; based on a re-ranking mechanism, re-rank the retrieval results to determine a recommended government affairs knowledge sub-graph; based on the multi-modal data and the recommended government affairs knowledge sub-graph, perform visual question answering and visual common sense reasoning processing to obtain a multi-modal reasoning result, and send the multi-modal reasoning result to the user as recommended content based on the user's needs.

[0017] The above at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects: The embodiment of the present application quickly locates candidate sub-graphs from the knowledge graph through coarse-grained retrieval, narrows the data range, reduces the subsequent calculation amount, and improves the recommendation response speed. The candidate knowledge sub-graph is semantically encoded into a vector and fused with the text feature vector of multi-modal data such as images and videos uploaded by the user, realizing the associated understanding of text and image information, improving the understanding accuracy of complex government affairs problems, and avoiding the one-sidedness of single-modal information. Secondly, the embodiment of the present application combines the aggregated graph node vector and the user query vector for fine-grained retrieval to ensure that the recommended content deeply matches the user's needs and improves the accuracy of policy interpretation and process guidance. And re-rank by comprehensively considering factors such as semantic relevance and timeliness to ensure that current and effective policies and the latest handling processes are preferentially displayed in the recommended results, improving the intelligence and interaction efficiency of government affairs services. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings: Figure 1 It is a flowchart of a government affairs information recommendation method that integrates a knowledge graph and multi-modalities provided by an embodiment of the present application; Figure 2 This is a schematic structural diagram of a government affairs information recommendation device that integrates a knowledge graph and multi-modalities provided by an embodiment of the present application.

[0019] Reference numerals: 200: Government affairs information recommendation device that integrates a knowledge graph and multi-modalities; 201: Processor; 202: Memory. Specific implementation manners

[0020] An embodiment of the present application provides a method and device for recommending government affairs information by integrating a knowledge graph and multi-modalities.

[0021] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0022] The technical solutions proposed in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of a method for recommending government affairs information by integrating a knowledge graph and multi-modalities provided by an embodiment of the present application. As Figure 1 shown, the method for recommending government affairs information by integrating a knowledge graph and multi-modalities includes the following steps: Step 101: Construct a government affairs knowledge graph based on government affairs entity information, government affairs relationship information, and government affairs attribute information.

[0024] In one implementation manner of the present application, government affairs entity, government affairs relationship, and government affairs attribute information are collected through channels such as government affairs public platforms, department database interfaces, publicly available government affairs documents (such as policies and regulations, service guides, government affairs service system logs, etc.). Taking government affairs entities as graph nodes, relationships as edges, and attribute information as additional attributes of nodes or edges, a government affairs knowledge graph is constructed. Among them, government affairs entities can be government departments and agencies, government affairs service items, government affairs personnel information, etc.

[0025] Step 102: In response to a user's government affairs information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge sub-graph.

[0026] In an implementation manner of the present application, in response to a user's government affairs information retrieval request, based on the user's historical retrieval behavior, a user retrieval interaction matrix is constructed, and through the user retrieval interaction matrix, the association strength between the user and government affairs knowledge entities is determined, and an initial preference vector is generated based on the association strength. According to the time series of the user's historical retrievals, a time decay factor is assigned to the government affairs knowledge entities, and, based on the types of the user's historical retrievals, different types of government affairs knowledge entities are weighted and aggregated. Through the time decay factor and weighted aggregation, the initial preference vector is calculated and adjusted to obtain the user preference vector. Based on the calculated scores corresponding to the user preference vector, seed nodes are determined, and with the seed nodes as the center, candidate knowledge subgraphs are constructed through the edge relationships of the knowledge graph.

[0027] Specifically, historical retrieval behavior data of the user on the government affairs service platform is collected, and the retrieval content in the user behavior data is mapped to the entities in the government affairs knowledge graph. A two-dimensional sparse matrix is constructed with the user ID as the row and the government affairs knowledge entities as the column. If the user has behaviors such as retrieval and click on a certain entity, an interaction record is marked at the corresponding matrix position. Based on the interaction data recorded in the interaction matrix, a basic weight is assigned, and the initial association score between the user and each entity is calculated by weighted summation. The association score is normalized to obtain the association strength, and the association strength is used as vector elements and arranged in the column order of the government affairs knowledge entities in the matrix to form the initial preference vector.

[0028] Furthermore, according to the time series of the user's historical retrievals, a time decay factor is assigned to each government affairs knowledge entity, and, the government affairs knowledge entities are divided into different types, such as policy and regulation type, handling process type, case reference type, and weights are set for each type of entity according to the distribution of the user's historical retrieval types. Each element in the initial preference vector is multiplied by the corresponding time decay factor and type weight to obtain the adjusted scores.

[0029] Furthermore, the adjusted scores are recombined into a new vector, which is the final user preference vector. This vector comprehensively considers the timeliness and preference tendency of the user's behavior and more accurately reflects the user's current needs. The scores of each element in the user preference vector are calculated, and the top-K entities with the highest scores are selected as the seed nodes. With the seed nodes as the center, 1-2 hop neighbor nodes are expanded outward through the edge relationships of the knowledge graph. For example, starting from the "Enterprise Social Insurance Payment Process" node, directly related nodes such as "Social Insurance Payment Materials" and "Social Insurance Handling Agencies" are expanded, and then further expanded to indirect nodes such as "Material Template Download Link" and "Agency Contact Information", finally forming a set of candidate knowledge subgraphs containing core information.

[0030] Step 103: Semantically encode the candidate knowledge subgraph to generate an initial semantic vector, and extract text feature vectors from the multimodal data uploaded by the user. Then fuse the initial semantic vector and the text feature vector to obtain a multimodal semantic vector.

[0031] In an implementation manner of this application, the entity text and relationship text in the candidate knowledge subgraph are respectively encoded through a pre-trained language model to generate an initial semantic vector. Classify the multimodal data uploaded by the user to determine the audio data and image data. Convert the audio data into text content, and take screenshots of key frames for the image data and extract text content from the key frames. Input the text content into the pre-trained language model to obtain text feature vectors.

[0032] Specifically, build a language model through RoBERTa (Robustly Optimized BERT Pretraining Approach), train it with text data in the government affairs field to obtain a qualified language model. Use this language model to encode the entity text and relationship text in the candidate knowledge subgraph respectively to generate an initial semantic vector. Divide the multimodal data uploaded by the user into audio data and image data. If it is audio data, convert the audio content into text content. If it is image data, define the frame with text as the key frame and extract the text content from the key frame. Input the text content extracted from the multimodal data into the trained language model to obtain text feature vectors. Concatenate the initial semantic vector and the text feature vector to obtain a multimodal semantic vector.

[0033] Step 104: Perform multi-hop information aggregation processing on the multimodal semantic vector, and perform fine-grained retrieval based on the graph node vector and the user query vector after the aggregation processing.

[0034] In an implementation manner of this application, use the multimodal semantic vector as the graph node feature. Determine the feature propagation between each graph node feature through a graph convolutional network, and aggregate multi-hop neighbor information through multiple layers of iteration. During the aggregation process, add position encoding to distinguish the hierarchical relationship of each node in the candidate knowledge subgraph, and integrate the topological structure features corresponding to the distinguished candidate knowledge subgraph into the node vector to obtain a graph node vector. Calculate the cosine similarity between the user query vector and the graph node vector, and perform fine-grained retrieval in the candidate knowledge subgraph based on the similarity result.

[0035] Specifically, the generated multimodal semantic vectors are correspondingly assigned to each node in the candidate knowledge subgraph. For example, if the subgraph contains a node of "Enterprise Income Tax Declaration Process", the multimodal semantic vector related to this node is used as its initial feature vector. Based on the topological structure of the candidate knowledge subgraph, the graph convolutional network model is initialized, and hyperparameters such as the number of network layers and the dimension of the hidden layer are set. In each layer of the GCN (Graph Convolutional Network), for each graph node, according to the features of its neighbor nodes and the connection relationship of the edges, the feature propagation is calculated through the convolution operation. Through the stacking of multiple layers of GCN, multiple rounds of iterative calculations are performed. In each round of iteration, the node not only aggregates the information of the direct neighbors but also indirectly obtains the features of the multi-hop neighbors, gradually enriching its own semantic expression. For example, after 3 layers of GCN, the feature vector of each node will fuse the information of the neighbor nodes within 3-hop range.

[0036] Furthermore, to distinguish the hierarchical relationship of nodes in the subgraph, a position encoding mechanism is introduced. According to the shortest path distance between the node and the query core entity of the subgraph, different position encoding vectors are assigned to each node. The global topological structure features of the candidate knowledge subgraph are extracted. The topological structure feature vector is concatenated with the node feature vector, so that the node vector not only contains its own semantics and neighbor information but also implicitly contains the overall structure features of the subgraph.

[0037] Furthermore, the cosine similarity is calculated one by one between the user query vector generated by the user query and the graph node vectors after the above processing, and fine-grained retrieval is performed in the candidate knowledge subgraph based on the similarity results.

[0038] Step 105: Based on the re-ranking mechanism, re-rank the retrieval results to determine the recommended government affairs knowledge subgraph.

[0039] In an implementation manner of this application, based on the cosine similarity, the initial semantic score is determined, the number of hops between the node and the query core entity is determined, and the structural association score is determined based on the number of hops. The time difference between the release time of the node information and the current retrieval time is determined, and based on the time difference, the time factor score is obtained. The user behavior score is determined based on the user's historical retrieval data; where the user's historical retrieval data at least includes historical click data, historical favorite data, and historical comment data. Based on the preset weight parameters, the initial semantic score, the structural association score, the time factor score, and the user behavior score, the retrieval results are re-ranked to determine the recommended government affairs knowledge subgraph.

[0040] Specifically, map the cosine similarity result to the interval [0, 1] as the initial semantic score. Starting from the core entity in the user query, calculate the shortest path hop count from each node in the candidate knowledge subgraph to the core entity through the graph search algorithm. For example, if the "declaration material preparation" node is 2 hops away from the core entity, the hop count is recorded as 2. Introduce an exponential decay function to convert the hop count into a structural correlation score. Among them, the exponential decay function is: ; Among them, Structural correlation score; Is the decay coefficient; h Is the hop count.

[0041] Furthermore, obtain the release time of the node information in the candidate knowledge subgraph, compare it with the current retrieval time, calculate the time difference, and convert the time difference into a time factor score through a preset time decay function. Among them, the time factor score function is: ; Among them, Is the time factor score; Is the time decay intensity parameter; Is the time difference.

[0042] Furthermore, collect the user's historical retrieval data, including click records, favorite records, comment records, etc., construct a user behavior data set, and assign weights to different behaviors according to the degree to which the behavior type reflects the user's needs. For each candidate node, count the number of times the node is involved in the user's historical behavior, multiply it by the corresponding behavior weight, and sum to obtain the user behavior score.

[0043] In an implementation manner of the present application, based on the function: ; Obtain the comprehensive scores corresponding to each core entity in the candidate knowledge subgraph. Arrange the core entities in descending order based on the comprehensive scores to determine the recommended government affairs knowledge subgraph. Among them, Is the comprehensive score function; Is the dynamic adjustment coefficient; Is the first preset weight coefficient; Is the second preset weight coefficient; Is the third preset weight coefficient; Is the fourth preset weight coefficient; Is the initial semantic score; Is the structural correlation score; Is the time factor score; Is the user behavior score; Is the non-linear adjustment threshold.

[0044] In an implementation manner of the present application, based on the retrieval results arranged in descending order, the top K nodes and the associated subgraphs corresponding to the top K nodes are respectively selected as the candidate set; Based on the function: ; The candidate set is screened by secondary indicators to obtain the final ranking, and based on the final ranking, the recommended government affairs knowledge subgraph is obtained; wherein, the secondary indicators in the embodiments of the present application can be subgraph integrity and information redundancy.

[0045] Wherein, is the secondary screening function; is the balance coefficient; is the secondary indicator.

[0046] Step 106: Based on the multi-modal data and the recommended government affairs knowledge subgraph, perform visual question answering and visual common sense reasoning processing to obtain multi-modal reasoning results, and based on the user's needs, send the multi-modal reasoning results to the user as recommended content.

[0047] In an implementation manner of the present application, based on the text feature vector, the current retrieval task is divided into a question answering basic task and a common sense reasoning task. In the case of belonging to the question answering basic task, the multi-modal data and the recommended government affairs knowledge subgraph are input into a preset question answering model. Through the preset question answering model, the question information corresponding to the multi-modal data is determined, and based on the question information, a search is performed in the recommended government affairs knowledge subgraph to obtain the recommended information. The recommended information is cross-validated with the structured information in the recommended government affairs knowledge subgraph, and if the validation passes, it is sent to the user.

[0048] Further, in the case of the common sense reasoning task, the multi-modal data and the recommended government affairs knowledge subgraph are input into a preset common sense reasoning model. Through the preset common sense reasoning model, the question information corresponding to the multi-modal data is determined, and based on the question information, relevant government affairs common sense knowledge is called in the knowledge base, and based on the relevant government affairs common sense knowledge and the recommended government affairs knowledge subgraph, reasoning is performed to obtain the recommended information.

[0049] Specifically, a pre-trained classifier is used to input the text feature vector and output the task type probability. When training the classifier, historical government affairs consultation data is used as a sample, and the sample is labeled with the question answering basic task and the common sense reasoning task for training.

[0050] Further, the multi-modal data and the recommended government affairs knowledge sub-graph are input into the government affairs dedicated question answering model. Question information is extracted from the multi-modal data. Based on the entities and relationships in the question information, graph traversal is performed in the knowledge sub-graph to locate associated nodes, and the text content of the nodes is integrated to generate a natural language answer. Among them, the training process of the government affairs dedicated question answering model in the embodiments of the present application is as follows: First, collect text data such as government affairs policies and regulations, service guides, and government affairs documents, as well as multi-modal data such as images and audio in government affairs scenarios to construct a training set; combine the entity and relationship information in the government affairs knowledge graph, and use the government affairs text data to fine-tune the pre-trained model; for multi-modal data, access image recognition and speech recognition modules to extract features, and after fusing with the text information, train the model with multi-round dialogue data to generate accurate answers, and finally form a government affairs dedicated question answering model through optimization and iteration.

[0051] Further, integrate the common sense knowledge in the government affairs field and store it in the knowledge base in the form of triples. According to the keywords in the question information, retrieve relevant common sense knowledge through semantic matching. Obtain specific policy clauses from the recommended government affairs knowledge sub-graph, and perform reasoning by combining the common sense knowledge and the sub-graph policy clauses.

[0052] Further, extract all structured data related to the recommended information from the recommended government affairs knowledge sub-graph, including node attributes and relationship constraints. Check whether the recommended information is consistent with the node text description in the sub-graph, and compare the policy date in the recommended information with the effective date of the sub-graph node to ensure that expired clauses are not cited. For common sense reasoning tasks, trace back the reasoning process to verify the logical coherence between the common sense knowledge and the sub-graph data. If the verification passes, integrate the recommended information and the structured information of the knowledge sub-graph into a visual answer. If the verification fails, trigger a secondary retrieval process, regenerate the recommended information and verify it until the result is reliable and then send it to the user.

[0053] Figure 2 This is a schematic structural diagram of a government affairs information recommendation device that fuses a knowledge graph and multi-modal data. As Figure 2As shown, the government affairs information recommendation device 200 integrating a knowledge graph and multimodality includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201; wherein, the memory 202 stores instructions executable by the at least one processor 201, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to: construct a government affairs knowledge graph based on government affairs entity information, government affairs relationship information, and government affairs attribute information; respond to a user's government affairs information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; perform semantic encoding on the selected knowledge subgraph to generate an initial semantic vector, and extract a text feature vector from the multimodal data uploaded by the user, and fuse the initial semantic vector and the text feature vector to obtain a multimodal semantic vector; perform multi-hop information aggregation processing on the multimodal semantic vector, and perform a fine-grained retrieval based on the aggregated graph node vector and the user query vector; based on a re-ranking mechanism, re-rank the retrieval results to determine a recommended government affairs knowledge subgraph; perform visual question answering and visual common sense reasoning processing based on the multimodal data and the recommended government affairs knowledge subgraph to obtain a multimodal reasoning result, and send the multimodal reasoning result as recommended content to the user based on the user's needs.

[0054] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, they can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the apparatus, device, and non-volatile computer storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0055] The above description is only for the embodiments of this application and is not intended to limit this application. For those skilled in the art, various changes and modifications can be made to the embodiments of this application. These modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of this application.

Claims

1. A government information recommendation method integrating knowledge graph and multi-modal, characterized in that, The method includes: Constructing a government affairs knowledge graph based on government affairs entity information, government affairs relationship information, and government affairs attribute information; Responding to a user's government affairs information retrieval request, generating a user preference vector based on the user's historical retrieval behavior, and performing a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; Performing semantic encoding on the candidate knowledge subgraph to generate an initial semantic vector, and extracting text feature vectors from the multimodal data uploaded by the user, and fusing the initial semantic vector with the text feature vectors to obtain a multimodal semantic vector; Performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained retrieval based on the aggregated graph node vectors and the user query vector; Based on a re-ranking mechanism, re-ranking the retrieval results to determine a recommended government affairs knowledge subgraph; Based on the multimodal data and the recommended government affairs knowledge subgraph, performing visual question answering and visual common sense reasoning processing to obtain a multimodal reasoning result, and based on the user's needs, sending the multimodal reasoning result to the user as recommended content.

2. The method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 1, characterized in that, The generating a user preference vector based on the user's historical retrieval behavior, and performing a coarse-grained retrieval on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph specifically includes: Based on the user's historical retrieval behavior, constructing a user retrieval interaction matrix, and through the user retrieval interaction matrix, determining the association strength between the user and government affairs knowledge entities, and generating an initial preference vector based on the association strength; According to the time series sequence of the user's historical retrievals, assigning a time decay factor to the government affairs knowledge entities, and, based on the user's historical retrieval types, performing weighted aggregation on different types of the government affairs knowledge entities; Calculating and adjusting the initial preference vector through the time decay factor and the weighted aggregation to obtain the user preference vector; Determining seed nodes based on the calculation scores corresponding to the user preference vector; Taking the seed nodes as the center, constructing the candidate knowledge subgraph through the edge relationships of the knowledge graph.

3. A method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 1, characterized in that, The performing semantic encoding on the candidate knowledge subgraph to generate an initial semantic vector, and extracting text feature vectors from the multimodal data uploaded by the user specifically includes: Encoding the entity text and relationship text in the candidate knowledge subgraph respectively through a pre-trained language model to generate the initial semantic vector; Classifying the multimodal data uploaded by the user to determine audio data and image data; Converting the audio data into text content; and, taking screenshots of key frames of the image data, and extracting text content from the key frames; Inputting the text content into the pre-trained language model to obtain the text feature vectors.

4. A method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 1, characterized in that, The performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained retrieval based on the aggregated graph node vectors and the user query vector specifically includes: Taking the multimodal semantic vector as the graph node feature; Determining the feature propagation between the graph node features through a graph convolutional network, and aggregating multi-hop neighbor information through multiple layers of iteration; During the aggregation process, the hierarchical relationship of each node in the candidate knowledge subgraph is distinguished by adding position encoding, and the topological structure features corresponding to the distinguished candidate knowledge subgraph are incorporated into the node vector to obtain the graph node vector; The cosine similarity between the user query vector and the graph node vector is calculated, and fine-grained retrieval is performed in the candidate knowledge subgraph based on the similarity result.

5. A method for recommending government affairs information by integrating a knowledge graph and multi-modalities according to claim 4, characterized in that, Based on the re-ranking mechanism, re-rank the retrieval results to determine the recommended government affairs knowledge subgraph, which specifically includes: Based on the cosine similarity, determine the initial semantic score; Determine the number of hops between the node and the query core entity, and determine the structural association score based on the number of hops; Determine the time difference between the release time of the node information and the current retrieval time, and obtain the time factor score based on the time difference; Based on the user's historical retrieval data, determine the user behavior score; where the user's historical retrieval data at least includes historical click data, historical favorite data, and historical comment data; Based on the preset weight parameters, the initial semantic score, the structural association score, the time factor score, and the user behavior score, re-rank the retrieval results to determine the recommended government affairs knowledge subgraph.

6. The government affairs information recommendation method for knowledge graph and multimodal fusion according to claim 5, wherein Based on the preset weight parameters, the initial semantic score, the structural association score, the time factor score, and the user behavior score, re-rank the retrieval results, which specifically includes: Based on the function: ； Obtain the comprehensive score corresponding to each of the core entities in the candidate knowledge subgraph; Based on the comprehensive score, arrange the core entities in descending order to determine the recommended government affairs knowledge subgraph; Among them, is the comprehensive score function; is the dynamic adjustment coefficient; is the first preset weight coefficient; is the second preset weight coefficient; is the third preset weight coefficient; is the fourth preset weight coefficient; is the initial semantic score; is the structural correlation score; is the time factor score; is the user behavior score; is the non - linear adjustment threshold.

7. A method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 6, characterized in that, Based on the comprehensive score, arrange the core entities in descending order to determine the recommended government affairs knowledge subgraph, which specifically includes: Based on the retrieval results arranged in descending order, select the top K nodes and the associated subgraphs corresponding to the top K nodes as the candidate set; Based on the function: ； Perform secondary index screening on the candidate set to obtain the final ranking, and based on the final ranking, obtain the recommended government affairs knowledge subgraph; Among them, is a secondary screening function; is a balance coefficient; is a secondary index.

8. A method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 1, characterized in that Based on the multi-modal data and the recommended government affairs knowledge subgraph, perform visual question answering and visual common sense reasoning processing to obtain multi-modal reasoning results, which specifically includes: Based on the text feature vector, divide the current retrieval task into a question answering basic task and a common sense reasoning task; In the case of belonging to the question answering basic task, input the multi-modal data and the recommended government affairs knowledge subgraph into a preset question answering model, and through the preset question answering model, determine the question information corresponding to the multi-modal data, and retrieve in the recommended government affairs knowledge subgraph according to the question information to obtain the recommended information; Cross-validate the recommended information with the structured information in the recommended government affairs knowledge subgraph, and send it to the user if the verification passes.

9. The method for recommending government affairs information by fusing a knowledge graph and multi-modalities according to claim 8, wherein, After dividing the current retrieval task into a question answering basic task and a common sense reasoning task based on the text feature vector, the method further includes: In the case of belonging to the common sense reasoning task, input the multi-modal data and the recommended government affairs knowledge subgraph into a pre-set common sense reasoning model; Through the pre-set common sense reasoning model, determine the question information corresponding to the multi-modal data, call relevant government affairs common sense knowledge in the knowledge base based on the question information, and perform reasoning based on the relevant government affairs common sense knowledge and the recommended government affairs knowledge subgraph to obtain recommended information.

10. A government affairs information recommendation device integrating a knowledge graph and multi-modalities, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1-9.

Images