A government information recommendation method and device based on knowledge graph and multimodal fusion

Abstract

The embodiment of the present application discloses a government information recommendation method and device that integrates knowledge graph and multimodality, belonging to the field of information recommendation technology, and solving the problem of low matching between recommendation results and needs when users obtain government information. The method includes generating a user preference vector based on the user's historical search behavior, performing a coarse-grained search on the government knowledge graph to obtain a candidate knowledge subgraph; fusing the candidate knowledge subgraph with the multimodal data uploaded by the user to obtain a multimodal semantic vector; performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained search based on the aggregated graph node vector and the user query vector; reordering the search results based on a reordering mechanism to determine the recommended government knowledge subgraph; performing visual question answering and visual common sense reasoning processing based on the multimodal data and the recommended government knowledge subgraph to obtain a multimodal reasoning result, and sending the multimodal reasoning result to the user as recommended content based on the user's needs.

Description

Technical Field

[0001] The present application relates to the field of information recommendation technology, and in particular to a method and device for recommending government information by integrating knowledge graph and multimodality. Background Art

[0002] With the rapid development of digital government services, massive government information such as policies and regulations, service guidelines, case data, etc. is growing exponentially. The public faces serious difficulties in information screening. The recommendation system, as the core technology of information filtering, has become a key means to optimize the government service experience and improve service efficiency.

[0003] Large language models are currently widely used to resolve government inquiries due to their semantic understanding and generation capabilities. However, in practice, large language models frequently suffer from hallucinations. Without real data support, they generate inconsistent, logically incorrect, or non-existent government information, hindering the user's service process.

[0004] Secondly, government systems usually include various types of data such as pictures of service materials and policy promotion videos, which are very different from text information modalities. It is difficult to accurately understand multiple types of government data through large-scale language models, and thus it is difficult to accurately grasp the user's multimodal consultation intentions, resulting in a low degree of match between recommendation results and needs. Summary of the Invention

[0005] The embodiments of the present application provide a government information recommendation method and device that integrates knowledge graphs and multimodalities, which are used to solve the following technical problems: In the existing technology, when users obtain government information, they frequently encounter hallucination problems in large language models and find it difficult to accurately understand various types of government data, resulting in a low degree of match between recommendation results and needs.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] The embodiment of the present application provides a government information recommendation method that integrates knowledge graph and multimodality. The method includes: constructing a government knowledge graph based on government entity information, government relationship information, and government attribute information; responding to a user's government information retrieval request, generating a user preference vector based on the user's historical retrieval behavior, and performing a coarse-grained retrieval of the government knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; semantically encoding the candidate knowledge subgraph to generate an initial semantic vector, extracting text feature vectors from the multimodal data uploaded by the user, fusing the initial semantic vector with the text feature vector to obtain a multimodal semantic vector; performing multi-hop information aggregation processing on the multimodal semantic vector, and performing fine-grained retrieval based on the aggregated graph node vector and the user query vector; reordering the retrieval results based on a reordering mechanism to determine a recommended government knowledge subgraph; performing visual question answering and visual common sense reasoning processing on the multimodal data and the recommended government knowledge subgraph to obtain a multimodal reasoning result, and sending the multimodal reasoning result to the user as recommended content based on user needs.

[0008] The embodiment of the present application uses coarse-grained retrieval to quickly locate candidate subgraphs from the knowledge graph, narrow the data range, reduce the amount of subsequent calculations, and improve the recommendation response speed. The candidate knowledge subgraph semantics are encoded into vectors and fused with text feature vectors of multimodal data such as images and videos uploaded by users to achieve the associated understanding of graphic and text information, improve the accuracy of understanding complex government affairs issues, and avoid the one-sidedness of single-modal information. Secondly, the embodiment of the present application combines the aggregated graph node vectors with the user query vectors for fine-grained retrieval to ensure that the recommended content is in deep alignment with user needs and improve the accuracy of policy interpretation and process guidance. It also re-sorts based on factors such as semantic relevance and timeliness to ensure that the current effective policies and the latest service processes are given priority in the recommendation results, thereby improving the intelligence and interactive efficiency of government services.

[0009] In one implementation of the present application, a user preference vector is generated based on the user's historical retrieval behavior, and a coarse-grained search is performed on the government knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph, specifically including: constructing a user retrieval interaction matrix based on the user's historical retrieval behavior, and determining the association strength between the user and the government knowledge entity through the user retrieval interaction matrix, and generating an initial preference vector based on the association strength; assigning a time decay factor to the government knowledge entity according to the time series of the user's historical retrieval, and performing weighted aggregation on different types of government knowledge entities based on the user's historical retrieval type; calculating and adjusting the initial preference vector through the time decay factor and weighted aggregation to obtain a user preference vector; determining a seed node based on the calculated score corresponding to the user preference vector; and constructing a candidate knowledge subgraph with the seed node as the center through the edge relationship of the knowledge graph.

[0010] In one implementation of the present application, the candidate knowledge subgraph is semantically encoded to generate an initial semantic vector, and the text feature vector is extracted from the multimodal data uploaded by the user, specifically including: encoding the entity text and relationship text in the candidate knowledge subgraph separately through a pre-trained language model to generate an initial semantic vector; classifying the multimodal data uploaded by the user to determine audio data and image data; converting the audio data into text content; and taking key frame screenshots of the image data and extracting text content from the key frames; inputting the text content into the pre-trained language model to obtain a text feature vector.

[0011] In one implementation of the present application, multi-hop information aggregation processing is performed on the multimodal semantic vector, and fine-grained retrieval is performed based on the graph node vector and the user query vector after aggregation, specifically including: using the multimodal semantic vector as the graph node feature; determining the feature propagation between the features of each graph node through a graph convolutional network, and aggregating multi-hop neighbor information through multi-layer iteration; in the aggregation process, the hierarchical relationship of each node in the candidate knowledge subgraph is distinguished by adding position coding, and the topological structure features corresponding to the distinguished candidate knowledge subgraph are integrated into the node vector to obtain the graph node vector; the cosine similarity between the user query vector and the graph node vector is calculated, and fine-grained retrieval is performed in the candidate knowledge subgraph based on the similarity result.

[0012] In one implementation of the present application, the search results are reordered based on a reordering mechanism to determine a recommended government knowledge subgraph, specifically including: determining an initial semantic score based on cosine similarity; determining the number of hops between the node and the query core entity, and determining a structural relevance score based on the number of hops; determining the time difference between the release time of the node information and the current retrieval time, and obtaining a time factor score based on the time difference; determining a user behavior score based on user historical retrieval data; wherein the user historical retrieval data includes at least historical click data, historical collection data, and historical comment data; reordering the search results based on preset weight parameters, initial semantic score, structural relevance score, time factor score, and user behavior score to determine a recommended government knowledge subgraph.

[0013] In one implementation of the present application, the search results are reordered based on preset weight parameters, initial semantic scores, structural relevance scores, time factor scores, and user behavior scores, specifically including: based on the function:

[0014] ;

[0015] Obtain the comprehensive scores corresponding to each core entity in the candidate knowledge subgraph; sort the core entities in descending order based on the comprehensive scores to determine the recommended government knowledge subgraph; is the comprehensive score function; is the dynamic adjustment coefficient; is the first preset weight coefficient; is the second preset weight coefficient; is the third preset weight coefficient; is the fourth preset weight coefficient; is the initial semantic score; Score for structural relevance; Score the time factor; Score user behavior; is the nonlinear adjustment threshold.

[0016] In one implementation of the present application, the core entities are sorted in descending order based on the comprehensive scores to determine the recommended government knowledge subgraph, specifically including: based on the search results sorted in descending order, selecting the first K nodes and the associated subgraphs corresponding to the first K nodes as the candidate set; based on the function:

[0017] ;

[0018] Perform secondary index screening on the candidate set to obtain the final ranking, and obtain the recommended government knowledge subgraph based on the final ranking; is the secondary selection function; is the balance coefficient; is a secondary indicator.

[0019] In one implementation of the present application, visual question answering and visual common sense reasoning are performed based on multimodal data and a recommended government affairs knowledge subgraph to obtain a multimodal reasoning result, specifically including: dividing the current retrieval task into a question-answering basic task and a common sense reasoning task based on a text feature vector; in the case of a question-answering basic task, inputting the multimodal data and the recommended government affairs knowledge subgraph into a preset question-answering model, determining the question information corresponding to the multimodal data through the preset question-answering model, and searching in the recommended government affairs knowledge subgraph based on the question information to obtain recommended information; cross-validating the recommended information with the structured information in the recommended government affairs knowledge subgraph, and sending it to the user if the verification passes.

[0020] In one implementation of the present application, after dividing the current retrieval task into a question-answering basic task and a common sense reasoning task based on a text feature vector, the method also includes: in the case of a common sense reasoning task, inputting the multimodal data and the recommended government knowledge subgraph into a preset common sense reasoning model, determining the question information corresponding to the multimodal data through the preset common sense reasoning model, calling relevant government common sense knowledge in the knowledge base based on the question information, and performing reasoning based on the relevant government common sense knowledge and the recommended government knowledge subgraph to obtain recommended information.

[0021] The embodiment of the present application provides a government information recommendation device that integrates knowledge graph and multimodality, including: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: construct a government knowledge graph based on government entity information, government relationship information, and government attribute information; respond to a user's government information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained search on the government knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; and perform language analysis on the selected knowledge subgraph. It performs semantic encoding to generate an initial semantic vector, extracts text feature vectors from the multimodal data uploaded by users, fuses the initial semantic vector with the text feature vector to obtain a multimodal semantic vector; performs multi-hop information aggregation on the multimodal semantic vector, and performs fine-grained retrieval based on the aggregated graph node vector and the user query vector; re-ranks the retrieval results based on the re-ranking mechanism to determine the recommended government knowledge sub-graph; performs visual question answering and visual common sense reasoning based on the multimodal data and the recommended government knowledge sub-graph to obtain multimodal reasoning results, and sends the multimodal reasoning results to users as recommended content based on user needs.

[0022] At least one of the above technical solutions adopted in the embodiment of the present application can achieve the following beneficial effects: The embodiment of the present application quickly locates candidate subgraphs from the knowledge graph through coarse-grained retrieval, narrows the data range, reduces the subsequent calculation amount, and improves the recommendation response speed. The candidate knowledge subgraph semantics are encoded into a vector, and fused with the text feature vectors of multimodal data such as images and videos uploaded by users to achieve the associated understanding of graphic and text information, improve the accuracy of understanding complex government affairs issues, and avoid the one-sidedness of single-modal information. Secondly, the embodiment of the present application combines the aggregated graph node vector with the user query vector for fine-grained retrieval to ensure that the recommended content is in deep alignment with user needs and improve the accuracy of policy interpretation and process guidance. It also re-sorts based on factors such as semantic relevance and timeliness to ensure that the current effective policies and the latest service processes are given priority in the recommendation results, thereby improving the intelligence and interactive efficiency of government services. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0024] Figure 1A flowchart of a government information recommendation method that integrates knowledge graph and multimodality provided in an embodiment of the present application;

[0025] Figure 2 A structural diagram of a government information recommendation device that integrates knowledge graph and multimodality provided in an embodiment of the present application.

[0026] Reference numerals:

[0027] 200: Government information recommendation device based on knowledge graph and multimodal fusion, 201: Processor, 202: Memory. DETAILED DESCRIPTION

[0028] The embodiments of the present application provide a method and device for recommending government information by integrating knowledge graph and multimodality.

[0029] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] The technical solutions proposed in the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A flowchart of a government information recommendation method based on knowledge graph and multimodal fusion provided in an embodiment of the present application is shown as follows: Figure 1 As shown in Figure 1, the government information recommendation method based on knowledge graph and multimodal fusion includes the following steps:

[0032] Step 101: Construct a government knowledge graph based on government entity information, government relationship information, and government attribute information.

[0033] In one implementation of this application, information on government entities, government relationships, and government attributes is collected through channels such as government transparency platforms, departmental database interfaces, and public government documents such as policies and regulations, service guidelines, and government service system logs. A government knowledge graph is constructed by treating government entities as graph nodes, relationships as edges, and attribute information as additional attributes of nodes or edges. Government entities can include government departments and organizations, government service items, and information on government personnel.

[0034] Step 102: In response to the user's government information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained search on the government knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph.

[0035] In one implementation of the present application, in response to a user's request to retrieve government information, a user retrieval interaction matrix is ​​constructed based on the user's historical retrieval behavior, and the strength of the association between the user and the government knowledge entity is determined through the user retrieval interaction matrix, and an initial preference vector is generated based on the strength of the association. According to the time series of the user's historical retrieval, a time decay factor is assigned to the government knowledge entity, and, based on the user's historical retrieval type, different types of government knowledge entities are weightedly aggregated. The initial preference vector is calculated and adjusted through the time decay factor and weighted aggregation to obtain a user preference vector. Based on the calculated score corresponding to the user preference vector, a seed node is determined, and with the seed node as the center, a candidate knowledge subgraph is constructed through the edge relationship of the knowledge graph.

[0036] Specifically, historical user search behavior data on the government service platform is collected. The search results in this user behavior data are mapped to entities in the government knowledge graph. A two-dimensional sparse matrix is ​​constructed with user IDs as rows and government knowledge entities as columns. If a user searches for or clicks on an entity, the interaction record is marked at the corresponding matrix position. Basic weights are assigned based on the interaction data recorded in the interaction matrix, and an initial association score between the user and each entity is calculated through weighted summation. This association score is normalized to obtain the association strength. The association strength is used as a vector element and arranged in the order of the government knowledge entities in the matrix column to form the initial preference vector.

[0037] Furthermore, a time decay factor is assigned to each government knowledge entity based on the temporal sequence of a user's historical searches. Furthermore, government knowledge entities are divided into different types, such as policies and regulations, procedures, and case references, and weights are assigned to each type based on the distribution of the types of searches performed by the user. Each element in the initial preference vector is multiplied by the corresponding time decay factor and type weight to obtain an adjusted score.

[0038] Furthermore, the adjusted scores are recombined into a new vector, which is the final user preference vector. This vector comprehensively considers the timeliness and preference of user behavior, and more accurately reflects the user's current needs. The score of each element in the user preference vector is calculated, and the top-K entities with the highest scores are selected as seed nodes. With the seed node as the center, the 1-2 hop neighbor nodes are expanded outward through the edge relationship of the knowledge graph. For example, starting from the "Enterprise Social Security Payment Process" node, it expands to directly related nodes such as "Social Security Payment Materials" and "Social Security Handling Agency", and further expands to indirect nodes such as "Material Template Download Link" and "Institution Contact Information", ultimately forming a candidate knowledge subgraph set containing core information.

[0039] Step 103: semantically encode the candidate knowledge subgraph to generate an initial semantic vector, extract text feature vectors from the multimodal data uploaded by the user, and fuse the initial semantic vector with the text feature vector to obtain a multimodal semantic vector.

[0040] In one implementation of this application, a pre-trained language model is used to encode the entity text and relationship text in the candidate knowledge subgraph to generate an initial semantic vector. Multimodal data uploaded by the user is classified to identify audio data and image data. The audio data is converted into text content, and keyframe screenshots of the image data are taken, and text content is extracted from the keyframes. The text content is input into the pre-trained language model to obtain a text feature vector.

[0041] Specifically, a language model was constructed using RoBERTa (Robustly Optimized BERT Pretraining Approach) and trained using government text data to obtain a language model that meets the requirements. This language model was used to encode the entity text and relationship text in the candidate knowledge subgraph separately to generate an initial semantic vector. Multimodal data uploaded by users was divided into audio and image data. If the data was audio, the audio content was converted into text. If the data was image data, the images containing text were defined as keyframes, and the text content in the keyframes was extracted. The text content extracted from the multimodal data was input into the trained language model to obtain a text feature vector. The initial semantic vector was concatenated with the text feature vector to obtain a multimodal semantic vector.

[0042] Step 104: Perform multi-hop information aggregation processing on the multimodal semantic vector, and perform fine-grained retrieval based on the aggregated graph node vector and the user query vector.

[0043] In one implementation of the present application, a multimodal semantic vector is used as a graph node feature. The feature propagation between the features of each graph node is determined through a graph convolutional network, and multi-hop neighbor information is aggregated through multi-layer iteration. During the aggregation process, the hierarchical relationship of each node in the candidate knowledge subgraph is distinguished by adding position encoding, and the topological structure features corresponding to the distinguished candidate knowledge subgraph are integrated into the node vector to obtain a graph node vector. The cosine similarity between the user query vector and the graph node vector is calculated, and fine-grained retrieval is performed in the candidate knowledge subgraph based on the similarity result.

[0044] Specifically, the fused multimodal semantic vector is assigned to each node in the candidate knowledge subgraph. For example, if the subgraph contains the node "Corporate Income Tax Declaration Process," the multimodal semantic vector associated with that node is used as its initial feature vector. Based on the topological structure of the candidate knowledge subgraph, the graph convolutional network (GCN) model is initialized, and hyperparameters such as the number of network layers and hidden layer dimensions are set. In each GCN layer, for each graph node, feature propagation is calculated through convolution operations based on the features of its neighboring nodes and edge connectivity. Multiple GCN layers are stacked and iterated over multiple rounds. With each iteration, a node not only aggregates information from its direct neighbors but also indirectly acquires features from multi-hop neighbors, gradually enriching its semantic representation. For example, after three GCN layers, the feature vector of each node incorporates information from neighboring nodes within a three-hop radius.

[0045] Furthermore, a position encoding mechanism is introduced to distinguish the hierarchical relationships of nodes within a subgraph. Each node is assigned a different position encoding vector based on the shortest path distance between the node and the core entity of the subgraph query. The global topological structure features of the candidate knowledge subgraph are extracted. This topological structure feature vector is concatenated with the node feature vector, so that the node vector not only contains its own semantics and neighborhood information, but also implicitly reflects the overall structural features of the subgraph.

[0046] Furthermore, the cosine similarity between the user query vector generated by the user query and the graph node vector after the above processing is calculated one by one, and fine-grained retrieval is performed in the candidate knowledge subgraph based on the similarity results.

[0047] Step 105: Reorder the search results based on the reordering mechanism to determine the recommended government affairs knowledge subgraph.

[0048] In one implementation of the present application, an initial semantic score is determined based on cosine similarity, the number of hops between the node and the query core entity is determined, and a structural relevance score is determined based on the number of hops. The time difference between the release time of the node information and the current retrieval time is determined, and a time factor score is obtained based on the time difference. A user behavior score is determined based on the number of historical user retrievals; wherein the user's historical retrieval data includes at least historical click data, historical collection data, and historical comment data. Based on preset weight parameters, the initial semantic score, the structural relevance score, the time factor score, and the user behavior score, the retrieval results are reordered to determine a recommended government knowledge subgraph.

[0049] Specifically, the cosine similarity result is mapped to the interval [0, 1] as the initial semantic score. Starting from the core entity in the user query, a graph search algorithm is used to calculate the number of hops in the shortest path from each node in the candidate knowledge subgraph to the core entity. For example, if the node "Application Materials Preparation" is 2 hops away from the core entity, the hop count is recorded as 2. An exponential decay function is introduced to convert the hop count into a structural relevance score. The exponential decay function is:

[0050] ;

[0051] in, Structural relevance score; is the attenuation coefficient; h is the number of hops.

[0052] Furthermore, the release time of the node information in the candidate knowledge subgraph is obtained, compared with the current search time, and the time difference is calculated. The time difference is converted into a time factor score through a preset time decay function, where the time factor score function is:

[0053] ;

[0054] in, Score the time factor; is the time decay intensity parameter; is the time difference.

[0055] Furthermore, we collect historical user search data, including click records, favorite records, and comment records, to construct a user behavior dataset. We then assign weights to different behaviors based on how well they reflect user needs. For each candidate node, we count the number of times the user's historical behavior involves that node, multiply the number by the corresponding behavior weight, and sum the results to obtain a user behavior score.

[0056] In one implementation of the present application, based on the function:

[0057] ;

[0058] Obtain the comprehensive scores corresponding to each core entity in the candidate knowledge subgraph. Arrange the core entities in descending order based on the comprehensive scores to determine the recommended government knowledge subgraph. is the comprehensive score function; is the dynamic adjustment coefficient; is the first preset weight coefficient; is the second preset weight coefficient; is the third preset weight coefficient; is the fourth preset weight coefficient; is the initial semantic score; Score for structural relevance; Score the time factor; Score user behavior; is the nonlinear adjustment threshold.

[0059] In one implementation of the present application, based on the search results arranged in descending order, the first K nodes and the associated subgraphs corresponding to the first K nodes are selected as the candidate set;

[0060] Function-based:

[0061] ;

[0062] The candidate set is screened by secondary indicators to obtain a final ranking, and based on the final ranking, a recommended government affairs knowledge subgraph is obtained; wherein, the secondary indicators in the embodiment of the present application can be subgraph integrity and information redundancy.

[0063] in, is the secondary selection function; is the balance coefficient; is a secondary indicator.

[0064] Step 106: Based on the multimodal data and the recommended government knowledge subgraph, visual question answering and visual common sense reasoning are performed to obtain multimodal reasoning results, and the multimodal reasoning results are sent to the user as recommended content based on user needs.

[0065] In one implementation of this application, the current retrieval task is divided into a basic question-answering task and a commonsense reasoning task based on text feature vectors. In the case of a basic question-answering task, the multimodal data and the recommended government knowledge subgraph are input into a pre-set question-answering model. The pre-set question-answering model determines the question information corresponding to the multimodal data. Based on the question information, the recommended government knowledge subgraph is searched to obtain recommended information. The recommended information is cross-validated with the structured information in the recommended government knowledge subgraph and, if verified, is sent to the user.

[0066] Furthermore, in the case of common sense reasoning tasks, the multimodal data and the recommended government knowledge subgraph are input into the preset common sense reasoning model. The preset common sense reasoning model is used to determine the question information corresponding to the multimodal data. Based on the question information, the relevant government common sense knowledge is called in the knowledge base, and reasoning is performed based on the relevant government common sense knowledge and the recommended government knowledge subgraph to obtain recommended information.

[0067] Specifically, a pre-trained classifier is used to input text feature vectors and output task type probabilities. The classifier is trained using historical government consultation data as samples, annotating the samples with basic question-answering tasks and common sense reasoning tasks.

[0068] Furthermore, the multimodal data and the recommended government knowledge subgraph are input into a government-specific question-answering model, question information is extracted from the multimodal data, and based on the entities and relationships in the question information, a graph traversal is performed in the knowledge subgraph to locate related nodes, and the node text content is integrated to generate natural language answers. The training process of the government-specific question-answering model in the embodiment of the present application is as follows: first, text data such as government policies and regulations, service guidelines, and government documents, as well as multimodal data such as images and audio in government scenarios are collected to construct a training set; the pre-trained model is fine-tuned using government text data in combination with entity and relationship information in the government knowledge graph; for multimodal data, image recognition and speech recognition modules are connected to extract features, which are then integrated with text information, and the model is trained with multiple rounds of dialogue data to generate accurate answers, and finally, a government-specific question-answering model is formed through optimization and iteration.

[0069] Furthermore, we integrate common sense knowledge in the government domain and store it in the knowledge base as triples. We then retrieve relevant common sense knowledge based on the keywords in the question information through semantic matching. We then obtain specific policy terms from the recommended government knowledge subgraph and perform reasoning based on the common sense knowledge and the subgraph policy terms.

[0070] Furthermore, all structured data related to the recommended information is extracted from the recommended government knowledge subgraph, including node attributes and relationship constraints. The recommended information is checked for consistency with the textual descriptions of the nodes in the subgraph. The policy dates in the recommended information are compared with the effective dates of the subgraph nodes to ensure that no expired clauses are referenced. For commonsense reasoning tasks, the reasoning process is backtracked to verify the logical coherence between the commonsense knowledge and the subgraph data. If verification passes, the recommended information and the structured information of the knowledge subgraph are integrated into a visual answer. If verification fails, a secondary search process is triggered to regenerate and verify the recommended information until the results are reliable and sent to the user.

[0071] Figure 2 This is a schematic diagram of the structure of a government information recommendation device that integrates knowledge graph and multimodal fusion provided in the embodiment of this application. Figure 2As shown, a government information recommendation device 200 integrating knowledge graph and multimodality includes: at least one processor 201; and a memory 202 in communication with the at least one processor 201; wherein the memory 202 stores instructions executable by the at least one processor 201, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to: construct a government knowledge graph based on government entity information, government relationship information, and government attribute information; respond to a user's government information retrieval request, generate a user preference vector based on the user's historical retrieval behavior, and perform a coarse-grained search on the government knowledge graph based on the user's preference vector to obtain a candidate knowledge subgraph; The selected knowledge subgraph is semantically encoded to generate an initial semantic vector, and the text feature vector is extracted from the multimodal data uploaded by the user. The initial semantic vector is fused with the text feature vector to obtain a multimodal semantic vector; multi-hop information aggregation is performed on the multimodal semantic vector, and fine-grained retrieval is performed based on the aggregated graph node vector and the user query vector; based on the re-ranking mechanism, the retrieval results are re-ranked to determine the recommended government knowledge subgraph; based on the multimodal data and the recommended government knowledge subgraph, visual question answering and visual common sense reasoning are performed to obtain multimodal reasoning results, and the multimodal reasoning results are sent to the user as recommended content based on user needs.

[0072] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0073] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. It will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present application. However, such modifications or substitutions do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A government information recommendation method based on knowledge graph and multimodal fusion, characterized in that: The method comprises: Construct a government knowledge graph based on government entity information, government relationship information, and government attribute information; In response to a user's government information retrieval request, a user preference vector is generated based on the user's historical retrieval behavior, and a coarse-grained search is performed on the government knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph; Performing semantic encoding on the candidate knowledge subgraph to generate an initial semantic vector, extracting a text feature vector from the multimodal data uploaded by the user, and fusing the initial semantic vector with the text feature vector to obtain a multimodal semantic vector; Performing multi-hop information aggregation processing on the multimodal semantic vector, and performing fine-grained retrieval based on the aggregated graph node vector and the user query vector; Based on the re-ranking mechanism, the search results are re-ranked to determine the recommended government knowledge sub-graph; Performing visual question answering and visual common sense reasoning based on the multimodal data and the recommended government affairs knowledge subgraph to obtain a multimodal reasoning result, and sending the multimodal reasoning result as recommended content to the user based on user needs; The method of generating a user preference vector based on the user's historical search behavior and performing a coarse-grained search on the government affairs knowledge graph based on the user preference vector to obtain a candidate knowledge subgraph specifically includes: Based on the user's historical search behavior, a user search interaction matrix is ​​constructed, and the association strength between the user and the government knowledge entity is determined through the user search interaction matrix, and an initial preference vector is generated based on the association strength; specifically, the search content in the user behavior data is mapped with the entities in the government knowledge graph, and a two-dimensional sparse matrix is ​​constructed with user ID as the row and government knowledge entity as the column; if the user has a search or click behavior on a certain entity, the interaction record is marked at the corresponding matrix position; a basic weight is assigned according to the interaction data recorded in the interaction matrix, and the initial association score between the user and each entity is calculated by weighted summation; the association score is normalized to obtain the association strength, and the association strength is used as a vector element and arranged in the column order of the government knowledge entity in the matrix to form an initial preference vector; According to the time series of user historical retrieval, a time decay factor is assigned to the government knowledge entity. and, based on the user's historical search type, performing weighted aggregation on different types of government knowledge entities; Calculating and adjusting the initial preference vector by using the time decay factor and the weighted aggregation to obtain the user preference vector; Determining a seed node based on a calculated score corresponding to the user preference vector; Taking the seed node as the center, construct the candidate knowledge subgraph through the edge relationship of the knowledge graph; The multi-hop information aggregation processing is performed on the multimodal semantic vector, and fine-grained retrieval is performed based on the aggregated graph node vector and the user query vector, specifically including: Using the multimodal semantic vector as a graph node feature; Determine the feature propagation between the features of each graph node through a graph convolutional network, and aggregate multi-hop neighbor information through multi-layer iteration; During the aggregation process, the hierarchical relationship of each node in the candidate knowledge subgraph is distinguished by adding position coding, and the topological structure features corresponding to the distinguished candidate knowledge subgraph are integrated into the node vector to obtain the graph node vector; Calculate the cosine similarity between the user query vector and the graph node vector, and perform fine-grained retrieval in the candidate knowledge subgraph based on the similarity result; The reordering mechanism is based on which the search results are reordered to determine the recommended government affairs knowledge subgraph, specifically including: Determining an initial semantic score based on the cosine similarity; Determining the number of hops between the node and the query core entity, and determining a structural relevance score based on the number of hops; Determine the time difference between the release time of the node information and the current search time, and obtain a time factor score based on the time difference; Determine a user behavior score based on user history retrieval data; wherein the user history retrieval data includes at least historical click data, historical favorite data, and historical comment data; Reordering the search results based on preset weight parameters, the initial semantic score, the structural relevance score, the time factor score, and the user behavior score to determine the recommended government affairs knowledge subgraph; The performing of visual question answering and visual common sense reasoning based on the multimodal data and the recommended government affairs knowledge subgraph to obtain a multimodal reasoning result specifically includes: Based on the text feature vector, the current retrieval task is divided into a question-answering basic task and a common sense reasoning task; specifically, using a pre-trained classifier, inputting the text feature vector and outputting a task type probability; In the case of the question-answering basic task, the multimodal data and the recommended government affairs knowledge subgraph are input into a preset question-answering model, the preset question-answering model is used to determine the question information corresponding to the multimodal data, and based on the question information, a search is performed in the recommended government affairs knowledge subgraph to obtain recommended information; Cross-verifying the recommendation information with the structured information in the recommended government affairs knowledge subgraph, and sending the recommended information to the user if the verification passes; After dividing the current retrieval task into a question-answering basic task and a common sense reasoning task based on the text feature vector, the method further includes: In the case of the common sense reasoning task, inputting the multimodal data and the recommended government affairs knowledge subgraph into a preset common sense reasoning model; The preset common sense reasoning model is used to determine the question information corresponding to the multimodal data, and the relevant government common sense knowledge is called in the knowledge base based on the question information, so as to perform reasoning based on the relevant government common sense knowledge and the recommended government knowledge subgraph to obtain recommended information.

2. The government information recommendation method based on knowledge graph and multimodal fusion according to claim 1 is characterized in that: The semantic encoding of the candidate knowledge subgraph, the generation of the initial semantic vector, and the extraction of the text feature vector of the multimodal data uploaded by the user specifically include: Encoding the entity text and the relationship text in the candidate knowledge subgraph respectively through a pre-trained language model to generate the initial semantic vector; Classifying the multimodal data uploaded by the user to determine audio data and image data; Converting the audio data into text content; and taking key frame screenshots of the image data and extracting text content from the key frames; The text content is input into the pre-trained language model to obtain the text feature vector.

3. The government information recommendation method based on knowledge graph and multimodal fusion according to claim 1 is characterized in that: The reordering of the search results based on the preset weight parameter, the initial semantic score, the structural relevance score, the time factor score, and the user behavior score specifically includes: Function-based: ； Obtaining comprehensive scores corresponding to each of the core entities in the candidate knowledge subgraph; Arrange the core entities in descending order based on the comprehensive scores to determine the recommended government affairs knowledge subgraph; in, is the comprehensive score function; is the dynamic adjustment coefficient; is the first preset weight coefficient; is the second preset weight coefficient; is the third preset weight coefficient; is the fourth preset weight coefficient; is the initial semantic score; Score for structural relevance; Score the time factor; Score user behavior; is the nonlinear adjustment threshold.

4. The government information recommendation method based on knowledge graph and multimodal fusion according to claim 3 is characterized in that: Arranging the core entities in descending order based on the comprehensive scores to determine the recommended government affairs knowledge subgraph specifically includes: Based on the search results in descending order, the first K nodes and the associated subgraphs corresponding to the first K nodes are selected as the candidate set; Function-based: ； Performing secondary index screening on the candidate set to obtain a final ranking, and obtaining the recommended government affairs knowledge subgraph based on the final ranking; in, is the secondary selection function; is the balance coefficient; is a secondary indicator.

5. A government information recommendation device that integrates knowledge graph and multimodality, characterized in that: The device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 4.

Images