Government affair question and answer method, device and equipment based on image-text understanding and storage medium

Abstract

The invention discloses a government affair question-answering method, device and equipment based on image-text understanding and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining initial image data of a government affair question, preprocessing the initial image data to obtain to-be-processed image data, and storing the to-be-processed image data in a database; determining image information by using an optical character recognition technology, an image classification technology and a target detection technology; carrying out correlation analysis on the image information, carrying out semantic relation and logic relation matching to generate structured information, constructing a knowledge graph, matching the structured information with the knowledge graph to obtain a matching result, generating an initial government affair reply by utilizing a context sensing model and dialogue context information, and sending the initial government affair reply to a server; constructing a user portrait based on the historical behavior data of the user, and determining policy content in the knowledge graph based on the structured information to obtain government affair recommendation information; and integrating the initial government affair reply and the government affair recommendation information by using an image-text generation model to obtain a target government affair reply comprising an interactive diagram so as to improve the efficiency of government affair question answering.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a government affairs question-and-answer method, apparatus, device, and storage medium based on image and text understanding. Background Technology

[0002] Currently, with the in-depth advancement of digital government construction, government services are transforming from "offline windows" to "online intelligence." At the same time, existing technologies have shortcomings such as single-modal limitations (only supporting text retrieval and unable to process structured data in images), lagging knowledge updates (rule-based knowledge graphs have high maintenance costs and synchronization delays), and insufficient interactive experience (lacking dynamic visual feedback and resulting in low processing efficiency).

[0003] As can be seen from the above, how to improve the efficiency of government Q&A in the process of text-image understanding is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for government affairs question answering based on image and text understanding, which can improve the efficiency of government affairs question answering in the process of image and text understanding-based government affairs question answering. The specific solution is as follows:

[0005] Firstly, this application provides a government affairs question-answering method based on image and text understanding, including:

[0006] Initial image data corresponding to government affairs issues is acquired, and the initial image data is preprocessed to obtain image data to be processed. Then, optical character recognition technology, image classification technology, and target detection technology are used to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text.

[0007] The image information is subjected to association analysis, and the association analysis results are matched with semantic relationships and logical connections to generate structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores triple relationships between policy nodes, event nodes, and material nodes.

[0008] An initial government response corresponding to the government issue is generated using a context-aware model and based on the matching results and dialogue context information. Then, a user profile is constructed based on the user's historical behavior data. Based on the user profile and the structured information, the corresponding policy content is determined in the knowledge graph to obtain personalized government recommendation information.

[0009] By integrating the initial government response and the government recommendation information using a text-image generation model, a target government response including interactive illustrations is obtained, which is then used to answer the government question.

[0010] Optionally, the step of acquiring initial image data corresponding to the government affairs issue, preprocessing the initial image data to obtain image data to be processed, and then using optical character recognition technology, image classification technology, and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, signature key elements, data cells, and cell text, includes:

[0011] Acquire initial image data corresponding to the government affairs issue, and perform image enhancement, denoising and normalization operations on the initial image data to obtain image data to be processed;

[0012] The image data to be processed is extracted by using an image recognition engine and optical character recognition technology to obtain the text information corresponding to the image data to be processed.

[0013] The seal is extracted from the image data to be processed using image classification technology to obtain the corresponding seal. Then, the key signature elements are extracted from the image data to be processed using the image classification technology to obtain the key signature elements.

[0014] The target detection technology is used to extract each data cell from the image data to be processed, and the text in each data cell is extracted to obtain the cell text.

[0015] Optionally, the step of performing association analysis on the image information, matching semantic relationships and logical connections based on the association analysis results to generate structured information, and then constructing a knowledge graph includes:

[0016] The association analysis of each piece of information in the image information is performed to obtain the association analysis results. Then, the semantic relationship matching of each association analysis result is performed to obtain the semantic relationship matching results. Finally, the logical connection of each semantic relationship matching result is generated to obtain structured information.

[0017] The policy attributes corresponding to the structured information, including policy identifier, policy name, release time and policy content, are determined to define policy nodes based on the policy attributes. Then, the item attributes corresponding to the structured information, including item name, handling department and handling process, are determined to define item nodes based on the item attributes.

[0018] The material attributes, including material name and material requirements, corresponding to the structured information are determined. Material nodes are defined based on the material attributes, and triple relationships between the policy node, the matter node, and the material node are determined. Then, the triple relationships are stored using a graph database. The triple relationships support knowledge retrieval and reasoning using a graph query language.

[0019] Optionally, matching the structured information with the knowledge graph to obtain a matching result includes:

[0020] The government policy system interface is polled according to a preset time period to monitor the change log of the government policy system in real time through the government policy system interface, and the policy identifier, change type and content fields in the change log are parsed to obtain the parsing results;

[0021] If the change type in the parsing result is characterized as a data update, the content attributes corresponding to the policy nodes in the knowledge graph are updated, and the material list in the change log is updated using preset material update rules to obtain an update result. Based on the update result, the structured information is matched with the knowledge graph to obtain a matching result.

[0022] Optionally, the step of generating an initial government response corresponding to the government affairs question using a context-aware model and based on the matching result and dialogue context information includes:

[0023] Record the user's historical question information to determine the dialogue context information corresponding to the matching result based on the historical question information, and use a context-aware model to perform user intent recognition on the dialogue context information to obtain the user intent recognition result; wherein, the architecture corresponding to the context-aware model is the Transformer architecture;

[0024] When the user intent recognition result indicates that the user's question is ambiguous, natural language processing technology is used to identify ambiguous keywords in the user's question. Then, a targeted question corresponding to the ambiguous keyword is generated using preset rules and preset templates. The targeted question is used to guide the user to supplement clear information and obtain new dialogue context information. Then, an initial government response is generated based on the dialogue context information.

[0025] Optionally, the step of constructing user profiles based on users' historical behavior data, and determining corresponding policy content in the knowledge graph based on the user profiles and the structured information to obtain personalized government affairs recommendation information, includes:

[0026] Collect user information corresponding to each user, including company type, transaction history, browsing history, and search keywords, and use data analysis and mining techniques to generate corresponding multi-dimensional user profiles based on the user information; the user profiles include information such as the user's industry, company size, transactions previously processed, and preferred areas;

[0027] The similarity between users is analyzed using a collaborative filtering algorithm to obtain similarity analysis results. Based on the similarity analysis results, several users to be identified corresponding to the target user are determined, and policies and service guides corresponding to the users to be identified are determined. Personalized government service recommendation information is generated based on the user profile, the structured information, the policies, and the service guides.

[0028] Optionally, the step of integrating the initial government response and the government recommendation information using a text-image generation model to obtain a target government response including interactive diagrams, and then using the target government response to answer the government question, includes:

[0029] Extract flowcharts corresponding to government processes from the knowledge graph, including government departments, process sequence, and material information. Then, use a graphic generation model to generate interactive diagrams in scalable vector graphics format based on the flowcharts.

[0030] By using a text-image generation model and integrating the initial government response and the government recommendation information based on the interactive diagram, a government response to be processed is obtained. Image elements in the knowledge file are identified and replaced with image tags to obtain a tag-replaced image.

[0031] The image to be replaced with the label is uploaded to the object storage service, and the object storage service is called to generate a corresponding unique identifier. When retrieving knowledge fragments, the corresponding image is obtained from the object storage service based on the unique identifier and replaced to obtain a replacement result. The replacement result is then used to process the government response to be processed to obtain a target government response, which is then used to answer the government question.

[0032] The interactive diagram includes a department navigation bar, a process progress bar, and a materials list floating window. The department navigation bar allows users to click on a department name and jump to the corresponding department's processing guide page to obtain the relevant service information and requirements. The process progress bar displays the current processing progress in real time and allows users to click on the progress bar to view the details and requirements of each process. The materials list floating window displays the materials and precautions for each process and allows users to adjust the size of the diagram view.

[0033] Secondly, this application provides a government affairs question-and-answer device based on image and text understanding, including:

[0034] The image data determination module is used to acquire initial image data corresponding to government affairs issues, preprocess the initial image data to obtain image data to be processed, and then use optical character recognition technology, image classification technology and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells and cell text.

[0035] The matching result determination module is used to perform association analysis on the image information, and generate semantic relationships and logical connections based on the association analysis results to obtain structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores triple relationships between policy nodes, event nodes, and material nodes.

[0036] The government affairs recommendation information generation module is used to generate an initial government affairs response corresponding to the government affairs question by using a context-aware model and based on the matching results and dialogue context information. Then, it constructs a user profile based on the user's historical behavior data, and determines the corresponding policy content in the knowledge graph based on the user profile and the structured information to obtain personalized government affairs recommendation information.

[0037] The government response generation module is used to integrate the initial government response and the government recommendation information using a graphic generation model to obtain a target government response including interactive graphics, so as to answer the government question using the target government response.

[0038] Thirdly, this application provides an electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned government affairs question-and-answer method based on image and text understanding.

[0041] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned government affairs question-and-answer method based on text and image understanding.

[0042] As can be seen from the above, before performing government question-and-answer based on image-text understanding, this application needs to acquire initial image data corresponding to the government question, preprocess the initial image data to obtain image data to be processed, and then use optical character recognition technology, image classification technology, and object detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text; perform association analysis on the image information, and generate structured information by matching semantic relationships and logical connections based on the association analysis results; then construct a knowledge graph to match the structured information with the knowledge graph to obtain matching results; the knowledge graph stores triple relationships between policy nodes, item nodes, and material nodes; use a context-aware model and generate an initial government response corresponding to the government question based on the matching results and dialogue context information; then construct a user profile based on user historical behavior data, and determine the corresponding policy content in the knowledge graph based on the user profile and structured information to obtain personalized government recommendation information; use an image-text generation model to integrate the initial government response and government recommendation information to obtain a target government response including interactive diagrams, so as to answer the government question using the target government response.

[0043] Therefore, this application first needs to acquire initial image data corresponding to the government affairs issue, and preprocess the initial image data to obtain image data to be processed. Then, optical character recognition technology, image classification technology, and object detection technology are used to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text. Second, the image information is subjected to association analysis, and the association analysis results are matched to generate semantic relationships and logical connections to obtain structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores the triple relationships between policy nodes, matter nodes, and material nodes. Finally, an initial government affairs response corresponding to the government affairs issue is generated using a context-aware model based on the matching results and dialogue context information. Then, a user profile is constructed based on the user's historical behavior data to determine the corresponding policy content in the knowledge graph based on the user profile and structured information, obtaining personalized government affairs recommendation information. The initial government affairs response and government affairs recommendation information are integrated using a graph-text generation model to obtain a target government affairs response including interactive diagrams, so as to answer the government affairs issue using the target government affairs response. This improves the efficiency of answering government questions based on text and image understanding, thereby enhancing the user experience. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This application discloses a flowchart of a government affairs question-and-answer method based on graphic and text understanding.

[0046] Figure 2 This is a schematic diagram of the structure of a government affairs question-and-answer device based on image and text understanding disclosed in this application;

[0047] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Currently, with the deepening of digital government construction, government services are transforming from "offline windows" to "online intelligence." However, users still face three major pain points: fragmented information access, lagging policy updates, and disjointed service experiences. Meanwhile, existing technologies suffer from limitations such as single-modality constraints, delayed knowledge updates, and insufficient interactive experiences. Therefore, this application provides a government question-answering method based on graph-text understanding, which can improve the efficiency of government question-answering in the process.

[0050] See Figure 1 As shown, this embodiment of the invention discloses a government affairs question-and-answer method based on image and text understanding, including:

[0051] Step S11: Obtain initial image data corresponding to the government affairs issue, and preprocess the initial image data to obtain image data to be processed. Then, use optical character recognition technology, image classification technology and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, signature key elements, data cells and cell text.

[0052] In this embodiment, the application adopts a layered architecture design, including a text-image joint understanding module, a dynamic knowledge engine module, a multi-turn dialogue management module, a personalized recommendation module, a text-image generation module, and a knowledge slice management module.

[0053] The image-text joint understanding module is a key component for the deep analysis and fusion of user-uploaded image and text information. It aims to accurately extract valuable information from complex and diverse image and text data, providing structured and standardized data support for subsequent business processing. First, in the data transmission stage, it supports uploading various file formats, including text, images, and PDF (Portable Document Format), strictly adhering to the HTTPS protocol (HyperTextTransfer Protocol Secure) and utilizing its encryption mechanism to ensure data security during transmission, preventing data leakage and tampering. Second, it preprocesses uploaded images, including image enhancement, noise reduction, and normalization, to improve image quality and provide a solid foundation for subsequent image recognition and analysis. Then, for text data, it performs format standardization processing, removing irrelevant characters and special symbols.

[0054] Specifically, the process involves acquiring initial image data corresponding to the government issue, preprocessing the initial image data to obtain image data to be processed, and then using optical character recognition (OCR), image classification, and object detection technologies to determine the image information corresponding to the image data to be processed, including text information, seals, key signature elements, data cells, and cell text. This can include: acquiring initial image data corresponding to the government issue and performing image enhancement, denoising, and normalization operations on the initial image data to obtain image data to be processed; extracting text information from the image data to be processed using an image recognition engine and OCR technology to obtain the text information corresponding to the image data to be processed; extracting seals from the image data to be processed using image classification technology to obtain the corresponding seals, and then extracting key signature elements from the image data to be processed using image classification technology to obtain the key signature elements; and extracting each data cell from the image data to be processed using object detection technology and extracting text from each data cell to obtain cell text.

[0055] Step S12: Perform association analysis on the image information, and generate structured information by matching semantic relationships and logical connections based on the association analysis results. Then, construct a knowledge graph to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores triple relationships between policy nodes, event nodes, and material nodes.

[0056] In this embodiment, a preset image recognition engine is used to analyze the image information. This engine, trained and optimized with a large amount of document data, supports layout analysis of over 100 types of documents, including business licenses and ID cards. In one specific implementation, the recognition accuracy for key information on the documents, such as "company name" and "unified social credit code," is ≥95.7%, providing highly accurate data for subsequent business processes. Subsequently, this embodiment associates the data extracted from the image with text descriptions, and then analyzes the semantic relationships and logical connections between the image and text information using preset rules and algorithms. For example, the company name extracted from the business license photo is matched and associated with the company name mentioned in the registration application text. Then, structured application information is generated based on the associated information and stored and displayed in a unified format. This structured application information facilitates subsequent processing and analysis by the system, and also provides users with clear and standardized application data for easy viewing and verification.

[0057] Specifically, the process involves performing association analysis on image information, matching the association analysis results with semantic relationships and logical connections to generate structured information, and then constructing a knowledge graph. This can include: performing association analysis on various pieces of information in the image information to obtain association analysis results; then matching the semantic relationships of each association analysis result to obtain semantic relationship matching results; and finally generating logical connections between the semantic relationship matching results to obtain structured information. The process also involves determining policy attributes corresponding to the structured information, including policy identifiers, policy names, release dates, and policy content, to define policy nodes based on these policy attributes; determining item attributes corresponding to the structured information, including item names, handling departments, and handling procedures, to define item nodes based on these item attributes; and determining material attributes corresponding to the structured information, including material names and material requirements, to define material nodes based on these material attributes. Finally, the process involves determining triplet relationships between policy nodes, item nodes, and material nodes, and then storing these triplet relationships using a graph database. These triplet relationships support knowledge retrieval and reasoning using a graph query language.

[0058] In this embodiment, the dynamic knowledge engine module is responsible for the dynamic management and updating of system knowledge, ensuring that the system can promptly acquire the latest government policy information and accurately integrate it into the knowledge system, providing users with real-time and accurate policy interpretation and service guidance. First, a dynamic knowledge graph of "policy-item-material" triple relationships is constructed, and then the Neo4j graph database is used to store the knowledge graph. The Neo4j graph database can efficiently store and query graph data and clearly present the complex relationships between policies, items, and materials, supporting rapid knowledge retrieval and reasoning. It is worth noting that the nodes and edges in the knowledge graph have rich attribute information. For example, policy nodes include attributes such as policy ID, policy name, release time, and policy content; item nodes include attributes such as item name, handling department, and handling process; and material nodes include attributes such as material name and material requirements. Secondly, the system can monitor the change log of the government policy system in real time, and automatically trigger the knowledge graph update process when a new policy is detected. In other words, this application embodiment can extract key information such as the ID and content of new policies through efficient log parsing technology, and use the reasoning engine of the graph database to reason about the implicit relationships in the knowledge graph. For example, based on the relationships between policies and the conditions for handling matters, policy recommendations and service paths that meet the specific circumstances of users can be inferred.

[0059] Specifically, matching structured information with the knowledge graph to obtain matching results can include: polling the government policy system interface according to a preset time period to monitor the change log of the government policy system in real time through the government policy system interface, and parsing the policy identifier, change type, and content fields in the change log to obtain the parsing results; if the change type in the parsing results indicates data update, updating the content attributes corresponding to the policy nodes in the knowledge graph, and updating the material list in the change log using preset material update rules to obtain update results, and matching the structured information with the knowledge graph based on the update results to obtain matching results.

[0060] Step S13: Utilize a context-aware model and generate an initial government response corresponding to the government issue based on the matching results and dialogue context information. Then, construct a user profile based on the user's historical behavior data. Based on the user profile and the structured information, determine the corresponding policy content in the knowledge graph to obtain personalized government recommendation information.

[0061] In this embodiment, the dynamic knowledge engine module aims to achieve natural and smooth multi-turn dialogue interaction between the system and the user. By understanding the user's contextual information and intent, it provides accurate and targeted responses, improving user experience and interaction efficiency. Subsequently, a context-aware model is constructed using the Transformer architecture. This model can effectively maintain the dialogue state and record the user's historical questions and the system's response information. Through the analysis and understanding of contextual information, the model can accurately grasp the user's intent and needs, avoiding misunderstandings caused by lost context, and supports more than 10 rounds of contextual referencing. During multi-turn dialogues, this embodiment can generate reasonable responses based on the user's current question and historical dialogue content. For example, in a multi-turn dialogue where a user inquires about the company registration process, the system can remember the steps and points of interest the user previously asked about, providing coherent and accurate answers. When the system detects that the user's question is ambiguous or unclear, it automatically generates targeted clarification questions. Natural language processing technology is used to analyze the semantics of the user's question, identify potentially ambiguous keywords or phrases, and then generate clarification questions according to preset rules and templates.

[0062] Specifically, generating an initial government response corresponding to a government issue using a context-aware model and based on matching results and dialogue context information can include: recording the user's historical question information to determine the dialogue context information corresponding to the matching results, and using a context-aware model to identify the user's intent in the dialogue context information to obtain the user intent identification result; wherein the architecture corresponding to the context-aware model is the Transformer architecture; when the user intent identification result represents an ambiguous user question, natural language processing technology is used to identify ambiguous keywords in the user question, and then preset rules and preset templates are used to generate targeted questions corresponding to the ambiguous keywords, so as to guide the user to supplement clear information with targeted questions, obtain new dialogue context information, and then generate an initial government response based on the dialogue context information.

[0063] Furthermore, this embodiment utilizes a personalized recommendation module and, based on user profiling and collaborative filtering algorithms, provides users with personalized policy recommendations and service guides to meet their diverse needs, thereby improving the efficiency and satisfaction of users in obtaining information. It then collects multi-dimensional user information, including enterprise type, application history, browsing records, and search keywords. Through analysis and mining of this data, a detailed user profile is constructed. This user profile includes information on the user's basic attributes, interests, preferences, and behavioral characteristics. For example, for a small-to-medium-sized technology enterprise, its user profile might include information such as the enterprise's industry (technology sector), enterprise size (small), previous high-tech enterprise certification, and frequent searches for science and technology innovation policies. Subsequently, a collaborative filtering algorithm based on user profiling and behavior is used to provide personalized recommendations. This algorithm analyzes the similarity between users to find other users with similar interests to the target user, and then recommends policies or service guides favored by these users to the target user.

[0064] Specifically, user profiles are constructed based on users' historical behavior data. These profiles, along with structured information, are used to determine corresponding policy content within a knowledge graph, resulting in personalized government service recommendations. This process includes: collecting user information corresponding to each user, such as company type, processing history, browsing records, and search keywords; utilizing data analysis and mining techniques to generate multi-dimensional user profiles based on this information; user profiles including the user's industry, company size, previously processed matters, and preferred areas; analyzing the similarity between users using collaborative filtering algorithms to obtain similarity analysis results; identifying several potential users corresponding to the target user based on these results; and determining the corresponding policies and service guides for each potential user; and generating personalized government service recommendations based on user profiles, structured information, policies, and service guides.

[0065] Step S14: Integrate the initial government response and the government recommendation information using a graphic generation model to obtain a target government response including interactive graphics, so as to use the target government response to answer the government question.

[0066] In this embodiment, a graphic generation module is used to transform complex business processes and knowledge information into intuitive and interactive graphic diagrams, providing users with clear and convenient service guidance and knowledge display. The Inspur Cloud graphic generation model is combined with a flowchart for graphic generation. It is worth mentioning that the above model has powerful image generation capabilities, capable of generating high-quality and diverse images based on the input text description. Subsequently, combined with the flowchart, each link and relationship in the business process can be accurately transformed into graphical elements. When a user initiates a query request, the system extracts relevant process data from the knowledge graph, including the departments involved, links, and the sequence of links. For example, when a user inquires about the "enterprise registration process," the system extracts detailed process data for multiple departments and links. That is, the graphic generation module can transform the extracted process data into an interactive diagram. The diagram includes a department navigation bar, allowing users to quickly jump to the specific department's service guide page by clicking on the department name in the navigation bar to obtain detailed service information and requirements for that department. In addition, a progress bar is set up to display the current processing progress in real time, so users can clearly understand which stage of the business registration they are in and how many steps are left to complete the entire process.

[0067] Furthermore, this application embodiment employs a knowledge slice management module to perform fine-grained slice management of knowledge in the system, improving knowledge reusability and maintainability, and facilitating knowledge updates and expansion. First, based on the knowledge's theme, function, and application scenario, the knowledge in the system is divided into multiple fine-grained slices. In the first specific embodiment, this application embodiment divides enterprise registration-related knowledge into multiple knowledge slices such as company name registration, unified social credit code application, and business scope determination. A hierarchical slice structure is adopted to organize related knowledge slices together, forming a complete knowledge system. In the second specific embodiment, this application embodiment uses knowledge slices from each specific stage as child nodes and the overall enterprise registration process as the parent node in the enterprise registration knowledge system, facilitating knowledge management and retrieval. Second, the divided knowledge slices are stored in a dedicated database, and each knowledge slice is assigned a unique identifier to facilitate knowledge retrieval and referencing. Finally, the knowledge slices are periodically reviewed and optimized, checking their accuracy, completeness, and timeliness, deleting outdated knowledge slices, and supplementing with new knowledge content to ensure the quality and usability of the knowledge slices.

[0068] Specifically, the process involves integrating initial government responses and government recommendation information using a graph-to-text generation model to obtain target government responses, including interactive diagrams. These target responses are then used to address government issues. This can include: extracting flowcharts from a knowledge graph that correspond to government processes, including government departments, process sequences, and material information; then using a graph-to-text generation model to generate scalable vector graphics interactive diagrams based on the flowcharts; integrating the initial government responses and government recommendation information using the graph-to-text generation model and the interactive diagrams to obtain the government responses to be processed; identifying image elements in the knowledge files and replacing them with image tags to obtain tag-replaced images; uploading the tag-replaced images to an object storage service and using the object storage service to generate corresponding unique identifiers. When retrieving knowledge fragments, the system retrieves the corresponding image from the object storage service based on a unique identifier and replaces it to obtain the replacement result. This replacement result is then used to process the pending government response to obtain the target government response, which in turn addresses the government issue. The interactive diagram includes a department navigation bar, a process progress bar, and a materials list floating window. The department navigation bar allows users to click on a department name and navigate to the corresponding department's processing guide page to obtain relevant service information and requirements. The process progress bar displays the current processing progress in real time and allows users to click on the progress bar to view details and requirements for each process. The materials list floating window displays the materials and precautions for each process and allows users to adjust the size of the diagram view.

[0069] As can be seen from the above, the embodiments of this application first need to obtain initial image data corresponding to the government affairs issue, and preprocess the initial image data to obtain image data to be processed. Then, optical character recognition technology, image classification technology, and target detection technology are used to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text. Second, the image information is analyzed for association, and the results of the association analysis are matched to generate semantic relationships and logical connections to obtain structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores the triple relationships between policy nodes, item nodes, and material nodes. Finally, an initial government affairs response corresponding to the government affairs issue is generated using a context-aware model based on the matching results and dialogue context information. Then, a user profile is constructed based on the user's historical behavior data to determine the corresponding policy content in the knowledge graph based on the user profile and structured information, and obtain personalized government affairs recommendation information. The initial government affairs response and government affairs recommendation information are integrated using a graph-text generation model to obtain a target government affairs response including interactive diagrams, so as to answer the government affairs issue using the target government affairs response. This improves the efficiency of answering government questions based on text and image understanding, thereby enhancing the user experience.

[0070] Accordingly, see Figure 2 As shown, this application also provides a government affairs question-and-answer device based on image and text understanding, including:

[0071] The image data determination module 11 is used to acquire initial image data corresponding to the government affairs issue, preprocess the initial image data to obtain image data to be processed, and then use optical character recognition technology, image classification technology and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells and cell text.

[0072] The matching result determination module 12 is used to perform association analysis on the image information, and generate semantic relationships and logical connections based on the association analysis results to obtain structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain a matching result. The knowledge graph stores the triplet relationships between policy nodes, event nodes, and material nodes.

[0073] The government affairs recommendation information generation module 13 is used to generate an initial government affairs response corresponding to the government affairs question by using a context-aware model and based on the matching results and dialogue context information. Then, it constructs a user profile based on the user's historical behavior data, and determines the corresponding policy content in the knowledge graph based on the user profile and the structured information to obtain personalized government affairs recommendation information.

[0074] The government response generation module 14 is used to integrate the initial government response and the government recommendation information using a graphic generation model to obtain a target government response including interactive graphics, so as to use the target government response to answer the government question.

[0075] In some specific embodiments, the image data determination module 11 may specifically include:

[0076] The image data processing unit is used to acquire initial image data corresponding to government affairs issues, and to perform image enhancement, denoising and normalization operations on the initial image data to obtain image data to be processed.

[0077] The text information extraction unit is used to extract text information from the image data to be processed using an image recognition engine and optical character recognition technology, so as to obtain text information corresponding to the image data to be processed.

[0078] The element extraction unit is used to extract the seal from the image data to be processed using image classification technology to obtain the corresponding seal, and then use the image classification technology to extract the key signature elements from the image data to be processed to obtain the key signature elements.

[0079] The cell text determination unit is used to extract each data cell in the image data to be processed using object detection technology, and to extract the text from each data cell to obtain the cell text.

[0080] In some specific embodiments, the matching result determination module 12 may specifically include:

[0081] The matching result determination unit is used to perform association analysis on each piece of information in the image information to obtain association analysis results, then perform semantic relationship matching on each of the association analysis results to obtain semantic relationship matching results, and generate logical connections between each of the semantic relationship matching results to obtain structured information.

[0082] The policy attribute determination unit is used to determine the policy attributes corresponding to the structured information, including policy identifier, policy name, release time and policy content, so as to define policy nodes based on the policy attributes. Then, it determines the matter attributes corresponding to the structured information, including matter name, handling department and handling process, so as to define matter nodes based on the matter attributes.

[0083] The material attribute determination unit is used to determine the material attributes, including material name and material requirements, corresponding to the structured information, to define material nodes based on the material attributes, and to determine the triple relationships between the policy node, the matter node, and the material node. Then, the triple relationships are stored using a graph database. The triple relationships support knowledge retrieval and reasoning using a graph query language.

[0084] In some specific embodiments, the matching result determination module 12 may specifically include:

[0085] The parsing result determination unit is used to poll the government policy system interface according to a preset time period, so as to monitor the change log of the government policy system in real time through the government policy system interface, and parse the policy identifier, change type and content fields in the change log to obtain the parsing result;

[0086] The matching result determination subunit is used to update the content attributes corresponding to the policy nodes in the knowledge graph when the change type in the parsing result is characterized as data update, and to update the material list in the change log using preset material update rules to obtain an update result, so as to match the structured information with the knowledge graph based on the update result to obtain a matching result.

[0087] In some specific embodiments, the government recommendation information generation module 13 may specifically include:

[0088] The user intent recognition result determination unit is used to record the user's historical question information, determine the dialogue context information corresponding to the matching result based on the historical question information, and use a context-aware model to perform user intent recognition on the dialogue context information to obtain the user intent recognition result; wherein, the architecture corresponding to the context-aware model is the Transformer architecture.

[0089] The initial government response determination unit is used to identify ambiguous keywords in the user's question when the user intent recognition result indicates that the user's question is ambiguous. Then, it uses natural language processing technology to identify the ambiguous keywords and generates targeted questions corresponding to the ambiguous keywords using preset rules and preset templates. The targeted questions are used to guide the user to supplement clear information and obtain new dialogue context information. Then, an initial government response is generated based on the dialogue context information.

[0090] In some specific embodiments, the government recommendation information generation module 13 may specifically include:

[0091] The user profile generation unit is used to collect user information corresponding to users, including enterprise type, transaction history, browsing history and search keywords, and to generate corresponding multi-dimensional user profiles based on the user information using data analysis and mining technology; the user profile includes information such as the user's industry, enterprise size, transactions previously handled and preferred areas;

[0092] The government affairs recommendation information generation subunit is used to analyze the similarity between users using a collaborative filtering algorithm to obtain similarity analysis results. Based on the similarity analysis results, it determines several users to be identified corresponding to the target user and determines the policies and service guides corresponding to the users to be identified. Based on the user profile, the structured information, the policies and the service guides, it generates personalized government affairs recommendation information.

[0093] In some specific embodiments, the government response generation module 14 may specifically include:

[0094] The flowchart generation unit is used to extract flowcharts corresponding to government processes from the knowledge graph, including government departments, process sequence and material information, and then use a graphic generation model to generate interactive diagrams in scalable vector graphics format based on the flowcharts.

[0095] The tag replacement image determination unit is used to use a graphic generation model and integrate the initial government response and the government recommendation information based on the interactive illustration to obtain the government response to be processed, and to identify image elements in the knowledge file to replace the image elements with image tags to obtain a tag replacement image;

[0096] An identifier generation unit is used to upload the label replacement image to an object storage service and call the object storage service to generate a corresponding unique identifier. When retrieving knowledge fragments, the unit retrieves the corresponding image from the object storage service based on the unique identifier and replaces it, obtaining a replacement result. This replacement result is then used to process the pending government response to obtain a target government response, which is then used to answer the government issue. The interactive diagram includes a department navigation bar, a process progress bar, and a materials list floating window. The department navigation bar allows users to click on a department name and jump to the corresponding department's processing guide page to obtain the relevant service information and requirements. The process progress bar displays the current processing progress in real time and allows users to click on the progress bar to view the details and requirements of each process. The materials list floating window displays the materials and precautions for each process and allows users to adjust the size of the diagram view.

[0097] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image-text understanding-based government affairs question-answering method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the image-text understanding-based government affairs question-and-answer method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for answering government affairs questions based on image and text understanding. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A government affairs question-and-answer method based on image and text understanding, characterized in that, include: Initial image data corresponding to government affairs issues is acquired, and the initial image data is preprocessed to obtain image data to be processed. Then, optical character recognition technology, image classification technology, and target detection technology are used to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text. The image information is subjected to association analysis, and the association analysis results are matched with semantic relationships and logical connections to generate structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores triplet relationships between policy nodes, event nodes, and material nodes. An initial government response corresponding to the government issue is generated using a context-aware model and based on the matching results and dialogue context information. Then, a user profile is constructed based on the user's historical behavior data. Based on the user profile and the structured information, the corresponding policy content is determined in the knowledge graph to obtain personalized government recommendation information. By integrating the initial government response and the government recommendation information using a text-image generation model, a target government response including interactive illustrations is obtained, which is then used to answer the government question.

2. The government affairs question-and-answer method based on image and text understanding according to claim 1, characterized in that, The process involves acquiring initial image data corresponding to the government affairs issue, preprocessing the initial image data to obtain image data to be processed, and then using optical character recognition technology, image classification technology, and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells, and cell text, including: Acquire initial image data corresponding to the government affairs issue, and perform image enhancement, denoising and normalization operations on the initial image data to obtain image data to be processed; The image data to be processed is extracted by using an image recognition engine and optical character recognition technology to obtain the text information corresponding to the image data to be processed. The seal is extracted from the image data to be processed using image classification technology to obtain the corresponding seal. Then, the key signature elements are extracted from the image data to be processed using the image classification technology to obtain the key signature elements. The target detection technology is used to extract each data cell from the image data to be processed, and the text in each data cell is extracted to obtain the cell text.

3. The government affairs question-and-answer method based on image and text understanding according to claim 1, characterized in that, The process involves performing association analysis on the image information, matching semantic relationships and logical connections based on the association analysis results to generate structured information, and then constructing a knowledge graph, including: The association analysis of each piece of information in the image information is performed to obtain the association analysis results. Then, the semantic relationship matching of each association analysis result is performed to obtain the semantic relationship matching results. Finally, the logical connection of each semantic relationship matching result is generated to obtain structured information. The policy attributes corresponding to the structured information, including policy identifier, policy name, release time and policy content, are determined to define policy nodes based on the policy attributes. Then, the item attributes corresponding to the structured information, including item name, handling department and handling process, are determined to define item nodes based on the item attributes. The material attributes, including material name and material requirements, corresponding to the structured information are determined. Material nodes are defined based on the material attributes, and triple relationships between the policy node, the matter node, and the material node are determined. Then, the triple relationships are stored using a graph database. The triple relationships support knowledge retrieval and reasoning using a graph query language.

4. The government affairs question-and-answer method based on image and text understanding according to claim 1, characterized in that, The step of matching the structured information with the knowledge graph to obtain the matching result includes: The government policy system interface is polled according to a preset time period to monitor the change log of the government policy system in real time through the government policy system interface, and the policy identifier, change type and content fields in the change log are parsed to obtain the parsing results; If the change type in the parsing result is characterized as a data update, the content attributes corresponding to the policy nodes in the knowledge graph are updated, and the material list in the change log is updated using preset material update rules to obtain an update result. Based on the update result, the structured information is matched with the knowledge graph to obtain a matching result.

5. The government affairs question-and-answer method based on image and text understanding according to claim 1, characterized in that, The step of generating an initial government response corresponding to the government issue using a context-aware model and based on the matching results and dialogue context information includes: Record the user's historical question information to determine the dialogue context information corresponding to the matching result based on the historical question information, and use a context-aware model to perform user intent recognition on the dialogue context information to obtain the user intent recognition result; wherein, the architecture corresponding to the context-aware model is the Transformer architecture; When the user intent recognition result indicates that the user's question is ambiguous, natural language processing technology is used to identify ambiguous keywords in the user's question. Then, a targeted question corresponding to the ambiguous keyword is generated using preset rules and preset templates. The targeted question is used to guide the user to supplement clear information and obtain new dialogue context information. Then, an initial government response is generated based on the dialogue context information.

6. The government affairs question-and-answer method based on image and text understanding according to claim 1, characterized in that, The process of constructing user profiles based on historical user behavior data, and determining corresponding policy content in the knowledge graph based on these user profiles and the structured information to obtain personalized government service recommendation information, includes: Collect user information corresponding to each user, including company type, transaction history, browsing history, and search keywords, and use data analysis and mining techniques to generate corresponding multi-dimensional user profiles based on the user information; the user profiles include information such as the user's industry, company size, transactions previously processed, and preferred areas; The similarity between users is analyzed using a collaborative filtering algorithm to obtain similarity analysis results. Based on the similarity analysis results, several users to be identified corresponding to the target user are determined, and policies and service guides corresponding to the users to be identified are determined. Personalized government service recommendation information is generated based on the user profile, the structured information, the policies, and the service guides.

7. The government affairs question-answering method based on image and text understanding according to any one of claims 1 to 6, characterized in that, The process of integrating the initial government response and the government recommendation information using a text-image generation model to obtain a target government response including interactive illustrations, and then using the target government response to answer the government question, includes: Extract flowcharts corresponding to government processes from the knowledge graph, including government departments, process sequence, and material information. Then, use a graphic generation model to generate interactive diagrams in scalable vector graphics format based on the flowcharts. By using a text-image generation model and integrating the initial government response and the government recommendation information based on the interactive diagram, a government response to be processed is obtained. Image elements in the knowledge file are identified and replaced with image tags to obtain a tag-replaced image. The image to be replaced with the label is uploaded to the object storage service, and the object storage service is called to generate a corresponding unique identifier. When retrieving knowledge fragments, the corresponding image is obtained from the object storage service based on the unique identifier and replaced to obtain a replacement result. The replacement result is then used to process the government response to be processed to obtain a target government response, which is then used to answer the government question. The interactive diagram includes a department navigation bar, a process progress bar, and a materials list floating window. The department navigation bar allows users to click on a department name and jump to the corresponding department's processing guide page to obtain the relevant service information and requirements. The process progress bar displays the current processing progress in real time and allows users to click on the progress bar to view the details and requirements of each process. The materials list floating window displays the materials and precautions for each process and allows users to adjust the size of the diagram view.

8. A government affairs question-and-answer device based on image and text understanding, characterized in that, include: The image data determination module is used to acquire initial image data corresponding to government affairs issues, preprocess the initial image data to obtain image data to be processed, and then use optical character recognition technology, image classification technology and target detection technology to determine the image information corresponding to the image data to be processed, including text information, seals, key elements of signatures, data cells and cell text. The matching result determination module is used to perform association analysis on the image information, and generate semantic relationships and logical connections based on the association analysis results to obtain structured information. Then, a knowledge graph is constructed to match the structured information with the knowledge graph to obtain matching results. The knowledge graph stores triple relationships between policy nodes, event nodes, and material nodes. The government affairs recommendation information generation module is used to generate an initial government affairs response corresponding to the government affairs question by using a context-aware model and based on the matching results and dialogue context information. Then, it constructs a user profile based on the user's historical behavior data, and determines the corresponding policy content in the knowledge graph based on the user profile and the structured information to obtain personalized government affairs recommendation information. The government response generation module is used to integrate the initial government response and the government recommendation information using a graphic generation model to obtain a target government response including interactive graphics, so as to answer the government question using the target government response.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the government affairs question-and-answer method based on graphic and text understanding as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the government affairs question-and-answer method based on graphic and text understanding as described in any one of claims 1 to 7.

Images