Government affair intelligent interaction and information extraction method and device, equipment and medium

Abstract

The invention discloses a government affair intelligent interaction and information extraction method and device, equipment and a medium, and relates to the technical field of digital government affair, digital cities, intelligent interaction and the like. The traditional manual item-by-item filling is converted into end-to-end automatic processing, so that the operation steps of a user are remarkably reduced, the filling burden of a hundred-item-level form is solved, and the filling efficiency is improved; and an ID voice corrector is adopted to realize three-layer progressive verification, so that the output ID offset of the multi-modal large model is effectively inhibited, the accurate mapping of field identifiers is ensured, and meanwhile, a lightweight knowledge graph is adopted to automatically capture logic rules among fields, interspecific compliance conflicts of government affair scenes are intercepted in real time, the logic error risk is reduced, and the filling accuracy is improved.

Description

Technical Field

[0001] This application belongs to the technical fields of digital government, digital city, and intelligent interaction, and specifically relates to a method, device, equipment, and medium for intelligent government interaction and information extraction. Background Technology

[0002] Currently, online government service applications (such as social security applications and newborn birth matters) heavily rely on users filling out forms independently, and generally suffer from the following bottlenecks: (1) Low form filling efficiency: Forms often contain dozens to hundreds of fields, which users need to fill out manually, which is time-consuming and labor-intensive. The semantics of the fields are highly specialized (such as "unified social credit code" and "standardized description of business scope"), which can easily lead to errors by non-professionals, resulting in repeated modifications. (2) Insufficient material information extraction capability: The mainstream uses traditional OCR technology to recognize documents, certificates and other materials, which can only output the original text and cannot understand the contextual semantics. (3) The logical relationship between fields is not utilized, resulting in information fragmentation. (4) Weak interactive experience: The current mainstream interactive form still relies on users manually inputting and filling out forms. The text stream output by speech recognition (ASR) is not bound to the form fields, and it is impossible to achieve "voice command → automatic form filling".

[0003] The existing technology has the following limitations:

[0004] (1) Insufficient single-modal processing capability: OCR technology only focuses on image to text conversion and does not integrate semantic understanding; ASR outputs text stream and is not bound to form fields; NLP (Natural Language Search) engine adopts traditional rule model that relies on preset keywords and has poor generalization ability (e.g., "address" can be expressed as "place of residence" or "place of permanent residence" in different scenarios).

[0005] (2) The multimodal collaborative mechanism is not in place: existing solutions are mostly serial pipelines (such as OCR→NLP→database matching), and errors accumulate step by step (OCR recognition errors directly lead to field matching failures); cross-modal information is not uniformly modeled: there is a lack of joint reasoning among material images, voice commands, and form fields.

[0006] (3) Poor adaptability to government affairs scenarios: Hard-coded rules are bound to fixed form templates (e.g., fields of item A cannot be reused for item B), which cannot dynamically adapt to the differences in government affairs in various regions; the government affairs knowledge base is not embedded, which leads to errors in field parsing and filling.

[0007] Currently, some government systems are attempting to introduce technological upgrades, but with limited success. For example, intelligent form pre-filling: this technology automatically fills in some fields (such as name, ID number, etc.) based on users' historical data, but it cannot handle new materials (such as the first upload of a property ownership certificate); RPA process automation: this technology simulates manual copying of material content to forms, relies on fixed-location data capture, and becomes ineffective when the material layout changes; using multimodal models: this technology directly calls APIs such as GPT-4V to parse materials, but the extraction of government field data requires strict matching of a preset JSON structure (such as converting "registered capital" to a numeric type), and the output format of the general model is uncontrollable.

[0008] In summary, current technologies cannot meet the needs of government services in terms of efficiency (automated form filling), accuracy (cross-modal field extraction), and user experience (natural voice interaction). There is an urgent need for an integrated solution that combines multimodal understanding, dynamic form adaptation, and enhanced government knowledge. Summary of the Invention

[0009] To address the problems of low form-filling efficiency and high error rates in existing government services caused by insufficient multimodal information collaboration capabilities, missing logical connections between fields, and the inability to accurately map unstructured data to forms, this application proposes a government intelligent interaction and information extraction method, device, equipment, and medium, which improves form-filling efficiency, accuracy, and dynamic adaptive capabilities.

[0010] Firstly, this application achieves its purpose through the following technical solution:

[0011] A method for intelligent government interaction and information extraction, comprising:

[0012] Non-image materials uploaded by users are uniformly converted into standardized image inputs into the multimodal large model. User voice input is processed into recognizable voice segments and input into the multimodal large model. Based on the business matter currently being handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints, and related rules contained in the form template are loaded into the multimodal large model in real time.

[0013] Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to generate only JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field. The user response text is the original user response content obtained by recognizing the input image and / or speech.

[0014] The business logic of the JSON array output by the multimodal large model is validated using knowledge graph technology.

[0015] In some implementations, it also includes:

[0016] Automatic form filling is performed based on a JSON array that has passed business logic validation.

[0017] In some implementations, processing the user's voice input into recognizable speech segments specifically includes:

[0018] A dual-threshold voice activity detection mechanism is used to process user voice input, including:

[0019] The short-time energy and zero-crossing rate of the speech signal are calculated in real time.

[0020] When the short-term energy exceeds a preset high threshold and reaches a preset duration, a valid voice segment is determined to have started.

[0021] For signal segments whose short-term energy is below a preset high threshold but whose zero-crossing rate is consistently above its preset threshold, they are identified as silent speech segments and merged with adjacent speech segments;

[0022] When the short-term energy is continuously lower than a preset low threshold and the zero-crossing rate is also continuously lower than its preset threshold, and the duration meets the preset silence duration threshold, then the current valid voice segment is determined to have ended.

[0023] In some implementations, the dynamic prompt word engineering constructs a dynamic, structured prompt word instruction generation model, deeply integrating the form template of the current business item, predefined question word sequences, and user response text to generate prompt word instructions that constrain the output of the multimodal large model, aligning the output of the multimodal large model with the form structure. The specific process includes:

[0024] The input form template is parsed, and field attributes are extracted to generate a field mapping table, which contains four columns of data: question word sequence, form field name, field type, and conversion rule;

[0025] The input user response text is segmented into a sequence of semantic units that match the question wording sequence;

[0026] Based on the field types and conversion rules defined in the field mapping table, the value of each semantic unit is standardized and converted.

[0027] The matched and transformed field data are forcibly encapsulated into standardized JSON arrays according to the output specifications; each JSON array contains only: field ID, field name, and field value;

[0028] The JSON array is double-validated using regular expressions and a syntax validator to ensure that the format is valid and the fields are effective, while filtering out any illegal or undefined fields.

[0029] In some implementations, the process of achieving multimodal large model output and form structure alignment based on the dynamic prompt word engineering further includes:

[0030] An ID semantic corrector is used to perform three layers of semantic verification on each field ID output by the multimodal large model. The first layer is exact matching: checking whether the field ID output by the multimodal large model exists in the set of correct field IDs in the form template. If no match is found, the second layer of semantic arbitration is performed: using model vectorization, the cosine similarity between the field name output by the multimodal large model and all field titles in the form template is calculated. When the highest similarity exceeds a preset threshold, the correct field ID in the form template is automatically replaced with the incorrect field ID output by the multimodal large model. If the highest similarity does not reach the preset threshold, the third layer of fictitious suppression is performed: the value of the field ID is marked as "null".

[0031] In some implementations, the use of knowledge graph technology to perform business logic validation on the JSON array output by the multimodal large model includes:

[0032] Based on the form template, a lightweight knowledge graph is constructed;

[0033] Iterate through each field output by the multimodal large model. For the current field, retrieve all association rules with the current field as the main field from the lightweight knowledge graph. If the current field does not have any association rules, mark it as "compliant" and add it to the compliant field set. If the current field has association rules, perform split processing according to the type of the rule:

[0034] If the rule type is conditional dependency, check whether the field value of the current field is equal to the preset trigger value. If so, further check whether the dependency field specified by the rule exists in the JSON array. If the dependency field is missing, record the "missing dependency field" error message to the error set.

[0035] If the rule type is a numerical constraint, check whether the field value of the current field conforms to the preset numerical range. If it does not conform, record the "value exceeds the limit" error message to the error set.

[0036] If the rule type is mutually exclusive, check whether the mutually exclusive fields specified by the rule exist in the JSON array at the same time. If mutually exclusive fields exist, record the "field mutual exclusion" error message to the error set.

[0037] Repeat the above rule validation process until all fields have been validated;

[0038] Once the rule verification is complete, the conflict resolution phase begins.

[0039] In some implementations, constructing a lightweight knowledge graph based on the form template includes:

[0040] Define a knowledge graph structure; wherein the knowledge graph structure consists of nodes and edges, each node represents a form field, each edge represents a logical relationship between fields, and the logical relationship consists of its subject field, object field, relationship type, and relationship parameters;

[0041] The hierarchical structure of the form template is parsed to extract the meaning of all fields and constraint attributes, and a JSON tree is generated;

[0042] Traverse the parsed JSON tree to identify constraint attributes;

[0043] Based on the semantics of the constraint attributes, the rules are divided into three types of logical relations;

[0044] Construct a triplet structure according to the relation type to which the logical relation belongs. The triplet structure includes a subject field, a relation type, and an object field. Transform all constraint attributes in the JSON tree into edges of the knowledge graph structure to form a structured relation network. Finally, create a set of nodes to make each form field a unique node and create a set of edges so that each constraint corresponds to one edge, thereby forming a complete knowledge graph structure.

[0045] Secondly, this application proposes a government affairs intelligent interaction and information extraction device, comprising:

[0046] The input acquisition and preprocessing module is configured to: uniformly convert non-image materials uploaded by users into standardized format image input multimodal large model; process user voice input into recognizable voice segments and input them into the multimodal large model; and dynamically obtain the form template corresponding to the business matter currently being handled by the user, and load all fields, field types, constraints and related rules contained in the form template into the multimodal large model in real time.

[0047] The structured adaptation module is configured to: dynamically inject form templates, user response text, and preset question word sequences into the prompt word instructions of the multimodal large model through dynamic prompt word engineering; control the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values; and automatically append status flags to each field; wherein the user response text is the original user response content obtained by recognizing the input image and / or speech;

[0048] In addition, the logic verification module is configured to perform business logic verification on the JSON array output by the multimodal large model using knowledge graph technology.

[0049] Thirdly, this application proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above-described methods for intelligent government interaction and information extraction.

[0050] Fourthly, this application proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described methods for intelligent government interaction and information extraction.

[0051] This application proposes a method for intelligent interaction and information extraction in government affairs. Through dynamic prompt word engineering, it binds cross-modal input (image / voice) with structured form fields, transforming traditional manual, item-by-item filling into end-to-end automated processing. This significantly reduces user steps, alleviates the burden of filling out hundreds of forms, and improves efficiency. It also employs an ID voice corrector to implement three-layer progressive verification (exact matching → semantic arbitration → fictitious suppression), effectively suppressing ID offsets in the output of large multimodal models and ensuring accurate mapping of field identifiers. Simultaneously, a lightweight knowledge graph automatically captures logical rules between fields (conditional dependencies / numerical constraints / mutual exclusion relationships), intercepting compliance conflicts unique to government scenarios in real time (such as married individuals not filling in spouse information), reducing the risk of logical errors and improving form accuracy. Furthermore, the method, through dynamic prompt word engineering, can adaptively adapt to form templates for different government affairs, and the lightweight knowledge graph is dynamically updated in real time based on the data from the input form template, improving dynamic adaptability. It also supports joint semantic reasoning between multimodal input (voice, image) and form fields, achieving semantic-level linkage and thus improving the accuracy of understanding complex user intentions.

[0052] Accordingly, the government affairs intelligent interaction and information extraction device, electronic device and computer-readable storage medium proposed in this application have the same technical effects as described above. Attached Figure Description

[0053] The accompanying drawings, which are included to provide a further understanding of the embodiments of this application and form part of this application, do not constitute a limitation on the embodiments of this application. In the drawings:

[0054] Figure 1 This is a flowchart of the government intelligent interaction and information extraction method proposed in the embodiments of this application;

[0055] Figure 2 This is a schematic diagram of the government intelligent interaction and information extraction device proposed in the embodiments of this application;

[0056] Figure 3 This is a schematic diagram of the government intelligent interaction and information extraction system architecture proposed in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the electronic device proposed in the embodiments of this application;

[0058] Figure 5 This is a schematic diagram of a computer-readable storage medium proposed in an embodiment of this application.

[0059] Figure reference numerals and corresponding component names:

[0060] 200-Government Intelligent Interaction and Information Extraction Device, 201-Input Acquisition and Preprocessing Module, 202-Structured Adaptation Module, 203-Logic Verification Module, 204-Automatic Form Filling Module, 300-Government Intelligent Interaction and Information Extraction System, 301-Input Device, 302-Output Device, 303-Processor A, 304-Memory A, 400-Electronic Device, 410-Memory B, 420-Processor B, 411-Computer Program A, 500-Computer-Readable Storage Medium, 511-Computer Program B. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0062] This application proposes a method for intelligent government interaction and information extraction, such as... Figure 1 As shown in the embodiments of this application, the government intelligent interaction and information extraction method includes the following steps:

[0063] Step 1, Multimodal Input Acquisition and Preprocessing: Convert the non-image materials uploaded by users into standardized image input multimodal large model, process the user's voice input into recognizable voice segments input multimodal large model, and dynamically obtain the form template corresponding to the business matter currently handled by the user, and load all fields, field types, constraints and related rules contained therein into the multimodal large model in real time;

[0064] Step 2, Aligning the Multimodal Large Model Output with the Form Structure: Through dynamic prompt word engineering, the form template, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input image and / or speech.

[0065] Step 3, Knowledge Graph-Driven Logic Verification: Use knowledge graph technology to perform business logic verification on the JSON array output by the multimodal large model.

[0066] Furthermore, the government intelligent interaction and information extraction method proposed in this application embodiment also includes:

[0067] Step 4, Auto-fill: Auto-fill the form based on the JSON array that has passed the business logic validation.

[0068] Furthermore, in step 1 of this application embodiment, the multimodal input acquisition and preprocessing process includes:

[0069] Raw material input processing: During the material upload stage, non-image format materials such as PDF or WORD documents uploaded by users are uniformly processed into standardized image formats (JPG or PNG), and then input into the multimodal large model for image recognition.

[0070] Voice input processing: During the user's form filling interaction phase, the microphone array captures the user's voice commands in real time. To cope with complex environmental noise interference and accurately extract valid voice commands, this application employs a dual-threshold voice activity detection mechanism based on short-time energy and zero-crossing rate. This effectively filters background noise and accurately segments continuous speech into clean, identifiable speech segments. Specifically, this dual-threshold voice activity detection mechanism includes: real-time calculation of two key parameters of the speech signal: short-time energy and zero-crossing rate; when the short-time energy continuously exceeds a preset high threshold for a preset duration (e.g., 300 milliseconds), a valid speech segment is determined to have begun. Simultaneously, for signal segments with short-time energy below the high threshold but a zero-crossing rate continuously above its preset threshold, they are identified as unvoiced segments and merged with adjacent speech segments. When the short-time energy continuously falls below a preset low threshold and the zero-crossing rate also continuously falls below its preset threshold, and the duration meets a preset silence duration threshold, the current valid speech segment is determined to have ended. This mechanism effectively filters out background noise and transient interference, ensuring that only clean, coherent, and valid user speech audio segments are reliably transmitted to the backend multimodal large model for semantic understanding and instruction parsing.

[0071] Form Data Loading: Whether in the material recognition or voice interaction phase, the multimodal big data model needs to accurately understand the user's intent and generate the correct response. This requires dynamically acquiring the specific form structure and requirements corresponding to the currently processed item. Therefore, during this step, a real-time query request needs to be initiated based on the currently processed business item (item ID) to dynamically retrieve and load the detailed form definition structure corresponding to that business item from the predefined form configuration center. This structure strictly adheres to the JSON Schema specification, clearly defining all form fields, field types, filling constraints, association rules, and business rules. The loaded form definition structure is the foundation for building accurate and dynamic prompts. It enables the multimodal big data model to understand the context, anticipate the information the user needs to fill in, and provide real-time and effective guidance, thereby driving the entire intelligent form filling process.

[0072] Furthermore, in step 2 of this embodiment, the dynamic prompt word engineering constructs different prompt words using different form template data, thereby aligning the multimodal large model output with the form structure. Specifically, the dynamic prompt word engineering constructs a dynamic, structured prompt word instruction generation model, deeply integrating the form template of the current business matter, predefined user guidance question sequences, and the user's actual input (speech-to-text or image recognition results) to generate precise prompt word instructions that constrain the multimodal large model output. This addresses the key issues of uncontrollable multimodal large model output format, semantic drift, and difficulty in accurately binding target form fields, ensuring that the multimodal large model output strictly adheres to the preset form field ID, name, and value triplet structure. Specifically, the dynamic prompt word engineering mainly includes:

[0073] The template engine stores the basic instruction template framework and defines the core roles, capability requirements, processing flow specifications, and output specifications of the model. Specifically, this includes: role definition, clearly defining the model's role as an expert in filling out government forms; core capability definition, emphasizing the model's ability to handle three key elements: accurately understanding the intent of pre-defined question sequences and fields, precisely parsing the semantics of user responses, and accurately matching the parsed results with the target form field definitions. It also clearly defines processing rules for different field types (such as text input, single-choice, multiple-choice, date, etc.) (e.g., directly extracting semantic values, mapping dictionary values, converting date formats, converting numerical units, etc.); processing flow specifications, stipulating that the model must follow the standard process of "parsing → conversion → matching": identifying key entities in user responses; performing data standardization transformation based on field types; generating structured output by associating form field IDs; and output specifications, strictly limiting the model to outputting only standard JSON object arrays containing field IDs (or names) and values, and forcibly appending status flags ("1" for a successful match, "0" for no match). Outputting fields not defined in the JSON Schema or missing required fields is explicitly prohibited.

[0074] Dynamic input is responsible for injecting dynamically changing business context information into the basic instruction template in real time, forming the final large-scale model prompts tailored to specific business needs. Specifically, the input information includes: a sequence of question prompts, i.e., predefined text guiding users to answer specific form fields (e.g., "Please enter the company's registered address"); user response text, which comes from the user's actual response via speech recognition (ASR) or image recognition (OCR + multimodal understanding); and form data, i.e., the detailed form template corresponding to the currently processed business item, containing information such as the ID, title, data type, constraint rules, and option dictionary for all fields.

[0075] In addition, a constraint enforcer is used to execute strict post-processing rules after model inference, forcing the model output to conform to predefined format and content specifications. The enforced specifications are as follows: ensure that the final output contains only a JSON array object containing field IDs, field names, and field values; when at least one valid field value is successfully matched, append a status flag {"state": "1"} to each corresponding output object. When the model determines that the user's answer has no valid association with any form field, it returns [{"state": "0"}], effectively preventing null values ​​or invalid data from polluting the final form.

[0076] The specific process for multimodal large model output and form structure alignment is as follows:

[0077] The input form template is parsed, and key field attributes are extracted to generate a field mapping table. The field mapping table contains four columns of data: Question wording sequence: original question description text; Form field name: field name; Field type: character / option / date, etc.; Conversion rule: field value standardization method.

[0078] For each field, establish a mapping function:

[0079]

[0080] in, This represents the i-th field in the question word sequence. This represents the j-th field in the form. Indicates the mapping relationship. This indicates the name to be displayed in the form field. This represents the semantic similarity calculation function, which is generated and judged by the model itself. The similarity threshold is represented, and is preferably 0.85 in this embodiment of the application.

[0081] The user's response text is segmented into a sequence of semantic units that match the question word sequence. The segmentation algorithm is based on the following principles: segmenting according to the question word sequence, preserving complete semantic units, and identifying and processing compound expressions (such as "December 5, 2024, 2:30 PM").

[0082] Based on the field types and conversion rules defined in the field mapping table, the value of each semantic unit is standardized and converted. This includes: Option field processing: For single-choice or multiple-choice fields, the text of the user's answer (e.g., "female") is mapped to a predefined option dictionary value (e.g., "2"); Numeric field processing: For numeric fields, the numbers in the answer are extracted and the units are processed (e.g., converting "50,000" to 50,000), supporting unit conversion rules specific to government scenarios (e.g., converting "ten thousand yuan" to basic units); Date field processing: Various natural language or non-standard format date descriptions (e.g., "Next New Year's Day", "2025 / 1 / 1") are uniformly converted to the standard ISO format (e.g., "2025-01-01"); Text field processing: Necessary cleaning and semantic integrity preservation are performed on plain text input fields.

[0083] The matched and transformed field data is forcibly encapsulated into standardized JSON arrays according to the output specifications. Each JSON array strictly contains: field ID (a unique identifier for the form field), field name (the display name of the form field), and field value (the field value after standardized transformation).

[0084] The final output JSON array is double-validated using regular expressions and a JSON Schema syntax validator to ensure that the format is valid and the fields are valid, filtering out any illegal or undefined fields.

[0085] Furthermore, during form filling, the corresponding content needs to be filled based on the ID field of the returned JSON array as an identifier. However, due to the large model's fictitious nature and the complexity of the 16-character ID, the ID field in the output JSON array may deviate, causing the field to fail to be correctly filled into the form. Therefore, step 2 of this embodiment further includes:

[0086] An ID semantic corrector is employed to perform three layers of semantic verification on the field IDs output by the multimodal large model to address potential field ID identification errors or fictitious IDs generated by the multimodal large model. The first layer is exact matching, checking if the field ID output by the multimodal large model exists in the correct set of field IDs in the form template. If no match is found, the second layer, semantic arbitration, is initiated. The cosine similarity between the field name output by the multimodal large model and all field titles in the form template is calculated using the bge-large-zh model vectorization. When the highest similarity exceeds a preset threshold (e.g., greater than 0.8), the corresponding correct field ID in the form template automatically replaces the incorrect field ID output by the multimodal large model. If the highest similarity does not reach the preset threshold, the third layer, fictitious ID suppression, is initiated, marking the field ID value as "null" to prevent erroneous information from contaminating the final form data. The specific process of the three-layer semantic verification is as follows:

[0087] ID matching validation: This function matches and validates the ID field in the standard JSON array output by the multimodal large model against the standard ID field in the form template. The JSON array... Represented as:

[0088]

[0089] in, For the field ID (which may contain biases generated by the large model). For field names, For the corresponding field value, For the number of fields, For the i-th field.

[0090] Standard field ID set for form templates Represented as:

[0091]

[0092] in, For the standard form field ID, For field headers, For the j-th standard form field, This represents the number of standard fields in the form.

[0093] Iterate through each field object in the JSON array output by the multimodal large model and perform existence verification:

[0094]

[0095] in, To output the id value from the field object in the JSON array, This is a collection of all standard field IDs in the form template. For matching and validation, true indicates validation passed, and false indicates validation failed.

[0096] Fields that pass validation are directly entered into the output queue, preserving the original data structure.

[0097]

[0098] in, To verify the JSON array composed of the ID field.

[0099] Semantic similarity arbitration: For IDs that do not match precisely, cross-modal semantic mapping is achieved through field name vectorization. Specifically, for fields that fail validation, name vectorization comparison is initiated. First, the field names output by the multimodal large model and the standard field titles in the form template are vectorized, represented as follows:

[0100]

[0101]

[0102] in, Vector encoding for the bge-large-zh model, Output semantic vectors for field names in a multimodal large model. A semantic vector for standard field headings;

[0103] Then, the cosine similarity between the semantic vector of each output field name and the semantic vector of the standard field title is calculated:

[0104]

[0105] in, The cosine similarity score is denoted as 1, and the closer it is to 1, the higher the semantic similarity.

[0106] Obtain the maximum similarity between the semantic vector of each output field name and the semantic vector of the standard field title. If the similarity exceeds a threshold, replace the field ID with the standard field ID from the form template.

[0107]

[0108] in, Index the standard field for the best match. A function to obtain the maximum value. As the similarity threshold, 0.8 is preferably used in the embodiments of this application.

[0109] Fictitious suppression and ID repair: Fields whose IDs in the JSON array output by the multimodal large model do not match the standard field IDs in the form template and fail to pass semantic similarity arbitration are identified as fictitious fields in the large model, and their fields in the JSON array are set to null.

[0110] Through the above three layers of semantic validation, a more accurate and reliable standard JSON array is finally output.

[0111] Furthermore, in step 3 of this application embodiment, the knowledge graph-driven logic verification process includes:

[0112] Based on the constraints and related rules of the form template, a lightweight knowledge graph is constructed. The constructed knowledge graph is defined as follows:

[0113]

[0114]

[0115]

[0116] in, This constitutes a complete knowledge graph structure. This is a set of nodes in the graph, where each node represents a form field. For a single node, This is a unique identifier for the field in the form template. For field type, This is the set of edges in the graph, where each edge represents a logical relationship between fields. It is a single logical relation edge. It is the main field of the relationship (the starting field for triggering rules). It is the object field of the relation (the target field that is constrained). It refers to relation types (divided into: conditional dependency, numerical constraint, and mutual exclusion relation). These are relational parameters (dynamically defined based on the relation type).

[0117] The relational parameters are defined as follows:

[0118]

[0119] in, The threshold is the conditional trigger threshold (e.g., "Foreign Investment" means the rule is triggered when the field value is "Foreign Investment"). min is the lower limit constraint for the value (e.g., 10,000,000 means the minimum value is 10 million). max is the upper limit constraint for the value (e.g., 50,000,000 means the maximum value is 50 million). This is a set of identifiers for mutually exclusive fields. The specific knowledge graph construction process includes:

[0120] Parse the hierarchical structure of the form template to extract the meaning of all fields (including field ID and field type, etc.) and constraint attributes (such as conditional dependency constraints, numerical range constraints, mutual exclusion constraints, etc.), and generate a JSON tree;

[0121] Traverse the parsed JSON tree to identify key constraint attributes, including but not limited to conditional dependency constraints, numerical range constraints, and mutual exclusion constraints, and generate a set of constraint attributes to provide input for subsequent classification.

[0122] Based on the semantics of the constraint attributes, the rules are divided into three types of logical relations: conditional dependencies with conditional trigger threshold parameters (such as C must be filled when A=B), numerical constraint fields with numerical thresholds, and mutually exclusive relation declarations of identifier sets with mutually exclusive fields, so as to clarify the semantic type and determine the graph edge generation rules.

[0123] Construct a triplet structure (subject field, relation type, and object field) based on the classification results, including: condition dependency generation ( "Conditional dependence" )+{trigger_value:X} indicates that when the main field When the object field equals a specific trigger value X (e.g., the enterprise type is "foreign-invested"), (For example, "foreign investment ratio" becomes a required field, and X is the specific value that triggers this dependency; numerical constraint generation ( "Numerical constraints", null)+{min:Y,max:Z}, represent constraints on the main field. (For example, the numerical range of "registered capital" is restricted, limited to a minimum value Y and a maximum value Z. null indicates that this constraint does not point to other fields; it is a numerical attribute constraint of the field itself. Mutually exclusive relationships are generated.) "Mutual exclusion relationship", null)+{excluded_fields:[F1,F2]}, means that when the main field is null, the main field is null. When filled in, fields listed in the mutually exclusive field list [F1, F2] cannot coexist with it. null indicates that this constraint does not point to other fields, but rather specifies a set of identifiers for one or more mutually exclusive fields (such as F1, F2) through the parameter excluded_fields. In this way, the constraint attributes in the JSON tree are transformed into edges of the knowledge graph to form a structured relationship network. Finally, a set of nodes is created to make each form field a unique node, and a set of edges is created so that each constraint corresponds to an edge, which are combined to form a complete knowledge graph structure, thereby generating a lightweight knowledge graph to support real-time logic verification.

[0124] After constructing the lightweight knowledge graph, further logical verification and judgment are performed on the JSON array output by the multimodal large model. The specific process is as follows:

[0125] Iterate through each field in the JSON array. For the current field, retrieve all association rules with that field as the main field from the lightweight knowledge graph: If the current field does not have any association rules, mark it as "compliant" and add it to the compliant field set; if the current field has association rules, process them according to the type of the rule.

[0126] Conditional Dependency: Check if the current field value is equal to the preset trigger value. If they are equal, further check if the dependent field specified by the rule exists in the JSON array. If the dependent field is missing, record the "missing dependent field" error message to the error set.

[0127] Numerical constraints: Check whether the field value of the current field conforms to the preset numerical range. If the field value is less than the minimum value or greater than the maximum value, record the "value out of limit" error message to the error set.

[0128] Mutual exclusion: Check whether the mutually exclusive fields specified by the rule exist in the JSON array at the same time. If mutually exclusive fields exist, record the "field mutual exclusion" error message to the error set.

[0129] Repeat the above rule validation process until all fields have been validated.

[0130] After rule verification is completed, the conflict resolution phase begins:

[0131] Strict mode: Checks if the error set is empty. If not empty, clears the compliant field set and outputs only the error set; otherwise, outputs the compliant field set directly. Alternatively, lenient mode: Filters out all error fields associated with errors from the JSON array, retaining compliant fields without associated errors to form a compliant field set, and outputs both the compliant field set and the error set. Conflict resolution is performed based on the error information in the error set, thus providing a more accurate and reliable data source for subsequent automatic form generation. This solves the problem of missing implicit logical dependencies between fields and reduces the form filling error rate.

[0132] The method proposed in this application uses dynamic prompt word engineering to load data defined by a preset expression structure in real time and generate role-based instructions. This enables the multimodal large model to adapt to form templates for different government affairs, dynamically binding image and voice inputs with the preset form structure to solve the problem of misaligned structured output. By using government entity relationship reasoning to model field causal dependencies, logical conflicts are avoided, and cross-modal information joint reasoning and automatic form filling are achieved. This significantly improves the accuracy and efficiency of government affairs declarations and enhances the dynamic adaptive capability of government affairs declarations.

[0133] Based on the same technical concept described above, this application also proposes a government affairs intelligent interaction and information extraction device, such as... Figure 2 As shown, the government affairs intelligent interaction and information extraction device 200 includes:

[0134] The input acquisition and preprocessing module 201 is configured to: uniformly convert user-uploaded non-image materials into a standardized format image input multimodal large model; process user voice input into a recognizable voice segment input multimodal large model; and dynamically obtain the form template corresponding to the current business matter handled by the user, loading all its fields, field types, constraints, and related rules into the multimodal large model in real time. The specific implementation process is as described in step 1 above and will not be repeated here.

[0135] The structured adaptation module 202 is configured to: dynamically inject form templates, user response text, and preset question word sequences into the prompt word instructions of the multimodal large model through dynamic prompt word engineering; control the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically append status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input image and / or speech. The specific implementation process is as described in step 2 above, and will not be repeated here.

[0136] Furthermore, the logic verification module 203 is configured to perform business logic verification on the JSON array output by the multimodal large model using knowledge graph technology. The specific business logic verification process is as described in step 3 above, and will not be repeated here.

[0137] Furthermore, the government affairs intelligent interaction and information extraction device 200 proposed in this application embodiment also includes:

[0138] The autofill module 204 is configured to automatically fill out forms based on a JSON array that has passed business logic validation.

[0139] Furthermore, the structured adaptation module 202 in this embodiment of the application is also configured as follows:

[0140] An ID semantic corrector is employed to perform three layers of semantic verification on the field IDs output by the multimodal large model to address potential field ID recognition errors or fictitious IDs generated by the large model. The first layer is exact matching, checking if the field ID output by the large model exists in the correct set of field IDs in the form template. If no match is found, the second layer, semantic arbitration, is initiated. The cosine similarity between the field description output by the large model and all field titles in the form template is calculated using the bge-large-zh model vectorization. When the highest similarity score exceeds a preset threshold (e.g., greater than 0.8), the corresponding correct field ID in the form template automatically replaces the incorrect field ID output by the large model. If the semantic similarity also fails to meet the threshold, the third layer, fictitious ID suppression, is initiated, marking the field ID value as "null" to prevent erroneous information from contaminating the final form data. The specific verification process is as described above and will not be repeated here.

[0141] Based on the same technical concept described above, this application also proposes a government affairs intelligent interaction and information extraction system, such as... Figure 3 As shown in the embodiments of this application, the government affairs intelligent interaction and information extraction system 300 includes:

[0142] The system comprises an input device 301, an output device 302, a processor A303, and a memory A304; wherein the number of processors A303 and memory A304 can be one or more. Figure 3 The following description uses a processor A303 and a memory A304 as an example. The input device 301, output device 302, processor A303, and memory A304 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0143] Specifically, by calling the operation instructions stored in memory A304, processor A303 executes the following steps:

[0144] Multimodal input acquisition and preprocessing: Non-image materials uploaded by users are uniformly converted into standardized format image input multimodal large model, user voice input is processed into recognizable voice segments input multimodal large model, and according to the business matter currently handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints and related rules contained therein are loaded into the multimodal large model in real time;

[0145] Multimodal large model output aligned with form structure: Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input images and / or speech.

[0146] Knowledge graph-driven logic validation: Utilize knowledge graph technology to validate the business logic of the JSON array output by a multimodal large model.

[0147] Optionally, by calling the operation instructions stored in memory A304, processor A303 is also used to execute any of the corresponding implementation methods in the above-described government affairs intelligent interaction and information extraction method.

[0148] Based on the same technical concept described above, this application also proposes an electronic device, such as... Figure 4 As shown, the electronic device 400 includes: a memory B410, a processor B420, and a computer program A411 stored in the memory B410 and executable on the processor B420. When the processor B420 executes the computer program A411, it performs the following steps:

[0149] Multimodal input acquisition and preprocessing: Non-image materials uploaded by users are uniformly converted into standardized format image input multimodal large model, user voice input is processed into recognizable voice segments input multimodal large model, and according to the business matter currently handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints and related rules contained therein are loaded into the multimodal large model in real time;

[0150] Multimodal large model output aligned with form structure: Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input images and / or speech.

[0151] Knowledge graph-driven logic validation: Utilize knowledge graph technology to validate the business logic of the JSON array output by a multimodal large model.

[0152] Optionally, when processor B420 executes computer program A411, it can implement any of the corresponding embodiments in the above-described government affairs intelligent interaction and information extraction method.

[0153] It should be noted that the electronic device proposed in this application embodiment is a device used to implement the above-mentioned government affairs intelligent interaction and information extraction method. Therefore, based on the above-mentioned government affairs intelligent interaction and information extraction method proposed in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this application embodiment. Therefore, how the electronic device specifically implements the above-mentioned government affairs intelligent interaction and information extraction method will not be described in detail here. Any electronic device used by those skilled in the art to implement the above-mentioned government affairs intelligent interaction and information extraction method falls within the scope of protection of this application.

[0154] Based on the same technical concept described above, embodiments of this application also propose a computer-readable storage medium, such as... Figure 5 As shown, the computer-readable storage medium 500 stores a computer program B511, which, when executed by a processor, performs the following steps:

[0155] Multimodal input acquisition and preprocessing: Non-image materials uploaded by users are uniformly converted into standardized format image input multimodal large model, user voice input is processed into recognizable voice segments input multimodal large model, and according to the business matter currently handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints and related rules contained therein are loaded into the multimodal large model in real time;

[0156] Multimodal large model output aligned with form structure: Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input images and / or speech.

[0157] Knowledge graph-driven logic validation: Utilize knowledge graph technology to validate the business logic of the JSON array output by a multimodal large model.

[0158] Optionally, when the computer program B511 is executed by the processor, it can implement any of the embodiments corresponding to the above-described government affairs intelligent interaction and information extraction method.

[0159] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0160] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for intelligent government interaction and information extraction, characterized in that, include: Non-image materials uploaded by users are uniformly converted into standardized image inputs into the multimodal large model. User voice input is processed into recognizable voice segments and input into the multimodal large model. Based on the business matter currently being handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints, and related rules contained in the form template are loaded into the multimodal large model in real time. Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to generate only JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field. The user response text is the original user response content obtained by recognizing the input image and / or speech. The business logic of the JSON array output by the multimodal large model is validated using knowledge graph technology.

2. The method for intelligent government interaction and information extraction according to claim 1, characterized in that, Also includes: Automatic form filling is performed based on a JSON array that has passed business logic validation.

3. A method for intelligent government interaction and information extraction according to claim 1 or 2, characterized in that, The process of converting the user's voice input into recognizable speech segments specifically includes: A dual-threshold voice activity detection mechanism is used to process user voice input, including: The short-time energy and zero-crossing rate of the speech signal are calculated in real time. When the short-term energy exceeds a preset high threshold and reaches a preset duration, a valid voice segment is determined to have started. For signal segments whose short-term energy is below a preset high threshold but whose zero-crossing rate is consistently above its preset threshold, they are identified as silent speech segments and merged with adjacent speech segments; When the short-term energy is continuously lower than a preset low threshold and the zero-crossing rate is also continuously lower than its preset threshold, and the duration meets the preset silence duration threshold, then the current valid voice segment is determined to have ended.

4. A method for intelligent government interaction and information extraction according to claim 1 or 2, characterized in that, The dynamic prompt word engineering involves constructing a dynamic, structured prompt word instruction generation model. This model deeply integrates the form template of the current business transaction, predefined question word sequences, and user response text to generate prompt word instructions that constrain the output of the multimodal large model, aligning the multimodal large model output with the form structure. The specific process includes: The input form template is parsed, and field attributes are extracted to generate a field mapping table, which contains four columns of data: question word sequence, form field name, field type, and conversion rule; The input user response text is segmented into a sequence of semantic units that match the question wording sequence; Based on the field types and conversion rules defined in the field mapping table, the value of each semantic unit is standardized and converted. The matched and transformed field data are forcibly encapsulated into standardized JSON arrays according to the output specifications; each JSON array contains only: field ID, field name, and field value; The JSON array is double-validated using regular expressions and a syntax validator to ensure that the format is valid and the fields are effective, while filtering out any illegal or undefined fields.

5. The method for intelligent government interaction and information extraction according to claim 4, characterized in that, The process of achieving multimodal large model output and form structure alignment based on the aforementioned dynamic prompt word engineering also includes: An ID semantic corrector is used to perform three layers of semantic verification on each field ID output by the multimodal large model. The first layer is exact matching: checking whether the field ID output by the multimodal large model exists in the set of correct field IDs in the form template. If no match is found, the second layer of semantic arbitration is performed: using model vectorization, the cosine similarity between the field name output by the multimodal large model and all field titles in the form template is calculated. When the highest similarity exceeds a preset threshold, the correct field ID in the form template is automatically replaced with the incorrect field ID output by the multimodal large model. If the highest similarity does not reach the preset threshold, the third layer of fictitious suppression is performed: the value of the field ID is marked as "null".

6. The method for intelligent government interaction and information extraction according to claim 4, characterized in that, The aforementioned use of knowledge graph technology to perform business logic validation on the JSON array output by the multimodal large model includes: Based on the form template, a lightweight knowledge graph is constructed; Iterate through each field output by the multimodal large model. For the current field, retrieve all association rules with the current field as the main field from the lightweight knowledge graph. If the current field does not have any association rules, mark it as "compliant" and add it to the compliant field set. If the current field has association rules, perform split processing according to the type of the rule: If the rule type is conditional dependency, check whether the field value of the current field is equal to the preset trigger value. If so, further check whether the dependency field specified by the rule exists in the JSON array. If the dependency field is missing, record the "missing dependency field" error message to the error set. If the rule type is a numerical constraint, check whether the field value of the current field conforms to the preset numerical range. If it does not conform, record the "value exceeds the limit" error message to the error set. If the rule type is mutually exclusive, check whether the mutually exclusive fields specified by the rule exist in the JSON array at the same time. If mutually exclusive fields exist, record the "field mutual exclusion" error message to the error set. Repeat the above rule validation process until all fields have been validated; Once the rule verification is complete, the conflict resolution phase begins.

7. The method for intelligent government interaction and information extraction according to claim 6, characterized in that, The construction of a lightweight knowledge graph based on the form template includes: Define a knowledge graph structure; wherein the knowledge graph structure consists of nodes and edges, each node represents a form field, each edge represents a logical relationship between fields, and the logical relationship consists of its subject field, object field, relationship type, and relationship parameters; The hierarchical structure of the form template is parsed to extract the meaning of all fields and constraint attributes, and a JSON tree is generated; Traverse the parsed JSON tree to identify constraint attributes; Based on the semantics of the constraint attributes, the rules are divided into three types of logical relations; Construct a triplet structure according to the relation type to which the logical relation belongs. The triplet structure includes a subject field, a relation type, and an object field. Transform all constraint attributes in the JSON tree into edges of the knowledge graph structure to form a structured relation network. Finally, create a set of nodes to make each form field a unique node and create a set of edges so that each constraint corresponds to one edge, thereby forming a complete knowledge graph structure.

8. A government affairs intelligent interaction and information extraction device, characterized in that, include: The input acquisition and preprocessing module is configured to: uniformly convert non-image materials uploaded by users into standardized format image input multimodal large model; process user voice input into recognizable voice segments and input them into the multimodal large model; and dynamically obtain the form template corresponding to the business matter currently being handled by the user, and load all fields, field types, constraints and related rules contained in the form template into the multimodal large model in real time. The structured adaptation module is configured to: dynamically inject form templates, user response text, and preset question word sequences into the prompt word instructions of the multimodal large model through dynamic prompt word engineering; control the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values; and automatically append status flags to each field; wherein the user response text is the original user response content obtained by recognizing the input image and / or speech; In addition, the logic verification module is configured to perform business logic verification on the JSON array output by the multimodal large model using knowledge graph technology.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the government intelligent interaction and information extraction method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the government intelligent interaction and information extraction method as described in any one of claims 1-7.

Images