A government affair intelligent interaction and information extraction method, device, equipment and medium

Abstract

The application discloses a government affair intelligent interaction and information extraction method and device, equipment and medium, relates to the technical field of digital government affairs, digital city, intelligent interaction and the like, and through dynamic prompt word engineering technology, the application realizes the binding of cross-modal input (image / speech) and structured form field, converts traditional manual item-by-item filling into end-to-end automatic processing, significantly reduces user operation steps, solves the burden of filling hundreds of forms, and improves filling efficiency; further, an ID voice corrector is adopted to realize three-layer progressive verification, effectively suppresses the ID offset of a multi-modal large model output, ensures accurate mapping of field identification, simultaneously adopts a lightweight knowledge graph to automatically capture logical rules between fields, real-time intercepts compliance conflicts specific to government affair scenes, reduces the risk of logical errors, and improves filling accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital government affairs, digital city, intelligent interaction, and specifically relates to a government affair intelligent interaction and information extraction method, device, equipment and medium. BACKGROUND

[0002] The current online declaration of government service (such as social security claim and newborn birth event) highly depends on user self-help form filling, and the following bottlenecks exist: (1) Low filling efficiency: the form often contains dozens to hundreds of fields, and the user needs to fill in each field manually, which is time-consuming and laborious. The field semantics is strong in professional nature (such as "unified social credit code" and "operating range specification expression"), and non-professionals are prone to errors, leading to repeated modification. (2) Insufficient material information extraction capability: the mainstream adopts traditional OCR technology to identify certificates and proofs, and can only output original text, and cannot understand the context semantics. (3) The logical association between fields is not utilized, resulting in information fragmentation. (4) Weak interaction experience: the current mainstream interaction form still relies on user manual input and form filling, and the language recognition (ASR) output text stream is not bound to the form field, and the "voice instruction → automatic form filling" cannot be realized.

[0003] The existing technology has the following limitations:

[0004] (1) Insufficient single modal processing capability: the OCR technology only focuses on image to text conversion, and does not integrate semantic understanding; the speech recognition (ASR) output text stream is not bound to the form field; the NLP (natural language search) engine adopts traditional rule model and relies on preset keywords, and has poor generalization ability (such as "address" can be expressed as "residence" and "permanent residence" in different scenarios).

[0005] (2) Multi-modal coordination mechanism is not realized: the existing solutions are mostly in series flow line (such as OCR → NLP → database matching), and the errors are accumulated step by step (OCR recognition error directly leads to field matching failure); cross-modal information is not uniformly modeled: there is lack of joint reasoning between material pictures, voice instructions and form fields.

[0006] (3) Poor adaptability to government affairs scenarios: the hard-coded rules bind fixed form templates (such as A event fields cannot be reused for B event), and cannot dynamically adapt to the differences of government affairs in different places; the government knowledge base is not embedded, leading to field analysis and filling errors.

[0007] Currently, some government systems attempt to introduce technical upgrades, but the results are limited. For example, intelligent form pre-filling: this technology automatically fills in some fields (such as name, ID number, etc.) based on user historical data, but cannot handle new materials (such as uploading a property certificate for the first time); RPA process automation: this technology simulates manual operations to copy material content to forms, relies on fixed position grabbing, and is invalid if the material format changes; using a multi-modal model: this technology directly calls the GPT-4V API to analyze materials, but the extraction of government fields needs to strictly match the pre-set JSON structure (such as "registered capital" needs to be converted to a numerical type), and the output format of the general model is uncontrollable.

[0008] In summary, current technologies cannot meet the needs of government services in terms of efficiency (automated form filling), accuracy (cross-modal field extraction), and experience (natural language interaction), and there is an urgent need for an integrated solution that combines multi-modal understanding, dynamic form adaptation, and government knowledge enhancement. SUMMARY

[0009] To solve the problems of low efficiency, high error rate, and other issues in existing government services due to insufficient multi-modal information coordination, missing logical associations between fields, and inability to accurately map unstructured data to forms, the present application proposes a government intelligent interaction and information extraction method, device, equipment, and medium, which improves the efficiency, accuracy, and dynamic adaptive ability of form filling.

[0010] In a first aspect, the present application is implemented by the following technical solution:

[0011] A government intelligent interaction and information extraction method, comprising:

[0012] Converting the non-image materials uploaded by the user into a standardized format image inputting a multi-modal large model, processing the user's voice input into recognizable voice segments inputting the multi-modal large model, and dynamically obtaining the form template corresponding to the business transaction currently handled by the user, and loading all fields, field types, constraint conditions and related rules contained in the form template to the multi-modal large model in real time;

[0013] By dynamic prompt word engineering, the form template, user answer text and pre-set question rhetoric sequence are dynamically injected into the prompt word instruction of the multi-modal large model, the multi-modal large model is controlled to generate only a JSON array containing field ID, field name and corresponding field value, and a state marker is automatically added for each field; wherein the user answer text is the original answer content of the user obtained by recognizing the input image and / or voice;

[0014] Using knowledge graph technology to perform business logic verification on the JSON array output by the multi-modal large model.

[0015] In some embodiments, further comprising:

[0016] Automatic filling of forms according to JSON arrays verified by business logic.

[0017] In some embodiments, the voice input of the user is processed into recognizable voice segments, specifically:

[0018] The voice input of the user is processed using a double-threshold voice activity detection mechanism, including:

[0019] The short-time energy and zero-crossing rate of the voice signal are calculated in real time;

[0020] When the short-time energy exceeds the preset high threshold for a preset duration, it is determined that the valid voice segment starts;

[0021] For a signal segment where the short-time energy is lower than the preset high threshold but the zero-crossing rate is continuously higher than its preset threshold, it is identified as a soft sound segment and merged with adjacent voice segments;

[0022] When the short-time energy continuously falls below the preset low threshold and the zero-crossing rate also continuously falls below its preset threshold, and the duration meets the preset threshold of the silent duration, it is determined that the current valid voice segment ends.

[0023] In some embodiments, the dynamic prompt word engineering generates a dynamic and structured prompt word instruction generation model, deeply integrates the form template of the current business matter, the pre-defined question script sequence, and the user answer text, generates a prompt word instruction that constrains the output of the multi-modal large model, and aligns the output of the multi-modal large model with the form structure, including:

[0024] Parse the input form template to extract field attributes and generate a field mapping table, where the field mapping table contains four columns of data: question script sequence, form field name, field type, and conversion rule;

[0025] Split the input user answer text into a sequence of semantic units that match the question script sequence;

[0026] According to the field type and conversion rule defined in the field mapping table, standardize and convert the value of each semantic unit;

[0027] Match and convert the field data according to the output specification to force encapsulation into a standardized JSON array; each JSON array only contains: field ID, field name, and field value;

[0028] Apply regular expressions and syntax validators to double-check the JSON array to ensure that the format is legal and the fields are valid, filtering out any illegal or undefined fields.

[0029] In some embodiments, the process of achieving multi-modal large model output and form structure alignment based on the dynamic prompt word engineering further comprises:

[0030] Performing three-layer semantic verification on each field ID of the multi-modal large model output using an ID semantic corrector; wherein the first layer is exact matching: checking whether the field ID of the multi-modal large model output exists in the correct field ID set of the form template; if no successful matching is achieved, then entering the second layer semantic arbitration: using model vectorization to calculate the cosine similarity of the field name of the multi-modal large model output and all field titles in the form template, when the highest similarity exceeds the preset threshold, automatically replacing the correct field ID of the form template with the error field ID of the multi-modal large model output, if the highest similarity does not reach the preset threshold, then performing the third layer of fictional suppression: marking the value of the field ID as "null".

[0031] In some embodiments, the business logic verification of the JSON array output by the multi-modal large model using knowledge graph technology comprises:

[0032] According to the form template, a lightweight knowledge graph is constructed;

[0033] Traverse each field of the multi-modal large model output, for the current field, retrieve all associated rules with the current field as the main field from the lightweight knowledge graph, if the current field does not exist any associated rule, then directly mark it as "compliant" and add it to the compliant field set, if the current field exists associated rules, then according to the type of the rules, perform shunt processing:

[0034] If the type of the rule is conditional dependence, then check whether the field value of the current field is equal to the preset trigger value, if yes, then further check whether the dependent field specified by the rule exists in the JSON array, if the dependent field is missing, then record the "missing dependent field" error information to the error set;

[0035] If the type of the rule is numerical constraint, then check whether the field value of the current field conforms to the preset numerical range, if not, then record the "value out of range" error information to the error set;

[0036] If the type of the rule is mutual exclusion relationship, then check whether the mutually exclusive fields specified by the rule exist in the JSON array at the same time, if the mutually exclusive field coexistence situation exists, then record the "field mutual exclusion" error information to the error set;

[0037] Repeat the above rule verification process until the verification of all fields is completed;

[0038] After the rule verification, the conflict resolution phase is entered.

[0039] In some embodiments, constructing the lightweight knowledge graph according to the form template comprises:

[0040] defining a knowledge graph structure; wherein the knowledge graph structure is composed of nodes and edges, each node represents a form field, and each edge represents a logical relationship between fields, the logical relationship is composed of a subject field, an object field, a relationship type, and a relationship parameter;

[0041] parsing the hierarchical structure of the form template to extract all field meanings and constraint attributes, and generating a JSON tree;

[0042] traversing the parsed JSON tree to identify constraint attributes;

[0043] dividing the rules into three types of logical relationships according to the semantics of the constraint attributes;

[0044] constructing the corresponding triple structure according to the relationship type of the logical relationship, the triple structure includes the subject field, the relationship type, and the object field, converting all constraint attributes in the JSON tree into edges of the knowledge graph structure to form a structured relationship network, and finally creating a node set to realize a unique node for each form field, and creating an edge set to make each constraint correspond to an edge, thereby forming a complete knowledge graph structure.

[0045] In a second aspect, the application provides a government affair intelligent interaction and information extraction device, comprising:

[0046] An input collection and preprocessing module is configured to uniformly convert non-image materials uploaded by a user into a standardized format of image, input a multi-modal large model, process voice input of the user into recognizable voice segments, and input the multi-modal large model, and dynamically obtain a form template corresponding to a business matter currently handled by the user, and load all fields, field types, constraint conditions, and related rules contained in the form template to the multi-modal large model in real time;

[0047] A structured adaptation module is configured to dynamically inject a form template, user answer text, and a pre-set question rhetoric sequence into a prompt word instruction of the multi-modal large model through a dynamic prompt word engineering, control the multi-modal large model to generate only a JSON array containing a field ID, a field name, and a corresponding field value, and automatically append a state mark to each field; wherein the user answer text is original answer content of the user obtained by recognizing input images and / or voice;

[0048] And a logic verification module is configured to perform business logic verification on the JSON array output by the multi-modal large model by using a knowledge graph technology.

[0049] In a third aspect, the present application provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor implementing any of the above-mentioned government affair intelligent interaction and information extraction methods when executing the computer program.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement any of the above-mentioned government affair intelligent interaction and information extraction methods.

[0051] The present application provides a government affair intelligent interaction and information extraction method, which realizes the binding of cross-modal input (image / speech) and structured form fields through dynamic prompt word engineering technology, converts traditional manual item-by-item filling into end-to-end automatic processing, significantly reduces user operation steps, solves the burden of filling hundreds of forms, and improves filling efficiency; also uses an ID voice corrector to realize three-layer progressive verification (accurate matching→ semantic arbitration→ fiction suppression), effectively suppresses the ID offset of the multi-modal large model output, ensures accurate mapping of field identification, and at the same time uses a lightweight knowledge graph to automatically capture the logical rules (conditional dependence / numerical constraint / exclusion relationship) between fields, real-time intercepts compliance conflicts specific to government affairs scenarios (such as married but not filled in spouse information), reduces the risk of logical errors, and improves the accuracy of filling. In addition, the method can also realize adaptive adaptation of different government affairs forms through dynamic prompt word engineering technology, and the lightweight knowledge graph is updated in real time according to the input form template data, improving the dynamic adaptive ability; it also supports joint semantic reasoning of multi-modal input (voice, image) and form fields, realizes semantic cascading, and improves the accuracy of understanding complex user intentions.

[0052] Correspondingly, the government affair intelligent interaction and information extraction device, electronic device and computer-readable storage medium provided by the present application have the same technical effects as described above. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings described herein are used to provide further understanding of the embodiments of the present application, and form a part of the present application, and do not constitute a limitation to the embodiments of the present application. In the drawings:

[0054] Figure 1 The government affair intelligent interaction and information extraction method flowchart provided by the embodiments of the present application;

[0055] Figure 2 The government affair intelligent interaction and information extraction device principle block diagram provided by the embodiments of the present application;

[0056] Figure 3 A government affair intelligent interaction and information extraction system architecture diagram is provided for the embodiments of the present application.

[0057] Figure 4 An electronic device diagram is provided for the embodiments of the present application.

[0058] Figure 5 A computer readable storage medium diagram is provided for the embodiments of the present application.

[0059] Reference signs and corresponding component names:

[0060] 200 - government affair intelligent interaction and information extraction device, 201 - input collection and preprocessing module, 202 - structured adaptation module, 203 - logic verification module, 204 - automatic form filling module, 300 - government affair intelligent interaction and information extraction system, 301 - input device, 302 - output device, 303 - processor A, 304 - memory A, 400 - electronic device, 410 - memory B, 420 - processor B, 411 - computer program A, 500 - computer readable storage medium, 511 - computer program B. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description of the present application will be given below in combination with embodiments and drawings, the illustrative embodiments of the present application and their descriptions are only used to explain the present application, and do not limit the present application.

[0062] The embodiments of the present application provide a government affair intelligent interaction and information extraction method, as shown in the following figure: Figure 1 The government affair intelligent interaction and information extraction method provided by the embodiments of the present application includes the following steps:

[0063] Step 1, multi-modal input collection and preprocessing: converting the non-image materials uploaded by the user into standardized format image input multi-modal large model, processing the user's voice input into recognizable voice segment input multi-modal large model, and dynamically obtaining the form template corresponding to the business matter according to the current business matter of the user, and loading all fields, field types, constraint conditions and related rules contained in the multi-modal large model in real time;

[0064] Step 2, aligning the multi-modal large model output with the form structure: by dynamic prompt word engineering, the form template, user answer text, and preset question rhetoric sequence are dynamically injected into the prompt word instruction of the multi-modal large model, controlling the multi-modal large model to only generate a JSON array containing field ID, field name, and corresponding field value, and automatically appending a state marker to each field; wherein the user answer question and answer is the user's original answer content obtained by recognizing the input image and / or voice;

[0065] Step 3, knowledge graph driven logic verification: using knowledge graph technology to verify the business logic of the JSON array output by the multi-modal large model.

[0066] Further, the government affairs intelligent interaction and information extraction method proposed in the embodiments of the present application further comprises:

[0067] Step 4, automatic form filling: automatically filling the form according to the JSON array verified by the business logic.

[0068] Further, in step 1 of the embodiments of the present application, the multi-modal input collection and preprocessing process comprises:

[0069] Raw material input processing: in the material uploading stage, the non-image format materials such as PDF or WORD documents in the various raw materials uploaded by the user are uniformly processed into standardized image formats (JPG or PNG), and then input into the multi-modal large model for image recognition.

[0070] Voice input processing: during the interaction stage of the user filling out the form, the user's voice instructions are captured in real time through a microphone array. To deal with complex environmental noise interference and accurately extract effective voice instructions, the present application adopts a double-threshold voice activity detection mechanism based on short-time energy and zero-crossing rate to process, effectively filter background noise, and accurately segment continuous speech into pure and recognizable speech segments. The double-threshold voice activity detection mechanism specifically includes: real-time calculation of the short-time energy and zero-crossing rate of the voice signal; when the short-time energy continuously exceeds the preset high threshold for a preset time length (for example, 300 milliseconds), it is determined that the effective speech segment starts. At the same time, for the signal segment whose short-time energy is lower than the high threshold but whose zero-crossing rate continuously exceeds its preset threshold, it is identified as a pure tone segment and merged with the adjacent speech segment. When the short-time energy continuously falls below the preset low threshold and the zero-crossing rate also continuously falls below its preset threshold, and the duration meets the preset silence duration threshold, it is determined that the current effective speech segment ends. This mechanism can effectively filter out background noise and short-term interference, ensuring that only pure and coherent user effective speech audio segments are reliably transmitted to the backend multi-modal large model for semantic understanding and instruction analysis.

[0071] Form data loading: Regardless of material identification or voice interaction stage, multi-modal large models need to accurately understand user intent and generate correct responses, and need to dynamically obtain the specific form structure and requirements corresponding to the current processing matter. Therefore, while performing this link, a query request is initiated in real time according to the current business matter (case ID) to dynamically obtain and load the detailed form definition structure corresponding to the business matter from the pre-defined form configuration center. This structure strictly follows the JSON Schema specification and clearly defines all fields, field types, filling constraints, association rules, and business rules of the form. The loaded form definition structure is the basis for constructing accurate dynamic prompt words, which enables the multi-modal large model to understand the context, predict the information that the user needs to fill in, and provide real-time and effective guidance, thereby driving the entire intelligent form filling process.

[0072] Further, in step 2 of the embodiments of the present application, the dynamic prompt word engineering constructs different prompt words through different form template data, so that the multi-modal large model output is aligned with the form structure. Among them, the dynamic prompt word engineering constructs a dynamic and structured prompt word instruction generation model, deeply integrates the form template of the current business matter, the pre-defined user guidance question sequence, and the actual input of the user (voice to text or image recognition result), generates prompt word instructions that accurately constrain the multi-modal large model output, to solve the key problems of uncontrolled multi-modal large model output format, semantic drift, and difficulty in accurately binding target form fields, and ensure that the multi-modal large model output strictly follows the pre-set form field ID, name, and value triple structure. Specifically, the dynamic prompt word engineering mainly includes:

[0073] Template engine, which is used to store basic instruction template framework, define the core role, capability requirement, processing flow specification and output specification of the model. Specifically, it includes: role definition, which clearly defines the model as a government form filling expert; core capability definition, which emphasizes that the model needs to have the ability to handle three elements, including accurately understanding the field intent of the pre-defined question sequence, accurately parsing the semantics of the user's answer, and accurately matching the parsed results with the target form field definition, and clearly defining the processing rules for different field types (such as text input, single selection, multiple selection, date, etc.) (such as directly extracting semantic value, mapping dictionary value, converting date format, numerical unit conversion, etc.); processing flow specification, which specifies that the model must follow the standard process of "analysis → conversion → matching": identify key entities in user answers; perform data standardization conversion according to field type; associate form field ID to generate structured output; output specification, which strictly limits the model to output only standard JSON object arrays containing field ID (or name), value, and forces to append status markers (matching success is "1", completely no matching is "0"). Clearly prohibit output of undefined fields or missing required fields in JSON Schema.

[0074] Dynamic input, responsible for injecting real-time dynamic business context information into the basic instruction template to form the final, specific business-oriented large model prompt. Specifically, the input information includes: question sequence, i.e. pre-defined text that guides the user to answer a specific form field (e.g. "Please enter the registered address of the enterprise"); user answer text, actual user answer content from speech recognition (ASR) or image recognition (OCR + multi-modal understanding); form data, i.e. the detailed form template corresponding to the current business matter being handled, containing information such as all field IDs, titles, data types, constraint rules, option dictionaries, etc.

[0075] In addition, the constraint executor is used to perform strict post-processing rules after model inference, forcing the model output to comply with pre-defined format and content specifications. The enforcement specification is to ensure that the final output only contains a JSON array object of field ID, field name, and field value, and when at least one valid field value is successfully matched, a status marker {"state": "1"} is appended to each corresponding output object. When the model determines that the user's answer has no valid association with any form field, it returns [{"state": "0"}], effectively preventing null or invalid data from polluting the final form.

[0076] The specific process of aligning the multi-modal large model output with the form structure is as follows:

[0077] Parse the input form template and extract key field attributes to generate a field mapping table, which contains four columns of data: question sequence: original question description text; form field name: field name; field type: character / option / date, etc.; conversion rule: field value standardization method.

[0078] For each field, a mapping relationship function is established:

[0079]

[0080] where, represents the i-th field in the question sequence, represents the j-th field in the form, represents the mapping relationship, represents the form field display name, represents the semantic similarity calculation function, which is generated by the model itself to determine, represents the similarity threshold, which is preferably 0.85 in the embodiments of the present application.

[0081] Split the user answer text into a sequence of semantic units matching the question script sequence, where the splitting algorithm is based on the following principles: cut according to the order of the question script sequence, keep the complete semantic unit, and identify and process complex expressions (such as "2024-12-5 14:30").

[0082] According to the field type and conversion rule defined in the field mapping table, the value of each semantic unit is standardized and converted. It includes option field processing: for single or multiple choice fields, the user's answer text (such as "female") is mapped to the predefined option dictionary value (such as "2"); numerical field processing: for numerical fields, extract the numbers in the answer and process the units (such as converting "50,000" to 50000), support unit conversion rules specific to government scenarios (such as "10,000 yuan" to the basic unit); date field processing: various natural language or non-standard format date descriptions (such as "next New Year's Day", "2025 / 1 / 1") are uniformly converted to standard ISO format (such as "2025-01-01"); text field processing: clean and preserve the semantic integrity of the necessary text input fields.

[0083] The matched and converted field data is forced to be packaged into a standardized JSON array according to the output specification, and each JSON array strictly contains: field ID (unique identifier of the form field), field name (display name of the form field), field value (field value after standardized conversion).

[0084] And apply regular expressions and JSON Schema syntax validators to double-check the final output JSON array, ensuring that the format is legal and the fields are valid, filtering out any illegal or undefined fields.

[0085] Further, when filling out the form, the field ID returned by the JSON array needs to be used as an identifier to fill in the corresponding content. However, due to the complexity of the 16-character ID generated by the large model, the field ID of the output JSON array may be biased, resulting in the field being unable to be correctly filled into the form. Therefore, step 2 of the embodiments of the present application further includes:

[0086] The ID semantic corrector is used to perform three-layer semantic verification on the field IDs output by the multi-modal large model to address possible identification errors or fabrications of the field IDs by the multi-modal large model. The first layer is precise matching, which checks whether the field ID output by the multi-modal large model exists in the correct field ID set of the form template. If no match is found, the second layer of semantic arbitration is entered, which uses the bge-large-zh model to vectorize and calculate the cosine similarity between the field name output by the multi-modal large model and all field titles in the form template. When the highest similarity exceeds a preset threshold (for example, greater than 0.8), the corresponding correct field ID in the form template is automatically replaced with the error field ID output by the multi-modal large model. If the highest similarity does not reach the preset threshold, the third layer of fabrication suppression is entered, which marks the value of the field ID as "null" to prevent the pollution of the final form data by error information. The specific process of the three-layer semantic verification is as follows:

[0087] ID matching verification: The field IDs in the standard JSON array output by the multi-modal large model are matched and verified with the standard field IDs in the form template. Among them, the JSON array is represented as:

[0088]

[0089] Among them, is the field ID (may contain the bias generated by the large model), is the field name, is the corresponding field value, is the number of fields, is the i-th field.

[0090] The standard field ID set of the form template is represented as:

[0091]

[0092] Among them, is the form standard field ID, is the field title, is the j-th form standard field, is the number of form standard fields.

[0093] Iterate through each field object in the multi-modal large model output JSON array to perform existence verification:

[0094]

[0095] Among them, is the id value in the field object in the output JSON array, is the set of all standard field IDs in the form template, For matching verification, true indicates that the verification is passed, and false indicates that the verification is failed.

[0096] The fields passed the verification directly enter the output queue, and the original data structure is reserved.

[0097]

[0098] wherein, is a JSON array composed of field IDs passed the verification.

[0099] Semantic similarity arbitration: for the IDs that are not accurately matched, semantic mapping across modalities is realized through field name vectorization, specifically: for the fields that do not pass the verification, name vectorization comparison is started, first, the field names output by the multi-modal large model and the standard field titles in the form template are vectorized, respectively represented as:

[0100]

[0101]

[0102] wherein, is the vector encoding of the bge-large-zh model, is the semantic vector of the field name output by the multi-modal large model, is the semantic vector of the standard field title;

[0103] Then, the cosine similarity of the semantic vector of each output field name and the semantic vector of the standard field title is calculated:

[0104]

[0105] wherein, is the cosine similarity score, and the closer to 1 indicates the higher semantic similarity.

[0106] The maximum similarity of the semantic vector of each output field name and the semantic vector of the standard field title is obtained, and if it is greater than the threshold value, the field ID is replaced with the standard field ID in the form template, that is:

[0107]

[0108] wherein, is the index of the best matching standard field, is a function for obtaining the maximum value, is a similarity qualified threshold, and the application embodiment preferably adopts 0.8.

[0109] Fabrication suppression and ID repair: fields in the JSON array output by the multi-modal large model that do not match the standard field ID of the form template and are not arbitrated by semantic similarity are judged to be large model fabrication, and the field setting in the JSON array is set to null.

[0110] Through the above three-layer semantic verification, a more accurate and reliable standard JSON array is finally output.

[0111] Further, in step 3 of the embodiments of the present application, the process of knowledge graph driven logical verification includes:

[0112] Based on the constraint conditions and related rules of the form template, a lightweight knowledge graph is constructed, and the constructed knowledge graph is defined as follows:

[0113]

[0114]

[0115]

[0116] Among them, is the complete knowledge graph structure, is the node set in the graph, each node represents a form field, is a single node, is the unique identifier of the field in the form template, is the field type, is the edge set in the graph, each edge represents the logical relationship between fields, is a single logical relationship edge, is the subject field of the relationship (the starting field of the trigger rule), is the object field of the relationship (the target field of the constraint), is the relationship type (divided into: conditional dependence, numerical constraint, and mutual exclusion relationship), is the relationship parameter (defined dynamically according to the relationship type).

[0117] The relationship parameter is defined as follows:

[0118]

[0119] Among them, is the conditional trigger threshold (for example: "foreign capital" means that the rule is triggered when the field value is foreign capital), min is the numerical lower limit constraint (for example: 10000000 means the minimum value is 1000 million), and max is the numerical upper limit constraint (for example: 50000000 means the maximum value is 5000 million), is the identifier set of mutually exclusive fields. The specific knowledge graph construction process includes:

[0120] Parsing the hierarchical structure of the form template to extract all field meanings (including field ID and field type, etc.), and constraint properties (such as conditional dependency constraints, numerical range constraints, mutual exclusion relationship constraints, etc.), generating a JSON tree;

[0121] Traverse the parsed JSON tree to identify key constraint properties, including but not limited to conditional dependency constraints, numerical range constraints, and mutual exclusion relationship constraints, to generate a set of constraint properties for subsequent classification;

[0122] According to the semantics of the constraint properties, the rules are divided into three types of logical relationships: conditional dependencies with conditional trigger threshold parameters (such as when A = B, C is required), numerical constraint fields with numerical threshold values, and mutual exclusion relationship declarations with identifier sets of mutually exclusive fields, to clearly define semantic types and determine graph edge generation rules;

[0123] Construct a triple structure (subject field, relationship type, and object field) according to the classification results, including: conditional dependency generation ( , "conditional dependency", )+{trigger_value:X}, indicating that when the subject field is equal to a specific trigger value X (for example, the enterprise type is "foreign capital"), the object field (for example, "foreign investment ratio") becomes a required field, and X is the specific value that triggers this dependency relationship; numerical constraint generation ( , "numerical constraint", null)+{min:Y,max:Z}, indicating that the numerical range of the subject field (for example, "registered capital") is limited to between the minimum value Y and the maximum value Z, and null indicates that this constraint does not point to other fields, but is a numerical attribute constraint of the field itself; mutual exclusion relationship generation ( , "mutual exclusion relationship", null)+{excluded_fields:[F1,F2]}, indicating that when the subject field is filled in, the fields listed in the mutual exclusion field list [F1, F2] cannot coexist with it, and null indicates that this constraint does not point to other fields, but specifies an identifier set of one or more mutually exclusive fields (such as F1, F2) through the parameter excluded_fields; In this way, the constraint properties in the JSON tree are converted into the edge formation structure of the knowledge graph to form a structured relationship network; finally, a set of nodes is created to realize unique nodes for each form field, and a set of edges is created to make each constraint correspond to an edge, combining into a complete knowledge graph structure, thereby generating a lightweight knowledge graph to support real-time logical verification.

[0124] After constructing the lightweight knowledge graph, further logical verification and judgment are performed on the JSON array output by the multi-modal large model, and the specific process is as follows:

[0125] Each field in the JSON array is traversed, and all associated rules with the current field as the subject field are retrieved from the lightweight knowledge graph: if the current field does not have any associated rules, it is directly marked as “compliant” and added to the compliant field set; if the current field has associated rules, it is processed according to the type of the rules:

[0126] Conditional dependency: check if the field value of the current field is equal to the preset trigger value, if equal, further check if the dependent field specified by the rule exists in the JSON array, if the dependent field is missing, record the “missing dependent field” error information to the error set;

[0127] Numerical constraint: check if the field value of the current field conforms to the preset numerical range, if the field value is less than the minimum value or greater than the maximum value, record the “value out of range” error information to the error set;

[0128] Mutually exclusive relationship: check if the mutually exclusive fields specified by the rule exist in the JSON array at the same time, if the mutually exclusive fields coexist, record the “field mutual exclusion” error information to the error set;

[0129] Repeat the above rule verification process until all fields are verified.

[0130] After the rule verification is completed, enter the conflict resolution phase:

[0131] Strict mode: check if the error set is empty, if not, clear the compliant field set and output only the error set, otherwise directly output the compliant field set; or, lenient mode: filter out all error fields associated with errors from the JSON array, retain the compliant fields not associated with errors to form a compliant field set, and output the compliant field set and error set; according to the error information of the error set, the conflict is resolved, thereby providing a more accurate and reliable data source for subsequent form auto-generation, solving the problem of missing implicit logical dependency between fields, and reducing the form filling error rate.

[0132] The method proposed in the embodiments of the present application dynamically prompts the word engineering, loads the data defined by the preset expression structure in real time and generates role-based instructions, so that the multi-modal large model adapts to different government affairs forms, dynamically binds images, voice inputs and preset form structures, solves the structured output misplacement; uses government entity relationship reasoning to model field causal dependence, avoids logical conflicts, realizes cross-modal information joint reasoning and automatic filling, significantly improves the accuracy and efficiency of government declaration, and improves the dynamic adaptive ability of government declaration.

[0133] Based on the same technical concept, the embodiment of the present application also proposes a government affair intelligent interaction and information extraction device, as shown in the figure, the government affair intelligent interaction and information extraction device 200 comprises: Figure 2

[0134] The input collection and preprocessing module 201 is configured to uniformly convert the non-image materials uploaded by the user into standardized format image input multi-modal large model, process the voice input of the user into recognizable voice segment input multi-modal large model, and dynamically obtain the form template corresponding to the business matter currently handled by the user, and load all fields, field types, constraint conditions and related rules contained in the multi-modal large model in real time. The specific implementation process is described in the above step 1, which will not be repeated here.

[0135] The structured adaptation module 202 is configured to dynamically inject the form template, the user answer text and the preset question rhetoric sequence into the prompt word instruction of the multi-modal large model through the dynamic prompt word engineering, control the multi-modal large model to generate only the JSON array containing the field ID, the field name and the corresponding field value, and automatically append the state mark to each field; wherein the user answer question and answer is the user's original answer content obtained by recognizing the input image and / or voice. The specific implementation process is described in the above step 2, which will not be repeated here.

[0136] And the logic verification module 203 is configured to use knowledge graph technology to perform business logic verification on the JSON array output by the multi-modal large model. The specific business logic verification process is described in the above step 3, which will not be repeated here.

[0137] Further, the government affair intelligent interaction and information extraction device 200 proposed by the embodiment of the present application further comprises:

[0138] The automatic filling module 204 is configured to automatically fill the form according to the JSON array verified by the business logic.

[0139] Further, the structured adaptation module 202 of the embodiment of the present application is also configured to:

[0140] ​The field ID output by the multi-modal large model is subjected to three-layer semantic verification by the ID semantic corrector to cope with the possible identification errors or fabrication of the field ID by the large model. The first layer is accurate matching, checking whether the field ID output by the large model exists in the correct field ID set of the form template; if no successful matching is made, the second layer of semantic arbitration is entered, the bge-large-zh model is used to vectorize the cosine similarity between the field description output by the large model and all field titles in the form template, and when the highest similarity score exceeds a preset threshold (for example, greater than 0.8), the correct field ID in the form template is automatically replaced with the error field ID output by the large model; if the semantic similarity also fails to meet the threshold requirement, the third layer of fabrication suppression is entered, and the value of the field ID is marked as "null" to prevent the error information from polluting the final form data. The specific verification process is as described in the above method, which will not be repeated here.

[0141] Based on the same technical concept, the embodiment of the present application also provides a government affair intelligent interaction and information extraction system, as shown in Figure 3 The government affair intelligent interaction and information extraction system 300 provided by the embodiment of the present application comprises:

[0142] an input device 301, an output device 302, a processor A 303 and a memory A 304; wherein the number of the processor A 303 and the memory A 304 can be one or more, Figure 3 The input device 301, the output device 302, the processor A 303 and the memory A 304 can be connected through a bus or other means, Figure 3 For example, the connection through the bus is taken as an example.

[0143] The processor A 303 is configured to execute the following steps by calling the operation instructions stored in the memory A 304:

[0144] Multi-modal input collection and preprocessing: the non-image materials uploaded by the user are uniformly converted into standardized format images, which are input into the multi-modal large model; the voice input of the user is processed into recognizable voice segments, which are input into the multi-modal large model, and according to the current business matter handled by the user, the form template corresponding to the business matter is dynamically acquired, and all fields, field types, constraint conditions and related rules contained in the form template are loaded into the multi-modal large model in real time;

[0145] Multimodal large model output aligned with form structure: Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input images and / or speech.

[0146] Knowledge graph-driven logic validation: Utilize knowledge graph technology to validate the business logic of the JSON array output by a multimodal large model.

[0147] Optionally, by calling the operation instructions stored in memory A304, processor A303 is also used to execute any of the corresponding implementation methods in the above-described government affairs intelligent interaction and information extraction method.

[0148] Based on the same technical concept described above, this application also proposes an electronic device, such as... Figure 4 As shown, the electronic device 400 includes: a memory B410, a processor B420, and a computer program A411 stored in the memory B410 and executable on the processor B420. When the processor B420 executes the computer program A411, it performs the following steps:

[0149] Multimodal input acquisition and preprocessing: Non-image materials uploaded by users are uniformly converted into standardized format image input multimodal large model, user voice input is processed into recognizable voice segments input multimodal large model, and according to the business matter currently handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints and related rules contained therein are loaded into the multimodal large model in real time;

[0150] Multimodal large model output aligned with form structure: Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field; wherein the user response questions are the original user responses obtained by recognizing the input images and / or speech.

[0151] Knowledge graph-driven logic validation: Utilize knowledge graph technology to validate the business logic of the JSON array output by a multimodal large model.

[0152] Optionally, when processor B420 executes computer program A411, it can implement any of the corresponding embodiments in the above-described government affairs intelligent interaction and information extraction method.

[0153] It should be noted that the electronic device proposed in the embodiments of the present application is a device used to implement the above-mentioned government affair intelligent interaction and information extraction method, and therefore based on the above-mentioned government affair intelligent interaction and information extraction method proposed in the embodiments of the present application, those skilled in the art can understand the specific implementation of the electronic device of the embodiments of the present application and its various forms of change, and therefore the specific implementation of the electronic device for implementing the above-mentioned government affair intelligent interaction and information extraction method will not be described in detail here, as long as those skilled in the art implement the electronic device used for the above-mentioned government affair intelligent interaction and information extraction method, it belongs to the scope of protection of the present application.

[0154] Based on the same technical concept, the embodiments of the present application also propose a computer readable storage medium, as shown in the figure, the computer readable storage medium 500 stores a computer program B511, and the computer program B511 is executed by a processor to implement the following steps: Figure 5

[0155] Multi-modal input collection and preprocessing: converting the non-image materials uploaded by the user into standardized format image input into the multi-modal large model, processing the voice input of the user into recognizable voice segment input into the multi-modal large model, and dynamically obtaining the form template corresponding to the business matter currently handled by the user, and loading all fields, field types, constraint conditions and related rules contained in the form template into the multi-modal large model in real time;

[0156] Multi-modal large model output and form structure alignment: through dynamic prompt word engineering, the form template, user answer text and pre-set question rhetoric sequence are dynamically injected into the prompt word instruction of the multi-modal large model, the multi-modal large model is controlled to generate only a JSON array containing field ID, field name and corresponding field value, and a state marker is automatically added to each field; wherein the user answer question and answer is the original answer content of the user obtained by recognizing the input image and / or voice;

[0157] Knowledge graph driven logic verification: using knowledge graph technology to verify the business logic of the JSON array output by the multi-modal large model.

[0158] Optionally, the computer program B511 executed by the processor can implement any of the embodiments of the above-mentioned government affair intelligent interaction and information extraction method.

[0159] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0160] ​Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one

[0161] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 an apparatus to perform the functions specified in the flowchart block or blocks.

[0162] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 an apparatus to perform the functions specified in the flowchart block or blocks.

[0163] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination thereof. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 an apparatus to perform the functions specified in the flowchart block or blocks.

[0164] The above detailed description has shown, described, and pointed out the aspects of the application in sufficient detail, that those skilled in the art are able to practice the application. It is understood that various modifications, substitutions, and changes can be made to the methodologies disclosed herein by those skilled in the art without departing from the intended spirit and scope of the application as defined by the appended claims. Accordingly, the particular embodiments described in the detailed description are illustrative only and not intended to limit the scope of the application as defined by the appended claims and equivalents thereof.

Claims

1. A method for intelligent government interaction and information extraction, characterized in that, include: Non-image materials uploaded by users are uniformly converted into standardized image inputs into the multimodal large model. User voice input is processed into recognizable voice segments and input into the multimodal large model. Based on the business matter currently being handled by the user, the form template corresponding to the business matter is dynamically obtained, and all fields, field types, constraints, and related rules contained in the form template are loaded into the multimodal large model in real time. Through dynamic prompt word engineering, form templates, user response text, and preset question word sequences are dynamically injected into the prompt word instructions of the multimodal large model. This controls the multimodal large model to generate only JSON arrays containing field IDs, field names, and corresponding field values, and automatically appends status flags to each field. The user response text is the original user response content obtained by recognizing the input image and / or speech. The business logic of the JSON array output by the multimodal large model is validated using knowledge graph technology. The dynamic prompt word engineering constructs a dynamic, structured prompt word instruction generation model, which deeply integrates the form template of the current business item, the predefined question script sequence, and the user's answer text to generate prompt word instructions that constrain the output of the multimodal large model, so that the output of the multimodal large model is aligned with the form structure. The process of achieving multimodal large model output and form structure alignment based on the aforementioned dynamic prompt word engineering also includes: An ID semantic corrector is used to perform three layers of semantic verification on each field ID output by the multimodal large model. The first layer is exact matching: checking whether the field ID output by the multimodal large model exists in the set of correct field IDs in the form template. If no match is found, the second layer of semantic arbitration is performed: using model vectorization, the cosine similarity between the field name output by the multimodal large model and all field titles in the form template is calculated. When the highest similarity exceeds a preset threshold, the correct field ID in the form template is automatically replaced with the incorrect field ID output by the multimodal large model. If the highest similarity does not reach the preset threshold, the third layer of fictitious suppression is performed: the value of the field ID is marked as "null".

2. The method for intelligent government interaction and information extraction according to claim 1, characterized in that, Also includes: Automatic form filling is performed based on a JSON array that has passed business logic validation.

3. A method for intelligent government interaction and information extraction according to claim 1 or 2, characterized in that, The process of converting the user's voice input into recognizable speech segments specifically includes: A dual-threshold voice activity detection mechanism is used to process user voice input, including: The short-time energy and zero-crossing rate of the speech signal are calculated in real time. When the short-term energy exceeds a preset high threshold and reaches a preset duration, a valid voice segment is determined to have started. For signal segments whose short-term energy is below a preset high threshold but whose zero-crossing rate is consistently above its preset threshold, they are identified as silent speech segments and merged with adjacent speech segments; When the short-term energy is continuously lower than a preset low threshold and the zero-crossing rate is also continuously lower than its preset threshold, and the duration meets the preset silence duration threshold, then the current valid voice segment is determined to have ended.

4. A method for intelligent government interaction and information extraction according to claim 1 or 2, characterized in that, The dynamic prompt word engineering involves constructing a dynamic, structured prompt word instruction generation model. This model deeply integrates the form template of the current business transaction, predefined question word sequences, and user response text to generate prompt word instructions that constrain the output of the multimodal large model, aligning the multimodal large model output with the form structure. The specific process includes: The input form template is parsed, and field attributes are extracted to generate a field mapping table, which contains four columns of data: question word sequence, form field name, field type, and conversion rule; The input user response text is segmented into a sequence of semantic units that match the question wording sequence; Based on the field types and conversion rules defined in the field mapping table, the value of each semantic unit is standardized and converted. The matched and transformed field data are forcibly encapsulated into standardized JSON arrays according to the output specifications; each JSON array contains only: field ID, field name, and field value; The JSON array is double-validated using regular expressions and a syntax validator to ensure that the format is valid and the fields are effective, while filtering out any illegal or undefined fields.

5. The method for intelligent government interaction and information extraction according to claim 4, characterized in that, The aforementioned use of knowledge graph technology to perform business logic validation on the JSON array output by the multimodal large model includes: Based on the form template, a lightweight knowledge graph is constructed; Iterate through each field output by the multimodal large model. For the current field, retrieve all association rules with the current field as the main field from the lightweight knowledge graph. If the current field does not have any association rules, mark it as "compliant" and add it to the compliant field set. If the current field has association rules, perform split processing according to the type of the rule: If the rule type is conditional dependency, check whether the field value of the current field is equal to the preset trigger value. If so, further check whether the dependency field specified by the rule exists in the JSON array. If the dependency field is missing, record the "missing dependency field" error message to the error set. If the rule type is a numerical constraint, check whether the field value of the current field conforms to the preset numerical range. If it does not conform, record the "value exceeds the limit" error message to the error set. If the rule type is mutually exclusive, check whether the mutually exclusive fields specified by the rule exist in the JSON array at the same time. If mutually exclusive fields exist, record the "field mutual exclusion" error message to the error set. Repeat the above rule validation process until all fields have been validated; Once the rule verification is complete, the conflict resolution phase begins.

6. The method for intelligent government interaction and information extraction according to claim 5, characterized in that, The construction of a lightweight knowledge graph based on the form template includes: Define a knowledge graph structure; wherein the knowledge graph structure consists of nodes and edges, each node represents a form field, each edge represents a logical relationship between fields, and the logical relationship consists of its subject field, object field, relationship type, and relationship parameters; The hierarchical structure of the form template is parsed to extract the meaning of all fields and constraint attributes, and a JSON tree is generated; Traverse the parsed JSON tree to identify constraint attributes; Based on the semantics of the constraint attributes, the rules are divided into three types of logical relations; Construct a triplet structure according to the relation type to which the logical relation belongs. The triplet structure includes a subject field, a relation type, and an object field. Transform all constraint attributes in the JSON tree into edges of the knowledge graph structure to form a structured relation network. Finally, create a set of nodes to make each form field a unique node and create a set of edges so that each constraint corresponds to one edge, thereby forming a complete knowledge graph structure.

7. A government affairs intelligent interaction and information extraction device, characterized in that, include: The input acquisition and preprocessing module is configured to: uniformly convert non-image materials uploaded by users into standardized format image input multimodal large model; process user voice input into recognizable voice segments and input them into the multimodal large model; and dynamically obtain the form template corresponding to the business matter currently being handled by the user, and load all fields, field types, constraints and related rules contained in the form template into the multimodal large model in real time. The structured adaptation module is configured to: dynamically inject form templates, user response text, and preset question word sequences into the prompt word instructions of the multimodal large model through dynamic prompt word engineering; control the multimodal large model to only generate JSON arrays containing field IDs, field names, and corresponding field values; and automatically append status flags to each field; wherein the user response text is the original user response content obtained by recognizing the input image and / or speech; In addition, the logic verification module is configured to: perform business logic verification on the JSON array output by the multimodal large model using knowledge graph technology; The dynamic prompt word engineering constructs a dynamic, structured prompt word instruction generation model, which deeply integrates the form template of the current business item, the predefined question script sequence, and the user's answer text to generate prompt word instructions that constrain the output of the multimodal large model, so that the output of the multimodal large model is aligned with the form structure. The process of achieving multimodal large model output and form structure alignment based on the aforementioned dynamic prompt word engineering also includes: An ID semantic corrector is used to perform three layers of semantic verification on each field ID output by the multimodal large model. The first layer is exact matching: checking whether the field ID output by the multimodal large model exists in the set of correct field IDs in the form template. If no match is found, the second layer of semantic arbitration is performed: using model vectorization, the cosine similarity between the field name output by the multimodal large model and all field titles in the form template is calculated. When the highest similarity exceeds a preset threshold, the correct field ID in the form template is automatically replaced with the incorrect field ID output by the multimodal large model. If the highest similarity does not reach the preset threshold, the third layer of fictitious suppression is performed: the value of the field ID is marked as "null".

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the government intelligent interaction and information extraction method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the government intelligent interaction and information extraction method as described in any one of claims 1-6.

Images