Multi-modal bill processing method based on dynamic knowledge enhancement

Abstract

The invention discloses a multi-modal bill processing method based on dynamic knowledge enhancement. The multi-modal bill processing method comprises the following steps: S1, constructing a dynamic knowledge base containing an aging weight; s2, synchronously processing text, image and format features of the bill by adopting a multi-modal feature fusion network to generate a composite feature vector; s3, semantic-level, format-level and timeliness three-level fusion retrieval is carried out based on the composite feature vector, and a three-level fusion retrieval engine comprises dynamic weighted sorting with timeliness attenuation, a difference degree triggered artificial review mechanism and a policy sensitive slope adjustment algorithm; s4, setting a multi-expert cooperative verification system, wherein the multi-expert cooperative verification system comprises cooperative work of a rule engine, a large language model and a logical reasoning module; s5, implementing a dynamic knowledge updating mechanism, and automatically triggering incremental learning of the knowledge base when policy change or format update is detected; and S6, outputting structured data, and synchronously generating an auditing traceability chain containing a decision path. According to the method, the key field identification accuracy can be improved, and auditing traceability and non-perceptual increment updating in the whole process are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal bill processing method based on dynamic knowledge enhancement. Background Art

[0002] As a core component of FinTech, intelligent bill processing technology has been widely used in fields such as fiscal and tax management, insurance claims, and medical settlement. Existing technologies primarily build processing systems based on optical character recognition (OCR), rule engines, and machine learning models. Typical technical solutions include:

[0003] 1. Static knowledge base technology: Compliance judgment systems based on rule engines or fixed policy libraries (such as OCR + rule matching) rely on regular manual updates to the knowledge base. These systems suffer from the following main issues: 1) Lack of timeliness: Policy changes require manual re-labeling of training data, resulting in delayed system updates and increased misjudgment rates; 2) Fixed weighting: The importance assessment of knowledge units relies on preset weights, making it impossible to dynamically adapt to regional policy differences. For example, the failure rate for regionally specific processing of medical bill reimbursement rules is high.

[0004] 2. Cascaded multimodal processing: This approach processes text, image, and layout features in a staged, serial manner. Feature fusion is achieved through simple concatenation or weighted averaging, resulting in the loss of cross-modal spatiotemporal correlations. This approach suffers from the following main issues: 1) Feature fragmentation: This approach ignores the spatial correlation of invoice elements, resulting in reduced recognition accuracy for key fields (such as invoice codes and verification digits); and 2) Inadequate spatiotemporal modeling: This approach fails to consider the dynamic nature of invoice layouts, leading to significant feature shifts between historical training data and the new layout.

[0005] 3. Single-dimensional retrieval technology: Mainstream solutions such as Elasticsearch semantic retrieval or CNN image similarity matching have bottlenecks related to the single retrieval dimension, including: 1) Semantic limitations: Relying solely on text similarity matching, it is unable to identify layout anomalies in tampered bills; 2) Timeliness disconnection: There is no mechanism to associate timeliness weights with retrieval results. In time-sensitive scenarios, the risk rate of using expiration policy rules is too high.

[0006] 4. Black-box AI model: The end-to-end bill classification system based on deep learning lacks records of the decision-making process, cannot meet audit traceability requirements, and manual review is too time-consuming. Summary of the Invention

[0007] In response to the common problems of static knowledge solidification, lack of modal interaction, single retrieval dimension, low decision credibility and lagging knowledge iteration in current intelligent bill processing technology, the present invention proposes a multimodal bill processing method based on dynamic knowledge enhancement. Through the construction of dynamic time-sensitive knowledge graph, multimodal spatiotemporal association modeling, three-level fusion retrieval engine and trusted decision traceability system, the knowledge update delay is reduced, the key field recognition accuracy is improved based on cross-modal feature association modeling, the auditable traceability of the entire process is achieved, and imperceptible incremental updates are used to ensure business continuity.

[0008] The present invention is achieved through the following technical solutions:

[0009] A multimodal bill processing method based on dynamic knowledge enhancement comprises the following steps:

[0010] S1. Construction of a dynamic time-sensitive knowledge graph: Build a dynamic knowledge base containing time-sensitive weights;

[0011] S2. Multimodal spatiotemporal correlation modeling: A multimodal feature fusion network is used to simultaneously process the text, image, and layout features of the bill to generate a composite feature vector.

[0012] S3. Three-level fusion search engine: This engine conducts three-level fusion search based on composite feature vectors, focusing on semantic, format, and timeliness. This engine includes dynamic weighted ranking with timeliness decay, a manual review mechanism triggered by difference, and a policy-sensitive slope adjustment algorithm.

[0013] S4. Implement a multi-expert collaborative verification mechanism: the rule engine, large language model, and logical reasoning module collaborate to complete verification;

[0014] S5. Implement a dynamic knowledge update mechanism to automatically trigger incremental learning of the knowledge base when policy changes or format updates are detected;

[0015] S6. Output structured data and simultaneously generate an auditable traceability chain including the decision path.

[0016] Furthermore, the specific operation of step S1 is as follows:

[0017] S1.1 Multi-source heterogeneous knowledge acquisition: Use distributed crawling technology to collect bill specification texts, industry standard documents, and historical identification records to build cross-modal knowledge units that include semantic features, image features, and business logic;

[0018] S1.2 Establish a timeliness scoring model based on dual-channel heterogeneous feature fusion: This includes a structured business rule feature channel, an unstructured policy text channel, a spatiotemporal feature fusion module, and a timeliness weight fusion module. This model evaluates timeliness based on the authority of the knowledge unit source, industry compliance requirements, and timeliness sensitivity, generating the initial timeliness weight w0 for the knowledge unit.

[0019] S1.3 Construct a dynamic decay factor mechanism with access frequency compensation: attach a time decay function to each knowledge unit:

[0020] η(t)=e^(-λΔt),

[0021] Among them, λ is the industry sensitivity coefficient, Δt is the interval between the current time and the knowledge effective time;

[0022] Generate real-time decay weights:

[0023] w t =w0×η(t)+μ×log(N_access+1),

[0024] Where μ is the adjustment coefficient, which is used to control the impact of access frequency on the overall weight. It is usually calibrated through experiments based on business needs. N_access represents the number of times a knowledge unit is accessed during the statistical period (such as the number of times the bill recognition service is called or the number of times a knowledge node is queried).

[0025] S1.4 Build a version identification system for blockchain evidence storage: Use a layered structured encoding method to generate a 29-bit version identifier. The encoding rules are: semantic prefix layer + time feature layer + security verification layer. Through blockchain evidence storage, an unalterable version traceability chain is formed, supporting version backtracking and conflict detection during multimodal bill verification.

[0026] S1.5 Build a dual-trigger incremental update mechanism for the knowledge base: Set up a dual-trigger mechanism (external compliance + internal attenuation). When changes in industry regulations are detected or the timeliness weight of a single knowledge unit decays to the threshold θ, the incremental version iteration of the knowledge base is started.

[0027] Furthermore, the specific operation of step S2 is as follows:

[0028] S2.1: Construct a parallel heterogeneous feature extraction network to obtain the text features T∈R^{d_t}, image features I∈R^{d_i}, and layout features L∈R^{d_l} of the bill through the text encoder, image encoder, and layout encoder respectively, where {d_t}, {d_i}, and {d_l} represent the dimensions of the text, image, and layout feature vectors, respectively;

[0029] S2.2: Implement cross-modal feature alignment and establish a shared semantic space mapping; map the features of each modality to a unified dimensional space through the learnable projection matrix text modality projection matrix W_t∈R^{d_t×d}, image modality projection matrix W_i∈R^{d_i×d}, and layout modality projection matrix W_l∈R^{d_l×d}:

[0030] T′=LayerNorm(W_t·T);

[0031] I' = LayerNorm(W_i·I);

[0032] L' = LayerNorm(W_l·L);

[0033] In the above formula, T' is the text feature in the unified semantic space obtained by linearly transforming the original text feature matrix T to dimension d through the learnable projection matrix W_t and then normalizing it through the LayerNorm layer; I' is the original image feature matrix I, which is linearly transformed to dimension d through the learnable projection matrix W_i and then normalized through the LayerNorm layer to obtain the image feature in the unified semantic space; L' is the original layout feature matrix L, which is linearly transformed to dimension d through the learnable projection matrix W_l and then normalized through the LayerNorm layer to obtain the layout feature in the unified semantic space; LayerNorm is a layer normalization operation, which standardizes the projected features to make their mean 0 and variance 1, eliminating the differences in feature distribution between modalities and improving the stability of model training;

[0034] S2.3: Perform cross-modal attention fusion with orthogonal constraints to complete interpretable feature synthesis with gradient scale regularization; use a multi-head cross-modal attention mechanism to calculate cross-modal (text to image and layout features) feature correlation:

[0035] α=Softmax((Q·K^T) / √d),

[0036] Where Q = T' is the text feature in dimension d, K = Linear_Projection(Concat(I',L')), which is the key matrix after the image and layout features are concatenated and projected to d dimensions via linear transformation, where Linear_Projection is a learnable linear transformation layer;

[0037] Generate composite feature vectors by adaptive weight fusion:

[0038] F=α·V+β_text·T',

[0039] Among them, α·V represents the weighted attention result of text features and image / layout features, where V = T'; β_text is a learnable dynamic gating parameter that controls the retention strength of text features, and β_text·T' is a residual connection that retains the original text features. The constraint condition is: β_text∈[0,1], which is generated by the Sigmoid function.

[0040] Furthermore, the three-level fusion search described in step S3 matches the bill features. The specific operation steps include:

[0041] S3.1: Semantic level retrieval: Match the text semantics of the field, map the composite feature vector Φ to the semantic space Ω through the semantic encoder E_s, and calculate the semantic similarity with the knowledge unit k_i in the dynamic knowledge base K:

[0042] S_s(Φ,k_i)=cos(E_s(Φ),E_s(Ψ_i))·α_t,

[0043] Among them, S_s(Φ,k_i) is the semantic similarity score between the composite feature vector Φ and the i-th knowledge unit k_i in the knowledge base. Its value consists of two parts: semantic space similarity: the matching degree between the vector E_s(Φ) mapped by the semantic encoder E_s and E_s(Ψ_i) is calculated by cosine similarity, reflecting the degree of alignment between the two in the semantic space Ω, where Ψ_i is the feature vector of the knowledge unit; timeliness weight α_t: a dynamic attenuation factor used to dynamically adjust the time sensitivity of the knowledge unit, satisfying α_t = max(0,1-λ·Δt), where λ∈(0,0.1] is the attenuation coefficient, Δt is the knowledge update interval, and the timeliness weight value decreases as the knowledge update interval Δt increases (λ controls the attenuation rate), ensuring that the retrieval results take into account both semantic relevance and timeliness;

[0044] S3.2: Layout-level retrieval: Compare the feature space layout, extract the bill layout feature vector Φ_g, construct the layout topology graph G_layout = (V, E), and calculate the layout similarity with the knowledge base template T_j through the graph topology matching algorithm:

[0045] S_g(T_j)=∑_{v∈V}w(v)·δ(l(v),l'(v'_v))+γ·|E∩E'| / |E∪E'|,

[0046] Among them, S_g(T_j) is the layout similarity score between the current bill layout topology graph G_layout and the knowledge base template T_j, which is composed of node matching weight (calculating local matching through node label consistency and spatial weight) and edge structure similarity (measuring the global alignment of topological structures through edge set intersection ratio); {v∈V} is the set of all nodes in the current bill layout topology graph G_layout, each node v corresponds to a layout element in the bill (such as title field, table area, QR code area), and V is the set of current bill layout elements; w(v) is the node spatial weight; δ(·) is the label matching function; l(v) is the semantic label of the current bill node v; l'(v'_v) is the template The corresponding node in T_j that is aligned with the current bill node v (alignment rule: v'_v is determined by label matching and spatial position similarity); γ∈[0.2,0.5] is the edge matching coefficient; |E∩E'| / |E∪E'| is the Jaccard similarity coefficient of the edge, which measures the overlap ratio of the two graphs in the topological structure, and the value range is [0,1]. The larger the value, the more similar the topological structure. Among them, E is the edge set in the current bill layout topology graph, which represents the connection relationship between layout elements (such as adjacent relationship, hierarchical relationship); E′ is the edge set in the template T_j layout topology graph, the numerator |E∩E′| is the number of edges shared by the two graphs (topological structure overlap), and the denominator |E∪E′| is the union number of all edges in the two graphs;

[0047] S3.3 Time-sensitive retrieval: including dynamic weighted sorting with time decay: screening knowledge records within the validity period, establishing a time-sensitive weight matrix W = [α_1, α_2, ..., α_n]^T, where α_i is the time-sensitive weight of the i-th knowledge unit, satisfying α_i = 1-λ·Δt_i; dynamic weighted sorting of search results based on three-level mixed similarity:

[0048] RankScore=S_s⊙W+S_g⊙W+β·f(t_c,t_e),

[0049] Among them, RankScore is the dynamic weighted ranking score of the knowledge retrieval result; ⊙ is the Hadamard product, which represents the element-by-element multiplication of the vector; S_s is the semantic similarity score vector, which is generated by the semantic-level retrieval in S3.1; S_s⊙W is the semantic similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the semantic similarity and the time weight, reflecting the time correction value of the semantic matching; S_g is the layout similarity score vector, which is generated by the layout-level retrieval in S3.2, and S_g⊙W is the layout similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the layout similarity and the time weight. element-wise multiplication), reflecting the timeliness correction value of layout matching; β·f(t_c,t_e) is the timeliness decay function term, which is a nonlinear decay compensation term based on the validity period of knowledge. β∈[0.3,0.6] is the timeliness coefficient, which controls the contribution weight of the timeliness decay function to the total score. The larger the value, the stronger the influence of timeliness on the ranking; f(t_c,t_e)=1 / (1+exp(k(t_e-t_c))) is the timeliness decay function, which is used to smoothly adjust the score of records approaching expiration. t_c is the current time, t_e is the expiration time of the knowledge record, and k is the adjustment slope.

[0050] S3.4 builds a three-level retrieval fusion model and generates the final retrieval results through a gating mechanism:

[0051] G_gate=σ(W_g·[S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}]+b_g),

[0052] FinalResult=[S_s^{(i)};S_g^{(i)};RankScore^{(i)}]⊙G_gate,

[0053] Among them, G_gate is the gating vector, which is used to dynamically adjust the contribution of different features to the final result and realize the fusion control of multimodal features; W_g∈R^(d×3) is the learnable gating matrix, d is the input feature dimension; b_g is the bias term; σ is the sigmoid function; S_s^{(i)} and S_g^{(i)} are the semantic and layout scores of the i-th candidate respectively; RankScore^{(i)} is the ranking score of the i-th candidate, which is used to measure the matching degree between the retrieval result and the query; FinalResult is the gated weighted fusion feature vector, which is obtained by performing the Hadamard product of the original features [S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}] with the gate vector G_gate, and multiplying them element by element, and is used to generate a ranked list or calculate the comprehensive score of the retrieval result.

[0054] Furthermore, the semantic encoder E_s described in step 3.1 adopts a dual-tower structure based on contrastive learning, and its loss function is:

[0055] L_c=-log[exp(s(Φ,Ψ^+) / τ) / (exp(s(Φ,Ψ^+) / τ)+∑_{Ψ^-}exp(s(Φ,Ψ^-) / τ))];

[0056] Among them, τ is the temperature hyperparameter, Ψ^+ is the positive sample, Ψ^- is the negative sample, and s(·) is the similarity calculation function.

[0057] Furthermore, the node space weight w(v) of the layout topology graph G_layout described in step 3.2 is calculated as follows:

[0058] w(v)=1 / (1+min(d(v,C),d(v,M))·σ_A),

[0059] Where d(v,C) is the normalized distance from the node to the bill center, d(v,M) is the distance from the node to the nearest page edge, and σ_A∈[0.1,0.3] is the regional sensitivity coefficient.

[0060] Furthermore, the construction of the multi-expert collaborative verification system described in step S4 includes the following steps:

[0061] S4.1: Build a collaborative verification system that includes a rule engine, a large language model, and a logical reasoning module. Each module integrates the results through a dynamic arbitration mechanism.

[0062] S4.2: The rule engine performs timeliness check: Based on the timeliness rule set in the dynamic knowledge base:

[0063] R={r_i|i∈N},

[0064] Create a verification function:

[0065] f_r(x)=Σw_i·σ(x·r_i),

[0066] Among them, w_i is the rule weight, σ is the Sigmoid activation function;

[0067] S4.3: The large language model performs semantic association analysis: Generate a semantic feature vector h = Φ(text) ⊕ Φ(image_caption) through the pre-trained language model Φ, and establish a semantic confidence score s = softmax(W·h+b);

[0068] S4.4: The logical reasoning module verifies the rationality of business logic: Construct a business rule graph G_rule = (V, E), implement logical constraint propagation through a graph neural network, and output the probability of logical compliance:

[0069] p=GNN(G_rule,node_feature);

[0070] S4.5: Establish a dynamic arbitration mechanism of rule engine + large language model + logical reasoning: When the difference Δ>δ_th output by each module, the arbitration decision D = argmax(η·f_r+θ·s+ι·p) is triggered, where the difference is calculated as: Δ = 1-min(f_r,s,p) / max(f_r,s,p), δ_th is the preset difference threshold, and the dynamic adjustment coefficient satisfies η, θ, ι are the weights of the rule engine, large language model, and logical reasoning module respectively, and η+θ+ι=1.

[0071] Furthermore, the semantic association analysis described in step S4.3 includes an attention enhancement mechanism: establishing a cross-modal attention matrix A = QK^T / √d, where Q is the text feature matrix, K is the image feature matrix, and d is the dimensionality scaling factor.

[0072] Furthermore, the implementation of the dynamic knowledge update mechanism described in step S5 includes the following steps:

[0073] S5.1 Change Detector: A dual-channel monitoring architecture is used to monitor the policy document version number and the distribution of bill format sample features in real time. When a version number update or a format feature offset exceeding a threshold is detected, the incremental learner is activated to perform a knowledge base update. The dual-channel monitoring architecture includes:

[0074] 1) Policy channel: Parse the DOM tree structure of government announcement webpages and capture version number revision patterns;

[0075] 2) Layout channel: Calculate the SSIM structural similarity index of the newly received bill image;

[0076] 3) When any channel triggers an exception, the incremental update process is started;

[0077] S5.2 Incremental Learner: Uses an elastic weight fixation algorithm to update knowledge base parameters while retaining important parameters of historical knowledge. The loss function constructed by the elastic weight fixation algorithm is:

[0078] L(θ)=L_task(θ)+λΣ_i F_i(θ_i-θ_{old,i}) 2 ;

[0079] Where F_i is the diagonal term of the Fisher information matrix of the historical task parameters, and λ = 0.5 is the elastic constraint coefficient;

[0080] The updating of the knowledge base parameters adopts a two-stage mechanism, including:

[0081] Freeze phase: lock the convolutional feature extraction layer parameters and only update the fully connected classification layer;

[0082] Unfreezing stage: When the classification accuracy drops by more than 5%, the convolutional layer parameters are unfrozen and global fine-tuning is performed;

[0083] S5.3 Version Controller: Maintains the parallel retrieval capability for multiple versions of knowledge and implements version routing based on the timeliness of retrieval requests. The version controller implements the following when implementing parallel retrieval of multiple versions:

[0084] 1) Construct a version routing matrix M∈R^{n×d}, where n is the number of versions and d is the time-sensitive feature dimension;

[0085] 2) Calculate the temporal similarity s = softmax(qM^T) between the query vector q and each version vector;

[0086] 3) The version with s_max>0.7 is selected as the main retrieval channel, and the remaining versions are used as auxiliary channels.

[0087] Furthermore, the method for generating the auditable traceability chain in step S6 includes the following steps:

[0088] S6.1 While outputting structured data, it also constructs an auditable traceability chain Γ, where:

[0089] Γ={N_i|i∈[1,n]}, each node N_i=(E_i,R_i,H_i), a traceability node containing the triple verification elements of elements, rules, and fingerprints:

[0090] a) Bill element E_i∈{T_j,I_k,L_m}, where T_j is the text feature vector, I_k is the image feature matrix, and L_m is the layout feature tensor;

[0091] b) Decision rule R_i∈{Ψ_sem,Ψ_lay,Ψ_time}, Ψ_sem is the semantic level retrieval function,

[0092] is the layout convolution kernel, Ψ_time=σ(t·w_t) is the timeliness weight function;

[0093] The generation method of decision rule R_i includes:

[0094] 1) Extracting multi-expert collaborative verification results

[0095] 2) Calculate the rule activation α = Softmax(W_c·C);

[0096] 3) Determine the dominant rule R_i = argmax{α_sem,α_lay,α_time};

[0097] c) Hash fingerprint H_i = Hash(E_i||R_i||t_i), where t_i is the timestamp;

[0098] S6.2 Dynamically generate traceability chain association topology to meet the following requirements:

[0099] Make

[0100] Form a decision dependency graph G_decision = (Γ, ε);

[0101] The steps for constructing the decision dependency graph G_decision include:

[0102] 1) Create a cross-modal association matrix A = ReLU(T·W_TI·I^T+L·W_LL·L^T);

[0103] 2) Define the dependency edge ε_ij = 1 if and only if A_{ij}>μ·max(A), where μ∈(0,1) is the dynamic association threshold;

[0104] S6.3 implements a double encryption mechanism and performs the following on the traceability chain:

[0105]

[0106] Among them, K_sym is the symmetric key and K_priv is the private key;

[0107] S6.4 Reverse verification mechanism for deconvolution reconstruction: The digital signature and gradient sensitivity are verified layer by layer through the traceability parser. The verification conditions are:

[0108]

[0109] Among them, Verify(Γ) is the overall legitimacy judgment of the traceability chain Γ, which is true if and only if all levels are verified. n is the total number of layers in the traceability chain, indicating the number of steps from the current data block back to the source. Verify_Sig(H_i') is the digital signature verification result of the hash H_i' of the data block at the i-th layer. is the gradient sensitivity of the reconstruction error of the i-th layer to the deconvolution result Recon{i-1} of the previous layer, and θ is the gradient sensitivity threshold, which is used to determine the risk of data tampering;

[0110] The reverse verification includes:

[0111] 1) Implement chained deconvolution verification: DeConv(Γ) = f^{-1}(W_d [H_1'; ...; H_n']^T);

[0112] 2) Output reconstruction error matrix Δ = ||X_orig - X_recon||_F;

[0113] 3) When Δ<ε, it is determined to be a valid audit chain, where ε=σ(n·logδ), and δ is the independent error parameter.

[0114] (3) Beneficial effects

[0115] The multimodal bill processing method based on dynamic knowledge enhancement proposed in this invention has the following beneficial effects compared with the prior art:

[0116] (1) The present invention proposes a new paradigm for multimodal intelligent bill processing with dynamic knowledge enhancement. By constructing a dynamic knowledge base with time-weighted effects, a spatiotemporal-aware multimodal fusion architecture, and an innovative three-level joint search engine, it reduces knowledge update delays, improves the accuracy of key field recognition based on cross-modal feature association modeling, achieves full-process auditability and traceability, and ensures business continuity through imperceptible incremental updates. It is suitable for intelligent bill management in the fields of finance, medical care, taxation, etc.

[0117] (2) When constructing a dynamic knowledge base containing timeliness weights, the present invention combines the timeliness weight modeling and adaptive attenuation mechanism of the dynamic knowledge base, introduces the industry sensitivity coefficient and time attenuation function, and combines the access frequency adjustment item to achieve dynamic attenuation and adaptive enhancement of the knowledge unit weight, balancing the influence of timeliness and usage frequency; constructs a dual-channel feature fusion model of structured business rules and unstructured policy texts, calculates the timeliness initial weight of the knowledge unit through the spatiotemporal feature fusion module, and forms a dual-channel heterogeneous timeliness scoring model.

[0118] (3) The present invention adopts hierarchical structured coding (semantic prefix layer + time feature layer + security verification layer) and blockchain evidence storage technology to generate an unalterable version traceability chain, supporting multi-version knowledge backtracking and conflict detection; by combining coding rules with blockchain, it solves the version confusion problem in multimodal bill verification and ensures the traceability and tamper-proofness of knowledge updates.

[0119] (4) This invention introduces learnable gating parameters into the multimodal feature fusion and three-level fusion retrieval mechanism, designs a parallel heterogeneous feature extraction network (text / image / layout encoder), generates composite feature vectors through a learnable projection matrix and a multi-head cross-modal attention mechanism, realizes nonlinear alignment and information complementarity of multimodal features, and can adaptively adjust the contribution of each modality, thus solving the limitation of fixed weights in traditional multimodal fusion. At the same time, in the time-weighted hierarchical retrieval model, a three-level retrieval fusion method of semantic level, layout level, and timeliness is proposed. The three-level retrieval scores are dynamically integrated through the gating matrix to improve the retrieval accuracy and timeliness sensitivity in complex bill scenarios.

[0120] (5) This invention solves the core problems of multimodal alignment difficulties, policy timeliness sensitivity, low verification reliability, knowledge update lag, and lack of audit traceability in bill intelligent processing through the technology of dynamic modeling of time-weighted dynamic modeling, cross-modal attention fusion, hierarchical retrieval gating mechanism, multi-expert collaborative arbitration, and incremental update-traceability chain integration. The time-weighted decay function and dual-trigger update mechanism enable the self-evolution of the knowledge base to achieve dynamicity; cross-modal attention and gated hierarchical retrieval break through the bottleneck of single-modal processing to achieve integration; rule-semantic-logic multi-expert collaboration and dynamic arbitration improve the robustness of the system to achieve synergy; blockchain version identification and double-encrypted traceability chain ensure that the entire process is auditable and achieve security; these features constitute an efficient, reliable, and explainable bill intelligent processing solution with significant technical advantages and industrial application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0122] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Under the premise of not departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in this field to the technical solutions of the present invention should fall within the scope of protection of the present invention.

[0123] Example 1:

[0124] like Figure 1 As shown, a multimodal bill processing method based on dynamic knowledge enhancement includes the following steps:

[0125] S1. Dynamic Time-Effective Knowledge Graph Construction: Build a dynamic knowledge base containing time-effective weights. The specific operation is as follows:

[0126] S1.1 Multi-source heterogeneous knowledge acquisition: Collect bill specification texts, industry standard documents and historical identification records through distributed crawling technology to build cross-modal knowledge units containing semantic features, image features and business logic.

[0127] S1.2 Establish a timeliness scoring model based on dual-channel heterogeneous feature fusion: This includes a structured business rule feature channel, an unstructured policy text channel, a spatiotemporal feature fusion module, and a timeliness weight fusion module. This model evaluates timeliness based on the authority of the knowledge unit source, industry compliance requirements, and timeliness sensitivity, generating the initial timeliness weight w0 for the knowledge unit.

[0128] The structured business rule feature channel: uses the time-series-aware gated recurrent unit (T-GRU) to generate the basic weight wb ,satisfy

[0129] w b =σ(Wg·[ht-1||(α t ⊕β t )]+bg),

[0130] Among them, σ is the Sigmoid activation function, Wg is the linear transformation matrix, h t-1 is the hidden state of the time-series-aware T-GRU at the previous moment, representing the dynamic characteristics of the historical time series, α t is the source authority of the knowledge unit at time t (such as institutional credit rating, citation volume, etc.), β t is the update frequency of the knowledge unit at time t (e.g., the number of updates per unit time), || represents vector concatenation, ⊕ represents element-by-element addition, and b g is the bias term used to adjust the baseline value of the linear transformation.

[0131] The unstructured policy text channel: parses industry compliance text through the Graph Attention Network (GAT) to generate industry correction coefficients:

[0132]

[0133] Among them, a n is the attention weight of the nth policy node in GAT, which satisfies normalization:

[0134]

[0135] Among them, W k is a linear transformation matrix used to linearly transform policy text features, φ(d n ) is a text encoder (such as the BERT model), which converts the nth policy document node d n The unstructured text is converted into a vector, and Softplus is a smooth non-negative activation function, defined as: Softplus(x) = log(1+e x ), ensuring k i ≥0, and the correction strength is adaptively adjusted through model training.

[0136] The spatiotemporal feature fusion module introduces a dynamic adapter to realize dual-channel feature interaction and calculates the time sensitivity factor γ_t to satisfy:

[0137] γ t =ReLU(w b ||k i ]W f ),

[0138] Among them, || represents vector splicing, is the fusion matrix.

[0139] The time-dependent weight synthesis described above: establishes a weight update mechanism with a time-decay effect and outputs the final weight:

[0140] w0=γ t ·(w b ⊙k i )·e -λΔt ,

[0141] Where ⊙ represents the Hadamard product, λ is the decay rate parameter, and Δt is the time interval from the last update.

[0142] S1.3 Construct a dynamic decay factor mechanism with access frequency compensation: attach a time decay function to each knowledge unit:

[0143] η(t)=e^(-λΔt),

[0144] Among them, λ is the industry sensitivity coefficient, Δt is the interval between the current time and the knowledge effective time;

[0145] Generate real-time decay weights:

[0146] w t =min(w0×η(t)+μ×log(N_access+1),1),

[0147] Among them, μ is the adjustment coefficient, which is used to control the impact of access frequency on the overall weight. It is usually calibrated through experiments based on business needs. N_access represents the number of times the knowledge unit is accessed within the statistical period (such as the number of times the bill recognition service is called and the number of times the knowledge node is queried). Min is the weight normalization operation.

[0148] The time decay factor is dynamically adjusted through a sliding window mechanism. The window period is negatively correlated with the frequency of policy and regulation updates. The specific calculation method is:

[0149] T window =k / f update

[0150] Where k is a constant coefficient (calibrated by business scenarios), and fupdate represents the frequency of policy and regulatory updates per unit time (e.g., the number of monthly updates). This negative correlation ensures that the timeliness weights of knowledge units in frequently updated areas are dynamically adjusted in a shorter cycle, accelerating the decay rate.

[0151] S1.4 Build a version identification system for blockchain evidence storage: Use a layered structured coding method to generate a 29-bit version identifier. The coding rules are: semantic prefix layer + time feature layer + security verification layer. Through blockchain evidence storage, an unalterable version traceability chain is formed, supporting version backtracking and conflict detection during multimodal bill verification.

[0152] The semantic prefix layer in the version identification includes: obtaining the data source identification (2 letters), which is used to distinguish the regulatory agency, enterprise customization or system-derived source; obtaining the policy type code (3 digits), which is coded based on the industry standard classification system; setting the version status identification bit (1 letter), which is used to identify the current valid, historical version or pending confirmation status.

[0153] The time feature layer in the version identification includes: acquisition effective timestamp (8 bits): using the YYYYMMDD compressed format to indicate the policy effective date; generation timestamp (6 bits): using the HHMMSS compressed format to indicate the version generation time, accurate to the second level.

[0154] The security verification layer in the version identification includes: based on the semantic prefix layer and time characteristics, a collision-resistant hash algorithm is used to generate a composite hash value (8-digit hexadecimal); based on the Luhn algorithm, a check digit (1 digit) is generated to verify the integrity of the version identification.

[0155] S1.5 Build a dual-trigger incremental update mechanism for the knowledge base: Set up a dual-trigger mechanism (external compliance + internal attenuation). When changes in industry regulations are detected or the timeliness weight of a single knowledge unit decays to the threshold θ, the incremental version iteration of the knowledge base is started.

[0156] The dual trigger mechanism includes: External compliance trigger: Real-time monitoring of industry standard change events through the regulatory agency API interface, when the bill format standard is updated, a first trigger signal is generated. Internal decay trigger: Using the time-based weight calculation unit, a weight decay operation is performed on each knowledge unit. When any knowledge unit decays its weight w in real time, t A second trigger signal is generated when the value is less than or equal to θ. The θ threshold is set differently by category (e.g., core knowledge units, general content, and time-sensitive content), and is determined based on historical data statistics and ROC curve analysis. To avoid version branch conflicts caused by multiple concurrent trigger signals (e.g., triggering external compliance and internal attenuation simultaneously), a priority arbitration strategy can be adopted (e.g., external triggers take precedence over internal triggers).

[0157] The incremental update mechanism includes: responding to any trigger signal, starting the incremental version iteration engine, creating a branch copy based on the current version to perform knowledge base update operations, while maintaining the online service availability of the main version; generating an incremental update package through a difference comparison module; the update package contains a new node set, an invalid node mark and an associated edge correction matrix; performing a hot switch after verification is completed, marking the old version as a historical snapshot and storing it in the version warehouse.

[0158] S2. Multimodal spatiotemporal correlation modeling: A multimodal feature fusion network is used to simultaneously process the text, image, and layout features of the bill to generate a composite feature vector. The specific operation is as follows:

[0159] S2.1: Construct a parallel heterogeneous feature extraction network to obtain the text features T∈R^{d_t}, image features I∈R^{d_i}, and layout features L∈R^{d_l} of the bill through the text encoder, image encoder, and layout encoder respectively, where {d_t}, {d_i}, and {d_l} represent the dimensions of the text, image, and layout feature vectors, respectively.

[0160] The text encoder includes: a deep bidirectional Transformer network based on the BERT architecture, processing the bill text sequence X_text∈R^{n×d}, and outputting a text feature vector T∈R^{d_t}, where {n×d} represents the two-dimensional tensor dimension of the text feature matrix, n (sequence length) represents the number of tokens in the text sequence (such as words, subwords or character-level units), corresponding to the input length limit in natural language processing, and d (feature dimension) represents the embedding vector dimension of each text token, which is determined by the hidden layer dimension of the pre-trained language model.

[0161] The image encoder includes: improving the coordinate attention image encoding of ResNet, adding a coordinate attention module based on the original residual structure, processing the bill image X_img∈R^{H×W×3}, and outputting the image feature vector I∈R^{d_i}, where {H×W×3} represents the three-dimensional tensor dimension of the input bill image: H (Height) represents the height of the image (pixel unit), corresponding to the vertical resolution of the bill image, W (Width) represents the width of the image (pixel unit), corresponding to the horizontal resolution of the bill image, 3 (Channels) represents the number of color channels, indicating that the image uses the RGB three-channel format (red, green, and blue).

[0162] The calculation process of the improved ResNet coordinate attention image encoding module is:

[0163]

[0164] Among them, AvgPool_h and AvgPool_w represent the average pooling in height and width directions respectively, Indicates element-by-element multiplication, and [] indicates channel concatenation operation.

[0165] The layout encoder includes a layout encoding hybrid architecture based on spatiotemporal perception, which integrates graph convolutional networks and spatial position coding. It processes the bill layout coordinate matrix X_layout∈R^{m×4} and outputs a layout feature vector L∈R^{d_l}, where {m×4} represents the two-dimensional structural dimension of the bill layout coordinate matrix: m (number of elements) represents the total number of layout elements to be parsed in the bill document, such as the number of structured elements such as text paragraphs, table cells, and seal areas; 4 (coordinate parameters) describes the bounding box coordinates of each layout element, in the form of normalized coordinates: (center point c_x, c_y, width w, height h).

[0166] S2.2: Implement cross-modal feature alignment and establish a shared semantic space mapping; map the features of each modality to a unified dimensional space through the learnable projection matrix text modality projection matrix W_t∈R^{d_t×d}, image modality projection matrix W_i∈R^{d_i×d}, and layout modality projection matrix W_l∈R^{d_l×d}:

[0167] T′=LayerNorm(W_t·T);

[0168] I' = LayerNorm(W_i·I);

[0169] L' = LayerNorm(W_l·L);

[0170] In the above formula, T' is the original text feature matrix T, which is linearly transformed to dimension d by the learnable projection matrix W_t and then normalized by the LayerNorm layer to obtain the text features in the unified semantic space; I' is the original image feature matrix I, which is linearly transformed to dimension d by the learnable projection matrix W_i and then normalized by the LayerNorm layer to obtain the image features in the unified semantic space; L' is the original layout feature matrix L, which is linearly transformed to dimension d by the learnable projection matrix W_l and then normalized by the LayerNorm layer to obtain the layout features in the unified semantic space; LayerNorm is a layer normalization operation, which standardizes the projected features to make their mean 0 and variance 1, eliminates the differences in feature distribution between modalities, and improves the stability of model training.

[0171] S2.3: Perform cross-modal attention fusion with orthogonal constraints to complete interpretable feature synthesis; use a multi-head cross-modal attention mechanism to calculate cross-modal (text to image and layout features) feature correlation:

[0172] α=Softmax((Q·K^T) / √d),

[0173] Where Q = T' is the text feature in dimension d, K = Linear_ProjectionConcat(I',L') is the key projection matrix after the image and layout features are concatenated and projected to the d-dimensional space via linear transformation, where Linear_Projection is a learnable linear transformation layer.

[0174] Generate composite feature vectors by adaptive weight fusion:

[0175] F=α·V+β_text·T',

[0176] Here, α·V represents the weighted attentional result of text features and image / layout features, where V = T'. β_text is a learnable dynamic gating parameter that controls the strength of text feature retention. β_text·T' is a residual connection that preserves the original text features, with the constraint that β_text∈[0,1]. This is generated using a sigmoid function. The resulting composite feature vector uses a gradient scaling regularization term to dynamically control the feature contribution, ensuring that the gradient scales of each modality meet the preset constraints.

[0177] Adaptive weight fusion includes a time-sensitive attenuation factor: β = σ(w_β·T'), where σ is the Sigmoid function, t∈R^+ is the feature time-sensitive parameter, and w_β is the learnable parameter matrix.

[0178] S3. Three-level fusion search engine: This engine performs three-level fusion search based on composite feature vectors, focusing on semantics, layout, and timeliness. This engine includes dynamic weighted ranking with timeliness decay, a manual review mechanism triggered by difference, and a policy-sensitive slope adjustment algorithm. The specific steps include:

[0179] S3.1: Semantic level retrieval: Match the text semantics of the field, map the composite feature vector Φ to the semantic space Ω through the semantic encoder E_s, and calculate the semantic similarity with the knowledge unit k_i in the dynamic knowledge base K:

[0180] S_s(Φ,k_i)=cos(E_s(Φ),E_s(Ψ_i))·α_t,

[0181] Among them, S_s(Φ,k_i) is the semantic similarity score between the composite feature vector Φ and the i-th knowledge unit k_i in the knowledge base. Its value consists of two parts: semantic space similarity: the matching degree between the vector E_s(Φ) mapped by the semantic encoder E_s and E_s(Ψ_i) is calculated by cosine similarity, reflecting the degree of alignment between the two in the semantic space Ω, where Ψ_i is the feature vector of the knowledge unit; timeliness weight α_t: a dynamic attenuation factor used to dynamically adjust the time sensitivity of the knowledge unit, satisfying α_t = max(0,1-λ·Δt), where λ∈(0,0.1] is the attenuation coefficient, Δt is the length of the knowledge update interval, and the timeliness weight value decreases as the knowledge update interval Δt increases (λ controls the attenuation rate), ensuring that the retrieval results take into account both semantic relevance and timeliness.

[0182] The semantic encoder E_s adopts a dual-tower structure based on contrastive learning, and its loss function is:

[0183] L_c=-log[exp(s(Φ,Ψ^+) / τ) / (exp(s(Φ,Ψ^+) / τ)+∑_{Ψ^-}exp(s(Φ,Ψ^-) / τ))];

[0184] Among them, τ is the temperature hyperparameter, Ψ^+ is the positive sample, Ψ^- is the negative sample, and s(·) is the similarity calculation function.

[0185] S3.2: Layout-level retrieval: Compare the feature space layout, extract the bill layout feature vector Φ_g, construct the layout topology graph G_layout = (V, E), and calculate the layout similarity with the knowledge base template T_j through the graph topology matching algorithm:

[0186] S_g(T_j)=∑_{v∈V}w(v)·δ(l(v),l'(v'_v))+γ·|E∩E'| / |E∪E'|,

[0187] Among them, S_g(T_j) is the layout similarity score between the current bill layout topology graph G_layout and the knowledge base template T_j, which is composed of node matching weight (calculating local matching through node label consistency and spatial weight) and edge structure similarity (measuring the global alignment of topological structures through edge set intersection ratio); {v∈V} is the set of all nodes in the current bill layout topology graph G_layout, each node v corresponds to a layout element in the bill (such as title field, table area, QR code area), and V is the set of current bill layout elements; w(v) is the node spatial weight; δ(·) is the label matching function; l(v) is the semantic label of the current bill node v; l'(v'_v) is the template The corresponding node in T_j that is aligned with the current bill node v (alignment rule: v'_v is determined by label matching and spatial position similarity); γ∈[0.2,0.5] is the edge matching coefficient; |E∩E'| / |E∪E'| is the Jaccard similarity coefficient of the edge, which measures the overlap ratio of the two graphs in the topological structure. The value range is [0,1]. The larger the value, the more similar the topological structure. Among them, E is the edge set in the current bill layout topology graph, which represents the connection relationship between layout elements (such as adjacent relationship, hierarchical relationship); E′ is the edge set in the template T_j layout topology graph, the numerator |E∩E′| is the number of edges shared by the two graphs (topological structure overlap), and the denominator |E∪E′| is the union number of all edges in the two graphs.

[0188] The node space weight w(v) of the layout topology graph G_layout is calculated as follows:

[0189] w(v)=1 / (1+min(d(v,C),d(v,M))·σ_A),

[0190] Where d(v,C) is the normalized distance from the node to the bill center, d(v,M) is the distance from the node to the nearest page edge, and σ_A∈[0.1,0.3] is the regional sensitivity coefficient.

[0191] S3.3 Time-sensitive retrieval: including dynamic weighted sorting with time decay: screening knowledge records within the validity period, establishing a time-sensitive weight matrix W = [α_1, α_2, ..., α_n]^T, where α_i is the time-sensitive weight of the i-th knowledge unit, satisfying α_i = 1-λ·Δt_i; dynamic weighted sorting of search results based on three-level mixed similarity:

[0192] RankScore=S_s⊙W+S_g⊙W+β·f(t_c,t_e),

[0193] Among them, RankScore is the dynamic weighted ranking score of the knowledge retrieval result; ⊙ is the Hadamard product, which represents the element-by-element multiplication of the vector; S_s is the semantic similarity score vector, which is generated by the semantic-level retrieval in S3.1; S_s⊙W is the semantic similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the semantic similarity and the time weight, reflecting the time correction value of the semantic matching; S_g is the layout similarity score vector, which is generated by the layout-level retrieval in S3.2, and S_g⊙W is the layout similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the layout similarity and the time weight. element-wise multiplication), reflecting the timeliness correction value of layout matching; β·f(t_c,t_e) is the timeliness decay function term, which is a nonlinear decay compensation term based on the validity period of knowledge. β∈[0.3,0.6] is the timeliness coefficient, which controls the contribution weight of the timeliness decay function to the total score. The larger the value, the stronger the influence of timeliness on the ranking; f(t_c,t_e)=1 / (1+exp(k(t_e-t_c))) is the timeliness decay function, which is used to smoothly adjust the score of records approaching expiration. t_c is the current time, t_e is the expiration time of the knowledge record, and k is the adjustment slope.

[0194] The dynamic adjustment rule of the parameter k of the time-effect decay function f(t_c, t_e) is:

[0195] k=k_0·(1+η·ΔP / P_ref),

[0196] Among them, k_0 is the benchmark slope, η∈[0.5,1.2] is the policy sensitivity factor, ΔP is the frequency of policy changes, and P_ref is the frequency of industry benchmark changes.

[0197] S3.4 builds a three-level retrieval fusion model and generates the final retrieval results through a gating mechanism:

[0198] G_gate=σ(W_g·[S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}]+b_g),

[0199] FinalResult=[S_s^{(i)};S_g^{(i)};RankScore^{(i)}]⊙G_gate,

[0200] Among them, G_gate is the gating vector, which is used to dynamically adjust the contribution of different features to the final result and realize the fusion control of multimodal features; W_g∈R^(d×3) is the learnable gating matrix, d is the input feature dimension; b_g is the bias term; σ is the sigmoid function; S_s^{(i)} and S_g^{(i)} are the semantic and layout scores of the i-th candidate respectively; RankScore^{(i)} is the ranking score of the i-th candidate, which is used to measure the matching degree between the retrieval result and the query; FinalResult is the gated weighted fusion feature vector, which is obtained by performing the Hadamard product of the original features [S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}] with the gate vector G_gate, and multiplying them element by element, and is used to generate a ranked list or calculate the comprehensive score of the retrieval result.

[0201] The generation process of the final search results includes an anomaly detection mechanism, which triggers manual review when the difference D>θ of the three-level search results:

[0202] D=(|S_s-S_g| / (S_s+S_g+ε)+|S_s-RankScore| / (S_s+RankScore+ε)+|S_g-RankScore| / (S_g+RankScore+ε)) / 3,

[0203] Where ε is a small constant to prevent division by zero, θ is the difference threshold, and an interpretability difference report is returned when an anomaly is detected.

[0204] S4. Implement a multi-expert collaborative verification mechanism: The rule engine, large language model, and logical reasoning module collaborate to complete the verification; including the following steps:

[0205] S4.1: Build a collaborative verification system that includes a rule engine, a large language model, and a logical reasoning module. Each module integrates the results through a dynamic arbitration mechanism.

[0206] S4.2: The rule engine performs timeliness check: Based on the timeliness rule set in the dynamic knowledge base:

[0207] R={r_i|i∈N},

[0208] Create a verification function:

[0209] f_r(x)=Σw_i·σ(x·r_i),

[0210] Among them, w_i is the rule weight and σ is the Sigmoid activation function.

[0211] The weight update of the aging rule satisfies:

[0212] w_i(t)=w_i0·e^(-λt),

[0213] Where λ is the time-dependent attenuation coefficient, and t is the rule-effective time interval.

[0214] S4.3: The large language model performs semantic association analysis: Generate a semantic feature vector h = Φ(text) ⊕ Φ(image_caption) through the pre-trained language model Φ, and establish a semantic confidence score s = softmax(W·h+b).

[0215] Semantic association analysis includes an attention enhancement mechanism: establishing a cross-modal attention matrix A = QK^T / √d, where Q is the text feature matrix, K is the image feature matrix, and d is the dimension scaling factor.

[0216] S4.4: The logical reasoning module verifies the rationality of business logic: Construct a business rule graph G_rule = (V, E), implement logical constraint propagation through a graph neural network, and output the probability of logical compliance:

[0217] p=GNN(G_rule,node_feature).

[0218] S4.5: Establish a dynamic arbitration mechanism of rule engine + large language model + logical reasoning: When the difference Δ>δ_th output by each module, the arbitration decision D = argmax(η·f_r+θ·s+ι·p) is triggered, where the difference is calculated as: Δ = 1-min(f_r,s,p) / max(f_r,s,p), δ_th is the preset difference threshold, and the dynamic adjustment coefficient satisfies η, θ, ι are the weights of the rule engine, large language model, and logical reasoning module respectively, and η+θ+ι=1.

[0219] S5. Implement a dynamic knowledge update mechanism to automatically trigger incremental learning of the knowledge base when a policy change or format update is detected; the implementation of the dynamic knowledge update mechanism includes the following steps:

[0220] S5.1 Change Detector: A dual-channel monitoring architecture is used to monitor the policy document version number and the distribution of bill format sample features in real time. When a version number update or a format feature offset exceeding a threshold is detected, the incremental learner is activated to perform a knowledge base update. The dual-channel monitoring architecture includes:

[0221] 1) Policy channel: Parse the DOM tree structure of government announcement webpages and capture version number revision patterns;

[0222] 2) Layout channel: Calculate the SSIM structural similarity index of the newly received bill image;

[0223] 3) When any channel triggers an exception, the incremental update process is started;

[0224] S5.2 Incremental Learner: Uses an elastic weight fixation algorithm to update knowledge base parameters while retaining important parameters of historical knowledge. The loss function constructed by the elastic weight fixation algorithm is:

[0225] L(θ)=L_task(θ)+λΣ_i F_i(θ_i-θ_{old,i}) 2 ;

[0226] Among them, F_i is the diagonal term of the Fisher information matrix of the historical task parameters, and λ=0.5 is the elastic constraint coefficient.

[0227] The updating of the knowledge base parameters adopts a two-stage mechanism, including:

[0228] Freeze phase: lock the convolutional feature extraction layer parameters and only update the fully connected classification layer;

[0229] Unfreezing stage: When the classification accuracy drops by more than 5%, the convolutional layer parameters are unfrozen and global fine-tuning is performed.

[0230] S5.3 Version Controller: Maintains the parallel retrieval capability for multiple versions of knowledge and implements version routing based on the timeliness of retrieval requests. The version controller implements the following when implementing parallel retrieval of multiple versions:

[0231] 1) Construct a version routing matrix M∈R^{n×d}, where n is the number of versions and d is the time-sensitive feature dimension;

[0232] 2) Calculate the temporal similarity s = softmax(qM^T) between the query vector q and each version vector;

[0233] 3) The version with s_max>0.7 is selected as the main retrieval channel, and the remaining versions are used as auxiliary channels.

[0234] S6. Output structured data and simultaneously generate an auditable traceability chain containing the decision path, including the following steps:

[0235] S6.1 While outputting structured data, it also constructs an auditable traceability chain Γ, where:

[0236] Γ={N_i|i∈[1,n]}, each node N_i=(E_i,R_i,H_i), a traceability node containing the triple verification elements of elements, rules, and fingerprints:

[0237] a) Bill element E_i∈{T_j,I_k,L_m}, where T_j is the text feature vector, I_k is the image feature matrix, and L_m is the layout feature tensor;

[0238] b) Decision rule R_i∈{Ψ_sem,Ψ_lay,Ψ_time}, Ψ_sem is the semantic level retrieval function,

[0239] is the layout convolution kernel, Ψ_time=σ(t·w_t) is the timeliness weight function;

[0240] The generation method of decision rule R_i includes:

[0241] 1) Extracting multi-expert collaborative verification results

[0242] 2) Calculate the rule activation α = Softmax(W_c·C);

[0243] 3) Determine the dominant rule R_i = argmax{α_sem,α_lay,α_time};

[0244] c) Hash fingerprint H_i = Hash(E_i||R_i||t_i), where t_i is the timestamp;

[0245] S6.2 Dynamically generate traceability chain association topology to meet the following requirements:

[0246] Make

[0247] Form a decision dependency graph G_decision = (Γ, ε);

[0248] The steps for constructing the decision dependency graph G_decision include:

[0249] 1) Create a cross-modal association matrix A = ReLU(T·W_TI·I^T+L·W_LL·L^T);

[0250] 2) Define the dependency edge ε_ij = 1 if and only if A_{ij}>μ·max(A), where μ∈(0,1) is the dynamic association threshold;

[0251] S6.3 implements a double encryption mechanism and performs the following on the traceability chain:

[0252]

[0253] Among them, K_sym is the symmetric key and K_priv is the private key;

[0254] S6.4 Reverse verification mechanism for deconvolution reconstruction: The digital signature and gradient sensitivity are verified layer by layer through the traceability parser. The verification conditions are:

[0255]

[0256] Among them, Verify(Γ) is the overall legitimacy judgment of the traceability chain Γ, which is true if and only if all levels are verified. n is the total number of layers in the traceability chain, indicating the number of steps from the current data block back to the source. Verify_Sig(H_i') is the digital signature verification result of the hash H_i' of the data block at the i-th layer. is the gradient sensitivity of the reconstruction error of the i-th layer to the deconvolution result Recon{i-1} of the previous layer, and θ is the gradient sensitivity threshold, which is used to determine the risk of data tampering;

[0257] The reverse verification includes:

[0258] 1) Implement chained deconvolution verification: DeConv(Γ) = f^{-1}(W_d [H_1'; ...; H_n']^T);

[0259] 2) Output reconstruction error matrix Δ = ||X_orig - X_recon||_F;

[0260] 3) When Δ<ε, it is determined to be a valid audit chain, where ε=σ(n·logδ), and δ is the independent error parameter.

[0261] The above describes in detail a multimodal bill processing method based on dynamic knowledge enhancement, as provided by the examples of the present invention. The description of the above embodiments is intended only to facilitate understanding of the method and core concept of the present invention. Furthermore, those skilled in the art will appreciate that variations in the specific implementation and scope of application may occur based on the concepts of the present invention. Therefore, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A multimodal bill processing method based on dynamic knowledge enhancement, characterized by: Including steps: S1. Construction of a dynamic time-sensitive knowledge graph: Build a dynamic knowledge base containing time-sensitive weights; S2. Multimodal spatiotemporal correlation modeling: A multimodal feature fusion network is used to simultaneously process the text, image, and layout features of the bill to generate a composite feature vector. S3. Three-level fusion search engine: This engine conducts three-level fusion search based on composite feature vectors, focusing on semantic, format, and timeliness. This engine includes dynamic weighted ranking with timeliness decay, a manual review mechanism triggered by difference, and a policy-sensitive slope adjustment algorithm. S4. Implement a multi-expert collaborative verification mechanism: the rule engine, large language model, and logical reasoning module collaborate to complete verification; S5. Implement a dynamic knowledge update mechanism to automatically trigger incremental learning of the knowledge base when policy changes or format updates are detected; S6. Output structured data and simultaneously generate an auditable traceability chain including the decision path.

2. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The specific operation method of step S1 is: S1.1 Multi-source heterogeneous knowledge acquisition: Use distributed crawling technology to collect bill specification texts, industry standard documents, and historical identification records to build cross-modal knowledge units that include semantic features, image features, and business logic; S1.2 Establish a timeliness scoring model based on dual-channel heterogeneous feature fusion: This includes a structured business rule feature channel, an unstructured policy text channel, a spatiotemporal feature fusion module, and a timeliness weight fusion module. This model evaluates timeliness based on the authority of the knowledge unit source, industry compliance requirements, and timeliness sensitivity, generating the initial timeliness weight w0 for the knowledge unit. S1.3 Construct a dynamic decay factor mechanism with access frequency compensation: attach a time decay function to each knowledge unit: η(t)=e^(-λΔt), Among them, λ is the industry sensitivity coefficient, Δt is the interval between the current time and the knowledge effective time; Generate real-time decay weights: w t =w0×η(t)+μ×log(N_access+1), Where μ is the adjustment coefficient, which is used to control the impact of access frequency on the overall weight. It is usually calibrated through experiments based on business needs. N_access represents the number of times a knowledge unit is accessed during the statistical period (such as the number of times the bill recognition service is called or the number of times a knowledge node is queried). S1.4 Build a version identification system for blockchain evidence storage: Use a layered structured encoding method to generate a 29-bit version identifier. The encoding rules are: semantic prefix layer + time feature layer + security verification layer. Through blockchain evidence storage, an unalterable version traceability chain is formed, supporting version backtracking and conflict detection during multimodal bill verification. S1.5 Build a dual-trigger incremental update mechanism for the knowledge base: Set up a dual-trigger mechanism (external compliance + internal attenuation). When changes in industry regulations are detected or the timeliness weight of a single knowledge unit decays to the threshold θ, the incremental version iteration of the knowledge base is started.

3. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The specific operation method of step S2 is: S2.1: Construct a parallel heterogeneous feature extraction network to obtain the text features T∈R^{d_t}, image features I∈R^{d_i}, and layout features L∈R^{d_l} of the bill through the text encoder, image encoder, and layout encoder respectively, where {d_t}, {d_i}, and {d_l} represent the dimensions of the text, image, and layout feature vectors, respectively; S2.2: Implement cross-modal feature alignment and establish a shared semantic space mapping; map the features of each modality to a unified dimensional space through the learnable projection matrix text modality projection matrix W_t∈R^{d_t×d}, image modality projection matrix W_i∈R^{d_i×d}, and layout modality projection matrix W_l∈R^{d_l×d}: T′=LayerNorm(W_t·T); I' = LayerNorm(W_i·I); L' = LayerNorm(W_l·L); In the above formula, T' is the text feature in the unified semantic space obtained by linearly transforming the original text feature matrix T to dimension d through the learnable projection matrix W_t and then normalizing it through the LayerNorm layer; I' is the original image feature matrix I, which is linearly transformed to dimension d through the learnable projection matrix W_i and then normalized through the LayerNorm layer to obtain the image feature in the unified semantic space; L' is the original layout feature matrix L, which is linearly transformed to dimension d through the learnable projection matrix W_l and then normalized through the LayerNorm layer to obtain the layout feature in the unified semantic space; LayerNorm is a layer normalization operation, which standardizes the projected features to make their mean 0 and variance 1, eliminating the differences in feature distribution between modalities and improving the stability of model training; S2.3: Perform cross-modal attention fusion with orthogonal constraints to complete interpretable feature synthesis with gradient scale regularization; use a multi-head cross-modal attention mechanism to calculate cross-modal (text to image and layout features) feature correlation: α=Softmax((Q·K^T) / √d), Where Q = T' is the text feature in dimension d, K = Linear_Projection(Concat(I',L')), which is the key matrix after the image and layout features are concatenated and projected to d dimensions via linear transformation, where Linear_Projection is a learnable linear transformation layer; Generate a composite feature vector by adaptive weight fusion: F=α·V+β_text·T', Among them, α·V represents the weighted attention result of text features and image / layout features, where V = T'; β_text is a learnable dynamic gating parameter that controls the retention strength of text features, and β_text·T' is a residual connection that retains the original text features. The constraint condition is: β_text∈[0,1], which is generated by the Sigmoid function.

4. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The three-level fusion search described in step S3 matches the bill features. The specific operation steps include: S3.1: Semantic level retrieval: Match the text semantics of the field, map the composite feature vector Φ to the semantic space Ω through the semantic encoder E_s, and calculate the semantic similarity with the knowledge unit k_i in the dynamic knowledge base K: S_s(Φ,k_i)=cos(E_s(Φ),E_s(Ψ_i))·α_t, Among them, S_s(Φ,k_i) is the semantic similarity score between the composite feature vector Φ and the i-th knowledge unit k_i in the knowledge base. Its value consists of two parts: semantic space similarity: the matching degree between the vector E_s(Φ) mapped by the semantic encoder E_s and E_s(Ψ_i) is calculated by cosine similarity, reflecting the degree of alignment between the two in the semantic space Ω, where Ψ_i is the feature vector of the knowledge unit; timeliness weight α_t: a dynamic attenuation factor used to dynamically adjust the time sensitivity of the knowledge unit, satisfying α_t = max(0,1-λ·Δt), where λ∈(0,0.1] is the attenuation coefficient, Δt is the knowledge update interval, and the timeliness weight value decreases as the knowledge update interval Δt increases (λ controls the attenuation rate), ensuring that the retrieval results take into account both semantic relevance and timeliness; S3.2: Layout-level retrieval: Compare the feature space layout, extract the bill layout feature vector Φ_g, construct the layout topology graph G_layout = (V, E), and calculate the layout similarity with the knowledge base template T_j through the graph topology matching algorithm: S_g(T_j)=∑_{v∈V}w(v)·δ(l(v),l'(v'_v))+γ·|E∩E'| / |E∪E'|, Among them, S_g(T_j) is the layout similarity score between the current bill layout topology graph G_layout and the knowledge base template T_j, which is composed of node matching weight (calculating local matching through node label consistency and spatial weight) and edge structure similarity (measuring the global alignment of topological structures through edge set intersection ratio); {v∈V} is the set of all nodes in the current bill layout topology graph G_layout, each node v corresponds to a layout element in the bill (such as title field, table area, QR code area), and V is the set of current bill layout elements; w(v) is the node spatial weight; δ(·) is the label matching function; l(v) is the semantic label of the current bill node v; l'(v'_v) is the template The corresponding node in T_j that is aligned with the current bill node v (alignment rule: v'_v is determined by label matching and spatial position similarity); γ∈[0.2,0.5] is the edge matching coefficient; |E∩E'| / |E∪E'| is the Jaccard similarity coefficient of the edge, which measures the overlap ratio of the two graphs in the topological structure, and the value range is [0,1]. The larger the value, the more similar the topological structure. Among them, E is the edge set in the current bill layout topology graph, which represents the connection relationship between layout elements (such as adjacent relationship, hierarchical relationship); E′ is the edge set in the template T_j layout topology graph, the numerator |E∩E′| is the number of edges shared by the two graphs (topological structure overlap), and the denominator |E∪E′| is the union number of all edges in the two graphs; S3.3 Time-sensitive retrieval: including dynamic weighted sorting with time decay: screening knowledge records within the validity period, establishing a time-sensitive weight matrix W = [α_1, α_2, ..., α_n]^T, where α_i is the time-sensitive weight of the i-th knowledge unit, satisfying α_i = 1-λ·Δt_i; dynamic weighted sorting of search results based on three-level mixed similarity: RankScore=S_s⊙W+S_g⊙W+β·f(t_c,t_e), Among them, RankScore is the dynamic weighted ranking score of the knowledge retrieval result; ⊙ is the Hadamard product, which represents the element-by-element multiplication of the vector; S_s is the semantic similarity score vector, which is generated by the semantic-level retrieval in S3.1; S_s⊙W is the semantic similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the semantic similarity and the time weight, reflecting the time correction value of the semantic matching; S_g is the layout similarity score vector, which is generated by the layout-level retrieval in S3.2, and S_g⊙W is the layout similarity time decay term, which represents the Hadamard product (element-by-element multiplication) of the layout similarity and the time weight. element-wise multiplication), reflecting the timeliness correction value of layout matching; β·f(t_c,t_e) is the timeliness decay function term, which is a nonlinear decay compensation term based on the validity period of knowledge. β∈[0.3,0.6] is the timeliness coefficient, which controls the contribution weight of the timeliness decay function to the total score. The larger the value, the stronger the influence of timeliness on the ranking; f(t_c,t_e)=1 / (1+exp(k(t_e-t_c))) is the timeliness decay function, which is used to smoothly adjust the score of records approaching expiration. t_c is the current time, t_e is the expiration time of the knowledge record, and k is the adjustment slope. S3.4 builds a three-level retrieval fusion model and generates the final retrieval results through a gating mechanism: G_gate=σ(W_g·[S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}]+b_g), FinalResult=[S_s^{(i)};S_g^{(i)};RankScore^{(i)}]⊙G_gate, Among them, G_gate is the gating vector, which is used to dynamically adjust the contribution of different features to the final result and realize the fusion control of multimodal features; W_g∈R^(d×3) is the learnable gating matrix, d is the input feature dimension; b_g is the bias term; σ is the sigmoid function; S_s^{(i)} and S_g^{(i)} are the semantic and layout scores of the i-th candidate respectively; RankScore^{(i)} is the ranking score of the i-th candidate, which is used to measure the matching degree between the retrieval result and the query; FinalResult is the gated weighted fusion feature vector, which is obtained by performing the Hadamard product of the original features [S_s^{(i)}; S_g^{(i)}; RankScore^{(i)}] with the gate vector G_gate, and multiplying them element by element, and is used to generate a ranked list or calculate the comprehensive score of the retrieval result.

5. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 4, characterized in that: The semantic encoder E_s described in step 3.1 adopts a dual-tower structure based on contrastive learning, and its loss function is: L_c=-log[exp(s(Φ,Ψ^+) / τ) / (exp(s(Φ,Ψ^+) / τ)+∑_{Ψ^-}exp(s(Φ,Ψ^-) / τ))]; Among them, τ is the temperature hyperparameter, Ψ^+ is the positive sample, Ψ^- is the negative sample, and s(·) is the similarity calculation function.

6. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 4, characterized in that: The node space weight w(v) of the layout topology graph G_layout described in step 3.2 is calculated as follows: w(v)=1 / (1+min(d(v,C),d(v,M))·σ_A), Where d(v,C) is the normalized distance from the node to the bill center, d(v,M) is the distance from the node to the nearest page edge, and σ_A∈[0.1,0.3] is the regional sensitivity coefficient.

7. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The construction of the multi-expert collaborative verification system described in step S4 includes the following steps: S4.1: Build a collaborative verification system that includes a rule engine, a large language model, and a logical reasoning module. Each module integrates the results through a dynamic arbitration mechanism. S4.2: The rule engine performs timeliness check: Based on the timeliness rule set in the dynamic knowledge base: R={r_i|i∈N}, Create a verification function: f_r(x)=Σw_i·σ(x·r_i), Among them, w_i is the rule weight, σ is the Sigmoid activation function; S4.3: The large language model performs semantic association analysis: Generate a semantic feature vector h = Φ(text) ⊕ Φ(image_caption) through the pre-trained language model Φ, and establish a semantic confidence score s = softmax(W·h+b); S4.4: The logical reasoning module verifies the rationality of business logic: Construct a business rule graph G_rule = (V, E), implement logical constraint propagation through a graph neural network, and output the probability of logical compliance: p=GNN(G_rule,node_feature); S4.5: Establish a dynamic arbitration mechanism of rule engine + large language model + logical reasoning: When the difference Δ>δ_th output by each module, the arbitration decision D = argmax(η·f_r+θ·s+ι·p) is triggered, where the difference is calculated as: Δ = 1-min(f_r,s,p) / max(f_r,s,p), δ_th is the preset difference threshold, and the dynamic adjustment coefficient satisfies η, θ, ι are the weights of the rule engine, large language model, and logical reasoning module respectively, and η+θ+ι=1.

8. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The semantic association analysis described in step S4.3 includes an attention enhancement mechanism: establishing a cross-modal attention matrix A = QK^T / √d, where Q is the text feature matrix, K is the image feature matrix, and d is the dimensionality scaling factor.

9. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The implementation of the dynamic knowledge update mechanism described in step S5 includes the following steps: S5.1 Change Detector: This uses a dual-channel monitoring architecture to monitor the policy document version number and the distribution of bill format sample features in real time. When a version number update or a format feature offset exceeding a threshold is detected, the incremental learner is activated to perform a knowledge base update. The dual-channel monitoring architecture includes: 1) Policy channel: Parse the DOM tree structure of government announcement webpages and capture version number revision patterns; 2) Layout channel: Calculate the SSIM structural similarity index of the newly received bill image; 3) When any channel triggers an exception, the incremental update process is started; S5.2 Incremental Learner: Uses an elastic weight fixation algorithm to update knowledge base parameters while retaining important parameters of historical knowledge. The loss function constructed by the elastic weight fixation algorithm is: L(θ)=L_task(θ)+λΣ_i F_i(θ_i-θ_{old,i})2; Where F_i is the diagonal term of the Fisher information matrix of the historical task parameters, and λ = 0.5 is the elastic constraint coefficient; The updating of the knowledge base parameters adopts a two-stage mechanism, including: Freeze phase: lock the convolutional feature extraction layer parameters and only update the fully connected classification layer; Unfreezing stage: When the classification accuracy drops by more than 5%, the convolutional layer parameters are unfrozen and global fine-tuning is performed; S5.3 Version Controller: Maintains the parallel retrieval capability for multiple versions of knowledge and implements version routing based on the timeliness of retrieval requests. The version controller implements the following when implementing parallel retrieval of multiple versions: 1) Construct a version routing matrix M∈R^{n×d}, where n is the number of versions and d is the time-sensitive feature dimension; 2) Calculate the temporal similarity s = softmax(qM^T) between the query vector q and each version vector; 3) The version with s_max>0.7 is selected as the main retrieval channel, and the remaining versions are used as auxiliary channels.

10. The multimodal bill processing method based on dynamic knowledge enhancement according to claim 1, characterized in that: The method for generating the auditable traceability chain in step S6 includes the following steps: S6.1 While outputting structured data, it also constructs an auditable traceability chain Γ, where: Γ={N_i|i∈[1,n]}, each node N_i=(E_i,R_i,H_i), a traceability node containing the triple verification elements of elements, rules, and fingerprints: a) Bill element E_i∈{T_j,I_k,L_m}, where T_j is the text feature vector, I_k is the image feature matrix, and L_m is the layout feature tensor; b) Decision rule R_i∈{Ψ_sem,Ψ_lay,Ψ_time}, Ψ_sem is the semantic level retrieval function, is the layout convolution kernel, Ψ_time=σ(t·w_t) is the timeliness weight function; The generation method of decision rule R_i includes: 1) Extracting multi-expert collaborative verification results 2) Calculate the rule activation α = Softmax(W_c·C); 3) Determine the dominant rule R_i = argmax{α_sem,α_lay,α_time}; c) Hash fingerprint H_i = Hash(E_i||R_i||t_i), where t_i is the timestamp; S6.2 Dynamically generate traceability chain association topology to meet the following requirements: Make Form a decision dependency graph G_decision = (Γ, ε); The steps for constructing the decision dependency graph G_decision include: 1) Create a cross-modal association matrix A = ReLU(T·W_TI·I^T+L·W_LL·L^T); 2) Define the dependency edge ε_ij = 1 if and only if A_{ij}>μ·max(A), where μ∈(0,1) is the dynamic association threshold; S6.3 implements a double encryption mechanism and performs the following on the traceability chain: Among them, K_sym is the symmetric key and K_priv is the private key; S6.4 Reverse verification mechanism for deconvolution reconstruction: The digital signature and gradient sensitivity are verified layer by layer through the traceability parser. The verification conditions are: Among them, Verify(Γ) is the overall legitimacy judgment of the traceability chain Γ, which is true if and only if all levels are verified. n is the total number of layers in the traceability chain, indicating the number of steps from the current data block back to the source. Verify_Sig(H_i') is the digital signature verification result of the hash H_i' of the data block at the i-th layer. is the gradient sensitivity of the reconstruction error of the i-th layer to the deconvolution result Recon{i-1} of the previous layer, and θ is the gradient sensitivity threshold, which is used to determine the risk of data tampering; The reverse verification includes: 1) Implement chained deconvolution verification: DeConv(Γ) = f^{-1}(W_d [H_1'; ...; H_n']^T); 2) Output reconstruction error matrix Δ = ||X_orig - X_recon||_F; 3) When Δ<ε, it is determined to be a valid audit chain, where ε=σ(n·logδ), and δ is the independent error parameter.

Images