A method and related device for generating a patrol document in compliance with a fusion of text and picture alignment and a spatiotemporal knowledge graph

Abstract

This invention provides a method and related apparatus for generating compliant inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs, belonging to the field of inspection and supervision work. First, this invention employs multimodal unified representation technology to convert heterogeneous data into a unified representation that combines general semantic information with inspection and supervision-specific business attributes. Second, it achieves the association and traceability of text descriptions and image evidence through text-image semantic alignment. Then, it automatically matches the optimal dedicated expert model based on the business characteristics of the input data. Subsequently, by constructing a dynamic spatiotemporal knowledge graph and introducing state transition legality verification rules, it achieves full-cycle closed-loop management and accountability traceability. Finally, it generates documents based on standardized structured factual data output from the knowledge graph, driven by a large language model, and strictly constrains the generated content through a factual consistency verification mechanism. This invention can effectively meet the practical needs of intelligent operation throughout the entire inspection and supervision process.

Description

Technical Field

[0001] This invention belongs to the field of inspection and supervision work, and specifically relates to a method and related apparatus for generating inspection documents in compliance with regulations by integrating text and image alignment with spatiotemporal knowledge graphs. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the deep application of large-scale models and multimodal learning, traditional manual inspection and supervision work is gradually transforming towards digitalization and intelligence. The intelligent analysis system for inspection and supervision has evolved from a single text processing mode to multimodal fusion. Its core objective is to efficiently process various heterogeneous data such as policies and regulations, historical inspection reports, and rectification documents, achieving intelligent processing of the entire process of problem identification, key point analysis, and rectification tracking, thus providing technical support for inspection and supervision work.

[0003] The inspection and supervision business scenario has significant unique characteristics, differing markedly from general document processing scenarios: document types are heterogeneous and diverse, encompassing various formats such as scanned documents, tables, handwritten annotations, and official documents, with complex layouts and significant OCR recognition noise; the business demands strict accountability, requiring clear identification of the responsible party for each issue, the workflow, and timelines to form a complete closed-loop chain of evidence; document generation must meet strong compliance and anti-illusion requirements, with output inspection reports, problem lists, and rectification suggestions strictly anchored to traceable factual evidence to prevent misallocation of responsibility or deviation from facts. Meanwhile, while general technologies such as image-text semantic alignment, multimodal routing, and knowledge graphs provide basic support for intelligent analysis in inspection and supervision, these technologies lack specific improvements tailored to the inspection and supervision scenario and cannot directly adapt to its business needs.

[0004] The main problem with existing intelligent analysis technologies for inspection and supervision is that they rely heavily on simple concatenation of single-modal models or general AI algorithms. They fail to deeply integrate the business rules specific to the inspection and supervision scenario, such as accountability, status transition, and compliance constraints, into the technical architecture. As a result, the overall technical solution cannot adapt to the special characteristics of the inspection and supervision scenario, making it difficult to achieve efficient processing of multi-source heterogeneous data, accountability association, and compliance document generation. Consequently, it cannot meet the actual needs of intelligentization of the entire inspection and supervision process. Summary of the Invention

[0005] Based on this, it is necessary to provide a method and related device for generating inspection documents in compliance with regulations by integrating text-image alignment and spatiotemporal knowledge graphs, in order to solve the problem of insufficient adaptability of existing technologies and meet the actual needs of intelligent inspection and supervision throughout the entire process.

[0006] In a first aspect, the present invention provides a method for generating compliance inspection documents by integrating image-text alignment and spatiotemporal knowledge graphs, comprising the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0007] Furthermore, the multi-source heterogeneous raw data obtained from inspections and audits are represented in a unified multimodal manner to generate a unified multimodal representation carrying business attributes, including: Text encoders, visual encoders, and table encoders are used to extract text semantic features, image visual features, and table structure features from multi-source heterogeneous raw data, respectively, and these features are used to construct multimodal semantic features. The structured auxiliary information of multi-source heterogeneous raw data is encoded into auxiliary information features; Multimodal semantic features are fused with auxiliary information features to generate a unified multimodal representation.

[0008] Furthermore, the unified multimodal representation is semantically aligned with text to generate joint text-image features with alignment confidence, including: Extracting text and image features from a unified multimodal representation; Based on text features and image features, basic semantic similarity is calculated using a pre-constructed similarity calculation function; Based on basic semantic similarity, the basic semantic alignment loss is calculated through a pre-constructed contrastive learning loss function, and the business constraint loss is calculated based on a pre-constructed anchor constraint loss function. The anchor constraint loss function is used to calculate the loss when violating institutional clause constraints, evidence association constraints, and responsible entity constraints. Using the sum of basic semantic alignment loss and business constraint loss as the overall optimization objective, semantic alignment optimization is performed on the unified multimodal representation to generate joint image and text features with alignment confidence.

[0009] Furthermore, based on the joint image and text features with alignment confidence, the business characteristics of the multi-source heterogeneous raw data are determined. An expert model optimally suited to these business characteristics is selected for targeted business processing, resulting in structured business processing results, including: A business state vector is constructed based on the combined image and text features with alignment confidence and auxiliary information from multi-source heterogeneous raw data; the business state vector is used to characterize the business features of multi-source heterogeneous raw data. Based on the business state vector, the matching probability of each expert model is calculated through a reinforcement learning policy network; the expert model whose matching probability meets the preset matching probability condition is taken as the optimal matching expert model. The corresponding business tasks are processed in a targeted manner by using the best-fit expert model, and the business processing results of each expert model are obtained. Integrate all business processing results to generate structured business processing results that include a set of issues, a set of responsibilities, and a set of evidence.

[0010] Furthermore, the structured business processing results are mapped to nodes and edges of a knowledge graph, constructing a spatiotemporal knowledge graph containing rectification status attributes, including: Based on the problem set, responsibility set, and evidence set in the structured business processing results, instantiate knowledge graph nodes and generate relation edges; Add a rectification status attribute to each node; the rectification status attribute includes rectification stage, timestamp, time limit node, and responsibility transfer record; Add a relationship attribute to each edge; the relationship attribute includes the relationship type, the time the relationship occurred, and the rectification state transition; A spatiotemporal knowledge graph containing rectification status attributes is constructed based on nodes and edges.

[0011] Furthermore, the dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured factual data, including: During the evolution and update of the spatiotemporal knowledge graph, the legality of candidate rectification state transitions is verified through state transition legality verification rules. If the rectification state transition conforms to the preset rules, the evolution and update of the spatiotemporal knowledge graph is allowed. If the rectification state transition violates the preset rules, the evolution and update of the spatiotemporal knowledge graph is prevented and a manual intervention review process is triggered. The evolved spatiotemporal knowledge graph is input into a pre-constructed temporal graph neural network to update the node representations of the spatiotemporal knowledge graph; The rectification risk score of each problem entity in the updated spatiotemporal knowledge graph is calculated using a pre-built rectification risk scoring function. When the rectification risk score exceeds a preset threshold or the rectification status remains unchanged for a preset duration, an early warning message is generated. Integrate issues, evidence, responsibilities, rectification status, and early warning information from the spatiotemporal knowledge graph to generate structured factual data.

[0012] Furthermore, based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated, including: Based on structured fact data, a prompt template is generated using a pre-built prompt template constructor. Input the prompt template into the pre-built large language model to generate the first draft of the document.

[0013] Secondly, the present invention provides a compliance generation device for inspection documents that integrates image-text alignment and spatiotemporal knowledge graphs, comprising: The multimodal unified representation module is used to perform multimodal unified representation on the multi-source heterogeneous raw data obtained from inspections and patrols, and generate a unified multimodal representation. The image-text semantic alignment module is used to perform image-text semantic alignment on the unified multimodal representation and generate joint image-text features with alignment confidence. The dynamic adaptive multimodal routing module is used to determine the business characteristics of multi-source heterogeneous raw data based on the joint features of text and images with alignment confidence; select the expert model that best matches the business characteristics for targeted business processing, and obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses; The spatiotemporal knowledge graph construction and rectification evaluation module is used to map structured business processing results into nodes and edges of the knowledge graph, constructing a spatiotemporal knowledge graph containing rectification status attributes; constraining the dynamic evolution process of the spatiotemporal knowledge graph through state transition legality verification rules, and performing rectification closed-loop evaluation based on the evolved spatiotemporal knowledge graph to obtain structured factual data; rectification status attributes are used to describe the direction of evolution, and state transition legality verification rules are used to constrain the legality of evolution; The inspection document generation module is used to determine the results, rectification status, and early warning information of the inspection based on structured factual data, and to generate a draft of the inspection document; the draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0016] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0017] In summary, this invention provides a method and related apparatus for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs. First, it employs multimodal unified representation technology to convert heterogeneous data such as policy texts, scanned images, and audit forms—which are diverse in format, complex in structure, and noisy—into a unified representation that possesses both general semantic information and inspection-specific business attributes. Based on this, text-image semantic alignment enables the association and traceability of textual descriptions and image evidence. Furthermore, it adopts a reinforcement learning-based dynamic adaptive multimodal routing mechanism to automatically match the optimal dedicated expert model according to the business characteristics of the input data, replacing the traditional fixed-path processing method and significantly improving the processing efficiency of different types of business materials, such as text-dominated, image-dominated, and table-dominated documents. The system improves the efficiency of material processing and the accuracy of tasks. Subsequently, by constructing a dynamic spatiotemporal knowledge graph and introducing state transition legality verification rules, the unique rectification closed-loop management rules of inspection and supervision are explicitly embedded into the technical architecture. This achieves full-cycle closed-loop management and accountability tracing from problem discovery, responsibility assignment, rectification promotion to review and cancellation, and rebound warning. Finally, the system generates documents based on standardized structured factual data output from the knowledge graph, driven by a large language model. The generated content is strictly constrained by a factual consistency verification mechanism, which fundamentally avoids factual deviations, responsibility mismatches, and compliance risks caused by the free generation of general large models. This systematically solves the problem that existing technologies cannot adapt to the special characteristics of inspection and supervision scenarios and meets the actual needs of intelligent inspection and supervision throughout the entire process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs, as described in one embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the principle of image-text semantic alignment in one embodiment of the present invention; Figure 3 This is a schematic diagram of a knowledge graph and a rectification closed-loop assessment in one embodiment of the present invention; Figure 4 This is a schematic diagram of the document generation consistency verification process in one embodiment of the present invention; Figure 5 This is a flowchart illustrating the implementation of the compliance generation method for inspection documents according to the present invention. Figure 6 This is a structural block diagram of a compliance generation device for inspection documents that integrates image-text alignment and spatiotemporal knowledge graph in one embodiment of the present invention. Figure 7 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] It should be noted that the user information (including but not limited to user images, user portrait information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0022] Please see Figure 1 This invention provides a method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs, comprising the following steps: S101: Perform multimodal unified representation on the multi-source heterogeneous raw data obtained from inspections and patrols, and generate a unified multimodal representation.

[0023] Among them, multi-source heterogeneous raw data refers to various materials in the inspection and supervision business that have diverse formats, complex layouts, discrete evidence distributions, and high OCR recognition noise, including but not limited to policy and institutional texts, inspection and supervision reports, rectification ledgers, audit attachments, scanned images, and tabular data; multimodal unified representation refers to the process of converting raw data of different types and structures into feature vectors with unified dimensions and semantic space, while retaining business-related auxiliary information.

[0024] It should be noted that, firstly, the various types of raw input data are preprocessed, then basic semantic features are extracted by encoders of the corresponding modalities, then business-related structured auxiliary information is encoded into feature vectors, and finally, the multimodal semantic features and auxiliary information features are fused to obtain a unified multimodal representation.

[0025] S102: Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence.

[0026] Among them, image-text semantic alignment refers to the process of establishing semantic correspondence between text fragments and image regions to solve the problem of cross-modal semantic bias; alignment confidence refers to a numerical value that quantifies the degree of semantic matching between text and image, and the value range is usually [0,1]. The higher the value, the higher the matching degree; image-text joint features refer to a unified feature vector that integrates text semantic information and image visual information.

[0027] It should be noted that, firstly, text features and image features are separated from the unified multimodal representation, then the basic semantic similarity between the two is calculated, and then a total loss function is constructed in combination with the constraints specific to the business scenario. Semantic alignment is achieved by optimizing this loss function, and the alignment confidence is output. Finally, the image-text joint features with confidence are obtained.

[0028] S103: Determine the business characteristics of multi-source heterogeneous raw data based on the combined features of text and images with alignment confidence; select the expert model that best matches the business characteristics for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses.

[0029] Among them, business characteristics refer to the set of features that can reflect the type, complexity, evidence completeness and risk level of the inspection and supervision business corresponding to the input data; expert models refer to specialized neural network models that are specially trained for specific tasks in inspection and supervision business, with each expert model focusing on handling a specific type of business problem; structured business processing results refer to the set of information with fixed format and clear semantics obtained after processing unstructured raw data.

[0030] It should be noted that, firstly, a state vector that can comprehensively represent business attributes is constructed based on the joint features of text and images and auxiliary information. Then, the matching probability between each expert model and the current business features is calculated through a policy network. The expert model with the highest matching probability is selected for processing. Finally, the output results of each expert model are integrated to obtain the structured business processing result.

[0031] S104: Map the structured business processing results to nodes and edges of the knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes; constrain the dynamic evolution process of the spatiotemporal knowledge graph through state transition legality verification rules, and conduct rectification closed-loop evaluation based on the evolved spatiotemporal knowledge graph to obtain structured fact data; rectification status attributes are used to describe the direction of evolution, and state transition legality verification rules are used to constrain the legality of evolution.

[0032] Among them, spatiotemporal knowledge graph refers to a knowledge graph that includes not only static information about entities and relationships, but also information about time dimensions and state evolution; rectification status attributes refer to a set of attributes used to describe the stage of a problem in the rectification closed loop and related time and responsibility information; state transition legality verification refers to the process of checking whether changes in the rectification status of a problem comply with the business management rules of inspection and supervision; rectification closed loop assessment refers to the process of assessing the progress, quality, risk and compliance of problem rectification; structured fact data refers to a set of standardized factual information extracted from the knowledge graph that can be directly used for document generation.

[0033] It should be noted that, firstly, the structured business processing results are mapped to nodes and edges of the knowledge graph, and rectification status and time attributes are added to the nodes and edges to construct a dynamic spatiotemporal knowledge graph; then, the legality of state transitions is verified when the graph is updated to ensure that the graph evolution conforms to business rules; next, time series prediction and risk assessment are performed based on the evolved graph to generate early warning information; finally, various types of information in the graph are integrated to obtain structured fact data.

[0034] S105: Based on structured factual data, determine the results, rectification status, and early warning information of the inspection and supervision, and generate a draft of the inspection document; perform factual consistency verification on the draft of the inspection document, and obtain a compliant inspection document after passing the verification.

[0035] Among them, the factual consistency verification refers to the process of checking whether the content of the generated document is consistent with the information on problems, evidence, responsibilities and rectification status in the structured factual data; compliant inspection documents refer to formal documents that are true and accurate in content, have clear responsibilities, are in a standardized format and meet the requirements of inspection and supervision business.

[0036] It should be noted that, firstly, a standardized prompt template is constructed based on structured factual data, and then the prompt template is input into a large language model to generate a draft document; next, the draft document is checked for factual consistency, and if it fails the check, the prompt template is corrected and regenerated; finally, the compliance inspection document that has passed the check is output.

[0037] This embodiment achieves end-to-end automated processing from raw heterogeneous data input to final compliant document output through progressive technologies such as multimodal data representation, graph-text semantic alignment, dynamic adaptive routing (i.e., dynamic task distribution processing), knowledge graph fusion and rectification closed-loop evaluation, and automated document generation. Unlike the simple chaining of general algorithms in existing technologies, this embodiment incorporates institutional anchors, factual anchors, responsibility anchors, state transition rules, and closed-loop supervision mechanisms unique to inspection and supervision operations at each technical stage. This enables the modules to form an organic synergy oriented towards the governance goals of inspection and supervision, rather than an isolated superposition of technical functions.

[0038] In an exemplary embodiment, multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation carrying business attributes, including: S201: By using a text encoder, a visual encoder, and a table encoder, text semantic features, image visual features, and table structure features are extracted from multi-source heterogeneous raw data, respectively, and multimodal semantic features are constructed.

[0039] Among them, a text encoder refers to a neural network model used to convert natural language text into semantic feature vectors, commonly including pre-trained language models such as BERT and RoBERTa; a visual encoder refers to a neural network model used to convert images into visual feature vectors, commonly including pre-trained visual models such as ResNet and ViT; a table encoder refers to a neural network model used to convert tabular data into structural feature vectors, commonly including Table Transformer; and multimodal semantic features refer to a feature set composed of text semantic features, image visual features, and table structural features.

[0040] For example, suppose the first i The input data for each sample is represented as follows: ; In the formula, Represents text data, Represents image data, The data represents a table. Features are extracted from the data using corresponding encoders to obtain text semantic features, image visual features, and table structure features, as follows: ; In the formula, For text encoders, For visual encoders, For a table encoder, the three types of features mentioned above together constitute multimodal semantic features.

[0041] S202: Encode the structured auxiliary information of multi-source heterogeneous raw data into auxiliary information features.

[0042] Among them, structured auxiliary information refers to non-semantic information related to the original data that can provide support for subsequent business processing. Preferably, it may include page number information, page coordinate information, OCR confidence level, evidence source identifier, and responsible entity identifier. Auxiliary information features refer to feature vectors obtained after converting the above structured auxiliary information into vector form.

[0043] For example, structured auxiliary information in inspection and patrol scenarios can be encoded into auxiliary information features: ; In the formula, Indicates page number information. Indicates page coordinate information. Indicates OCR confidence level. Indicates the source of evidence. This indicates the identifier of the responsible party; this structural feature is the auxiliary information feature.

[0044] S203: Fuse multimodal semantic features with auxiliary information features to generate a unified multimodal representation.

[0045] Feature fusion refers to the process of combining multiple feature vectors from different sources and dimensions into a unified feature vector. Common fusion methods include vector concatenation and weighted summation. Unified multimodal representation refers to the final input feature vector that integrates multimodal semantic features and auxiliary information features, which can simultaneously reflect the semantic information and business attribute information of the original data.

[0046] For example, to place different modal features in a unified semantic space, the multimodal semantic features are first linearly mapped, resulting in the following mapped text features, image features, and table features: ; In the formula, , and These are the corresponding projection matrices.

[0047] Then the mapped multimodal semantic features and auxiliary information features s i Vector concatenation is performed to generate a unified multimodal representation as follows: ; In the formula, To unify the representation of multimodalities.

[0048] This embodiment extracts semantic features from text, images, and tables respectively, and integrates business auxiliary information such as page numbers, page positions, evidence sources, and responsibility identifiers. This enables the generated unified multimodal representation to not only have general multimodal semantic representation capabilities, but also evidence location and responsibility traceability applicable to inspection and patrol scenarios.

[0049] In an exemplary embodiment, graph-text semantic alignment is performed on the unified multimodal representation to generate joint graph-text features with alignment confidence, including: S301: Extract text features and image features from the unified multimodal representation.

[0050] Among them, text features refer to the text semantic feature vector after linear mapping in the unified multimodal representation; image features refer to the image visual feature vector after linear mapping in the unified multimodal representation.

[0051] For example, from unified multimodal representation Extract the mapped text features and image features .

[0052] S302: Based on text features and image features, the basic semantic similarity is calculated using a pre-constructed similarity calculation function.

[0053] Among them, basic semantic similarity refers to the degree of matching between text and image calculated based on semantic features, without considering the special constraints of business scenarios; cosine similarity is a commonly used method in this field to calculate the similarity between two vectors.

[0054] For example, cosine similarity is used as the basic semantic similarity calculation function for text features. and image features The basic semantic similarity between them is defined as: ; In the formula, For vector dot product, L2 norm of a vector S303: Based on basic semantic similarity, the basic semantic alignment loss is calculated through a pre-constructed contrastive learning loss function, and the business constraint loss is calculated based on a pre-constructed anchor constraint loss function; the anchor constraint loss function is used to calculate the loss when violating institutional clause constraints, evidence association constraints, and responsible entity constraints.

[0055] Among them, the contrastive learning loss function is a loss function that learns feature representations by bringing positive sample pairs closer to each other and negative sample pairs further apart in the feature space; the anchor point constraint loss function is a loss function designed for the constraints of inspection and supervision business scenarios to ensure that the image-text alignment results meet business requirements; the institutional clause constraint loss is a loss used to constrain the alignment of policy and institutional clauses in the text with relevant evidence areas in the image; the evidence association constraint loss is a loss used to constrain the alignment of problem descriptions in the text with evidence materials in the image; and the responsible entity constraint loss is a loss used to constrain the alignment of responsible entities in the text with relevant identifiers in the image.

[0056] For example, the contrastive learning loss function constructed based on basic semantic similarity is as follows: ; in, N Indicates the number of samples in the batch. τ This represents the temperature parameter.

[0057] To enhance the business constraint capability of text and image alignment in inspection and supervision scenarios, an anchor point constraint loss is further introduced as follows: ; in, This indicates that the institutional provisions restrict losses. Indicates loss due to evidence association constraints. This indicates that the responsible party is bound by the loss. , and These are the weighting coefficients.

[0058] S304: Using the sum of basic semantic alignment loss and business constraint loss as the overall optimization objective, semantic alignment optimization is performed on the unified multimodal representation to generate joint image and text features with alignment confidence.

[0059] Among them, the overall optimization objective refers to the total loss function that needs to be minimized during model training; semantic alignment optimization refers to the process of adjusting model parameters through backpropagation algorithm to minimize the total loss function; and image-text joint representation refers to a unified representation that integrates aligned text and image features, while also including alignment confidence information.

[0060] For example, the overall optimization objective for image-text semantic alignment is expressed as: ; By optimizing the objective function, semantically consistent text fragments and image regions can be brought closer together in the feature space, while semantically inconsistent samples are moved further apart. After optimization, the output image-text joint representation is as follows: ; in, Indicates the combined features of text and images. This indicates the alignment confidence level.

[0061] like Figure 2 As shown, the image-text semantic alignment process in this embodiment consists of two parallel paths: one on the text side and one on the image side. Features are extracted by the text encoder and the other by the visual encoder, respectively, and then projected onto a unified semantic space. Cosine similarity is calculated, and optimization is performed using contrastive learning loss and anchor point constraint loss. The final output is a joint image-text representation and alignment confidence. Specifically, on the text side, the text encoder extracts semantic feature vectors such as policy clauses and problem descriptions; on the image side, the visual encoder extracts visual feature vectors such as scanned document layouts and evidence screenshots. Both are mapped to a unified semantic space via a projection matrix and then subjected to contrastive learning. Based on this, a triple anchor point loss is introduced, consisting of institutional clause constraints, evidence association constraints, and responsible party constraints, ensuring that the alignment result simultaneously meets the requirements of semantic consistency, evidence association, and responsibility localization, thus outputting a joint image-text representation and alignment confidence.

[0062] This embodiment introduces three types of anchor point constraint losses—institutional clauses, evidence association, and responsible parties—on top of the general contrastive learning loss. This effectively solves the cross-modal semantic bias problem caused by multi-column layout, official seal obscuration, scanning noise, annotation interference, and evidence fragmentation in inspection and supervision documents, so that the image-text matching results simultaneously satisfy semantic consistency, evidence association, and responsibility positioning.

[0063] In an exemplary embodiment, business characteristics of multi-source heterogeneous raw data are determined based on the joint features of text and images with alignment confidence. An expert model optimally suited to these business characteristics is then selected for targeted business processing, resulting in structured business processing results, including: S401: Construct a business state vector based on the combined image and text features with alignment confidence and auxiliary information from multi-source heterogeneous raw data; the business state vector is used to characterize the business features of multi-source heterogeneous raw data.

[0064] Among them, the business state vector refers to a vector that can comprehensively reflect the business attributes corresponding to the current input data.

[0065] For example, let the current input state be defined as: ; in, Indicates the combined features of text and images. Indicates alignment confidence. Indicates the completeness of evidence. Describing modal complexity, This vector represents the risk level and is the business status vector.

[0066] S402: Based on the business state vector, calculate the matching probability of each expert model through a reinforcement learning policy network; select the expert model whose matching probability meets the preset matching probability condition as the optimally matched expert model.

[0067] Among them, the reinforcement learning policy network refers to the neural network model used to output the probability distribution of actions based on the input state, and is the core of the dynamic routing mechanism; the expert model set refers to the set of special models designed for different tasks in the inspection and patrol business; the matching probability refers to the probability that a certain expert model output by the policy network is suitable for processing the current input data.

[0068] For example, the policy network is based on the business state vector Output the expert choice probability, its expression is: ; in, Indicates the expert's actions, Indicates the policy network parameters, and These are network parameters.

[0069] The expert set is defined as: ; in, Indicates an expert in evidence collection, Indicates a problem-solving expert, Indicates an expert in chain of responsibility analysis. Indicates an expert in identifying rectification status. This refers to experts who verify document specifications; the expert model with the highest matching probability (or exceeding a certain threshold) is selected as the optimally matched expert model.

[0070] S403: The corresponding business task is processed by the best-fit expert model, and the business processing results of each expert model are obtained.

[0071] Targeted processing refers to the process by which each expert model processes the input data and outputs corresponding results based on its specific training task.

[0072] For example, an adaptive multimodal routing method is used to distribute tasks. Specifically, this method dynamically selects the appropriate expert sub-model based on the modal complexity, evidence completeness, alignment confidence, and risk level of the input samples to achieve differentiated processing of different types of materials from inspections and investigations. Since inspection and investigation operations involve various data scenarios such as text-dominated, image-dominated, and table-dominated data, this embodiment uses a dynamic routing mechanism to replace the fixed-path processing method, thereby improving the accuracy and efficiency of task assignment.

[0073] To ensure the scientific and rational nature of routing decisions, the reward function is constructed as follows during the routing training process: ; in, Indicates the accuracy of problem identification. Indicates evidence coverage. Indicates the completeness of responsibility. Indicates processing time. Indicates the error routing rate. , β , γ , δ and η These are the weighting coefficients.

[0074] The optimization objective of the dynamic routing module can be expressed as: ; in, γ As a discount factor, T The number of decision steps.

[0075] During the inference execution phase, if the optimally matched expert model is the evidence extraction expert... e 1. The expert model will extract all relevant evidence from the input data; if a problem-specific expert is selected... e 2. The expert model will then qualitatively classify the problems found based on policy and institutional provisions; other expert models will similarly complete their respective business tasks and output the corresponding business processing results.

[0076] S404: Integrate all business processing results to generate a structured business processing result containing a set of issues, a set of responsibilities, and a set of evidence.

[0077] Among them, the structured business processing result refers to the information set with a unified format obtained by integrating the scattered information output by various expert models.

[0078] For example, by integrating the business processing results of all expert models, a structured processing result is generated as follows: ; in, Represents a set of problems. Represents a set of responsibilities. This represents a set of evidence.

[0079] This embodiment uses a reinforcement learning strategy network to dynamically select the optimal expert model based on the business characteristics of the input data, replacing the traditional fixed-path processing method. It can effectively cope with various data scenarios in inspection and patrol operations, such as text-dominated, image-dominated, and table-dominated data, and significantly improve the accuracy of task assignment and processing efficiency.

[0080] In an exemplary embodiment, the structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes, including: S501: Based on the problem set, responsibility set, and evidence set in the structured business processing results, instantiate knowledge graph nodes and generate relation edges.

[0081] Among them, knowledge graph nodes refer to the basic units used to represent business entities; relation edges refer to the connection units used to represent the semantic relationship between two entities; the instantiation process is achieved through entity alignment, coreference resolution, and relation merging. Entity alignment refers to the process of merging the same entity from different sources; coreference resolution refers to the process of unifying different expressions in the text that point to the same entity; relation merging refers to the process of merging different edges that represent the same semantic relationship.

[0082] For example, based on the structured business processing results and the responsibility identifier, timestamp, page position, and evidence anchor information in the multimodal representation, graph nodes are instantiated through entity alignment, coreference resolution, and relation merging. v i And generate relation edges e ij This maps the front-end output to a dynamic spatiotemporal knowledge graph structure.

[0083] S502: Add a rectification status attribute to each node; the rectification status attribute includes rectification stage, timestamp, time limit node, and responsibility transfer record.

[0084] Among them, node attributes refer to the set of characteristic information used to describe the entity represented by the node; rectification stage refers to the specific stage of the problem in the rectification closed loop; timestamp refers to the specific time when the node status changes; time limit node refers to the deadline for the problem rectification to be completed; and responsibility transfer record refers to the historical record of the transfer of problem responsibility between different entities.

[0085] For example, node Represented as:

[0086] in, For entity identification, For entity categories, This is the rectification phase. For timestamps, For time-limited nodes, This serves as a record of the flow of responsibility. Preferably, the entity category includes at least personnel, departments, policy clauses, issue categories, evidence materials, rectification measures, rectification deadlines, KPI indicators, warning levels, and historical cases; the rectification stage includes at least pending rectification, in progress, pending review, completed, and rebound warning. Thus, nodes are used not only to represent business entities but also to represent the status attributes of that entity in the rectification closed loop.

[0087] S503: Add a relation attribute to each edge; the relation attribute includes the relation type, the time the relation occurred, and the rectification state transition.

[0088] Among them, edge attributes refer to the set of feature information used to describe the relationship represented by the edge; relationship type refers to the specific type of semantic relationship between two entities; relationship occurrence time refers to the specific time when the relationship is established between two entities; rectification state transition refers to the process of a problem changing from one rectification stage to another.

[0089] For example, edge Represents a node With nodes The relationship between them can be represented as follows: ; in, For relational types, For the time when the relationship occurred, This refers to state transition attributes. Preferably, the relationship type may include involving, violating, referencing, belonging to, corresponding to, rectifying, responsible for, originating from, triggering, and completing; state transition attributes are used to describe the evolution direction of the problem in the inspection and supervision process, and preferably, they include at least: .

[0090] S504: Based on nodes and edges, a spatiotemporal knowledge graph containing rectification status attributes is constructed.

[0091] Spatiotemporal knowledge graphs refer to knowledge graphs that simultaneously contain entity relationship information in the spatial dimension and state evolution information in the temporal dimension, and can reflect the changes of business entities over time.

[0092] For example, by integrating all instantiated nodes and generated edges, a time-space representation can be constructed. t The knowledge graph is represented as: ; in, For a set of nodes, Let be the set of edges. It is a collection of attributes.

[0093] This knowledge graph is a spatiotemporal knowledge graph that includes rectification status attributes. For example... Figure 3 As shown, the dynamic spatiotemporal knowledge graph constructed in this embodiment includes two parts: node attributes and edge attributes. The node attributes cover the rectification stage, time limit nodes, and responsibility transfer records, while the edge attributes cover the relationship type, timestamp, and state transition.

[0094] This embodiment maps the structured business processing results to nodes and edges of a graph, and adds attributes such as rectification status, timestamps, and responsibility transfer to the nodes and edges, so that the knowledge graph can not only represent the static relationship between business entities, but also the dynamic evolution process of the problem in the rectification closed loop.

[0095] To avoid state updates violating government closed-loop management rules, in an exemplary embodiment, a state transition legality verification mechanism is introduced during graph updates. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured factual data, including: S601: During the evolution and update of the spatiotemporal knowledge graph, the legality of the candidate rectification state transition is verified by the state transition legality verification rules; if the rectification state transition conforms to the preset rules, the evolution and update of the spatiotemporal knowledge graph is allowed; if the rectification state transition violates the preset rules, the evolution and update of the spatiotemporal knowledge graph is prevented and the manual intervention review process is triggered.

[0096] Among them, the legality verification of state transition refers to the process of checking whether the change in the rectification status of the problem complies with the business management rules of inspection and supervision; the preset rules refer to the legal state transition rules set in advance according to the management specifications for inspection and supervision rectification.

[0097] For example, let the current state be... Candidate state is Then the state transition legality function can be expressed as: ; Preferably, the preset rules include at least the following: the completed status should not automatically revert to the pending rectification status; after a case is closed, the same item should not be added as an unclosed issue without manual intervention; the pending review status can only enter the completed status after review is passed. If detected... If this happens, the system will prevent automatic updates and trigger a manual review process to ensure that the evolution of the map status complies with the inspection and rectification management standards.

[0098] S602: Input the evolved spatiotemporal knowledge graph into the pre-constructed temporal graph neural network to update the node representation of the spatiotemporal knowledge graph.

[0099] Temporal graph neural networks (TPNs) are neural network models that can process graph-structured data and simultaneously consider time-dimensional information, used to update node representations in knowledge graphs. Attention mechanisms are commonly used techniques in temporal graph neural networks to measure the degree of influence of different neighboring nodes on the current node.

[0100] For example, during graph propagation, a time-aware graph neural network is used to update the node representations. Let the nth node be an example of this. l Layer nodes are represented as Then its update formula can be expressed as: ; in, R For a set of relation types, In relation r Next and Node i The set of connected neighbor nodes, The time difference between nodes For state transition attributes, and For trainable parameters, For activation function, This indicates vector concatenation.

[0101] Attention weight It is given by the following formula: ; in, For attention scoring functions.

[0102] Furthermore, to reflect the enhanced constraint features of this embodiment for inspection and patrol business rules, a penalty term for violating state transition logic is introduced during the training process. Let the state sequence predicted by the model be... Then the constraint loss can be expressed as: ; The total loss function of the graph propagation network can then be expressed as: ; in, For tasks involving problem identification, relation extraction, or state prediction, the loss is... These are the weight coefficients for the constraint terms. By using the constraint loss formula and the total loss function, the model can learn the legal state transition rules in the inspection and patrol scenario during the training phase, thereby avoiding the generation of unreachable state sequences and improving the business consistency of the graph evolution.

[0103] S603: Calculate the rectification risk score of each problem entity in the updated spatiotemporal knowledge graph using a pre-built rectification risk scoring function; generate early warning information when the rectification risk score exceeds a preset threshold or the rectification status remains unchanged for a preset duration.

[0104] Among them, the rectification risk score refers to a numerical value that quantifies the degree of risk in rectifying a problem; the early warning threshold refers to a pre-set critical value for the risk score that triggers a risk warning; and the status stagnation refers to a situation where the rectification status of a problem does not change within multiple consecutive time steps.

[0105] For example, in the rectification closed-loop assessment phase, the problematic entity is linked to key performance indicators to construct a rectification status sequence. Let the first... i The question at any moment t The state vector is: ; in, Indicates the completion rate of rectification. Indicates the delinquency rate. Indicates the recurrence rate. Indicating consistency of evidence, Indicates the responsibility closure rate. This represents the approval rate. Based on historical rectification data, current status transition records, and node representations after graph propagation, a time-series prediction model is used to predict the rectification status at the next moment. Its expression is: ; in, This indicates LSTM, GRU, or other time-series prediction functions. This represents the final node representation after the graph propagation.

[0106] Furthermore, a rectification risk scoring function is constructed: ; in, , , , , , Let be the weighting coefficient, and satisfy: ; Preferably, the weighting coefficients can be determined based on historical case data from inspections and audits, using the analytic hierarchy process (AHP), expert scoring, or a combination of both, and can be dynamically adjusted based on operational feedback, thereby enhancing the interpretability and applicability of the risk score.

[0107] A risk warning will be triggered when any of the following conditions are met: or ; in, This indicates the preset warning threshold. L It represents the length of time during which the continuous state remains unchanged. This indicates the rectification phase. Used to identify situations where the risk of rectification is too high. It is used to identify situations where problems have been stalled for a long time, rectification progress is slow, or the status has not changed effectively.

[0108] S604: Integrate issues, evidence, responsibilities, rectification status, and early warning information from the spatiotemporal knowledge graph to generate structured factual data.

[0109] Structured fact data refers to a standardized set of information extracted from knowledge graphs that can be directly used for document generation, including information such as problems, evidence, responsibilities, rectification status, and warnings.

[0110] For example, when the warning conditions are met or the review and cancellation are completed, the system writes the evaluation results back to the dynamic spatiotemporal knowledge graph, updating the node status, edge status, and risk level. The update rules can be expressed as follows: ; in, This represents the set of newly added or updated nodes. This represents the set of newly added or updated edges. This represents a set of newly added or updated attributes.

[0111] The current subgraph, evaluation results, and early warning information are further serialized to form a structured object: ; in, P For a set of questions, E For the evidence set, R For the responsibility set, S For rectification status set, W For the early warning set, this structured object is structured factual data. For example... Figure 3 As shown, the rectification closed-loop evaluation process in this embodiment is based on a dynamic spatiotemporal knowledge graph. Through state transition legality verification, time sequence prediction, risk scoring, and early warning triggering, it ultimately generates structured factual data and writes it back to the graph, forming a complete rectification closed-loop management link.

[0112] This embodiment ensures that the graph evolution conforms to business rules by introducing a state transition legality verification mechanism, uses a temporal graph neural network to update node representations and predict rectification status, constructs a multi-dimensional rectification risk scoring function to achieve risk warning, and forms a closed-loop management through an evaluation result write-back mechanism, which significantly enhances the ability to trace responsibility, the ability to track the rectification process, the legality of state evolution, and the ability to provide closed-loop risk warning.

[0113] In one exemplary embodiment, the results, rectification status, and early warning information of the inspection and supervision are determined based on structured fact data, and a draft inspection document is generated, including: S701: Based on structured fact data, generate prompt templates through a pre-built prompt template constructor.

[0114] The prompt template constructor is a function that converts structured factual data into prompt text that meets the input requirements of a large language model; the prompt template is a standardized template used to guide a large language model to generate text that meets specific format and content requirements.

[0115] For example, a prompt template can be constructed based on structured factual data: ; in, T ( ) is the prompt template constructor, which will clearly inform the large language model of the document type, content structure, factual basis and format requirements to be generated.

[0116] S702: Input the prompt template into the pre-built large language model to generate the first draft of the document.

[0117] Among them, the large language model refers to a pre-trained language model with powerful natural language understanding and generation capabilities, which can generate text content that meets the requirements based on the prompt template; the factual consistency verification refers to the process of checking whether the generated document content is consistent with the structured factual data, preferably including four dimensions: problem consistency, evidence consistency, responsibility consistency, and rectification status consistency.

[0118] For example, inputting the prompt template into the large language model generates document content: ; in, Y To generate the result.

[0119] To ensure that the generated results are consistent with the facts in the knowledge graph, a consistency verification function is constructed: ; in, This indicates a consistency check for the issue. This indicates the consistency verification of evidence. This indicates a consistency check of responsibilities. This indicates a consistency check of the rectification status.

[0120] If the following formula is met, it indicates that the generated result has not met the preset consistency requirements: ; in, This is the consistency threshold. At this point, the regeneration mechanism is triggered: ; in, Prompt' This is the revised prompt template.

[0121] The final output of the inspection and supervision documents includes: ; in, This indicates the inspection report. This indicates a list of issues. Suggestions for rectification were given. This indicates a summary of the content.

[0122] like Figure 4 As shown, the document generation process in this embodiment constructs a prompt template based on structured data output from a knowledge graph. After generating a draft using a large language model, a four-dimensional consistency check is performed. If the check fails, the prompt template is revised and regenerated. If the check passes, a compliant document is output, effectively avoiding the illusion problem caused by the free generation of general large models. Specifically, using structured objects (problem set, evidence set, responsibility set, rectification status set, and warning set) output from the knowledge graph as input, a constrained prompt template is constructed. A draft of the inspection and supervision document is generated by the large language model, and then a four-dimensional consistency check of problems, evidence, responsibility, and rectification status is performed. If the threshold is met, a compliant document is output; otherwise, a revision and regeneration are triggered, ensuring that the output content is strictly anchored to traceable factual evidence.

[0123] This embodiment constructs a constrained prompt template based on the structured results of a knowledge graph, and combines it with a four-dimensional consistency verification mechanism to ensure that the generated document content meets the inspection and supervision business norms in terms of language expression, factual citation, responsibility orientation, and rectification description. This effectively reduces the risk of generated content deviating from the facts, mismatch of responsibilities, or distorted conclusions, and achieves high-quality and high-efficiency automated generation of inspection documents.

[0124] like Figure 5 As shown, Figure 5 A method for generating compliance inspection documents based on the above embodiments is shown, including: Step 1: Layout-aware structured multimodal data representation.

[0125] The data processed consists of raw supervisory data sets generated during inspections and audits, including policy and regulatory texts, scanned copies of inspection reports, rectification ledgers, audit attachments, and tabular data. Scanned documents undergo resolution enhancement, noise reduction, skew correction, and page segmentation. Text content is extracted using OCR, and a page layout analysis model is used to identify paragraphs, headings, tables, and signature areas, generating structured intermediate results. Based on this, a text encoder is employed. Visual encoder and table encoder After extracting semantic features and mapping them to a unified semantic space using a projection matrix, page numbers are explicitly encoded. Page coordinates OCR confidence Identification of the source of evidence and identification of responsible parties Structured auxiliary information is used to form a unified input representation. This indicates that evidence naturally carries the characteristics of locatability and accountability.

[0126] Step 2: Anchor Point Constraint-Enhanced Graphic-Text Semantic Alignment

[0127] Based on the multimodal representation output in step one, cross-modal semantic alignment is performed between the policy clause text and the corresponding factual description regions in the inspection scan documents. Text features are extracted separately. and image features Cosine similarity is calculated as the basic semantic matching degree, and a contrastive learning loss is constructed. Based on this, a three-pronged approach to constraining losses, unique to inspection and supervision operations, is introduced: institutional clauses constrain losses. Ensure that the alignment results meet the matching requirements of the legal basis; evidence association constraint loss. Strengthen the correspondence between visual evidence and textual facts; hold responsible parties accountable for losses. To ensure accurate linking of responsible parties with evidentiary materials. Total losses amounted to... By optimizing this objective, a joint representation of text and graphics is output. and alignment confidence To form an alignment result .

[0128] Step 3: Dynamic adaptive service routing based on reinforcement learning.

[0129] The image-text joint features output in step two Alignment confidence and the completeness of evidence assessed from the data. Modal complexity and risk level Construct state vector The policy network outputs the selection probability of each business expert sub-model based on the state. The expert set EE is designed for inspection and patrol tasks, including evidence extraction experts. Problem qualitative experts Chain of responsibility analysis expert Rectification Status Identification Expert Document verification experts The routing training constructs a reward function based on business metrics such as problem identification accuracy, evidence coverage, and responsibility integrity rate. Through reinforcement learning, it maximizes the expected return, enabling the system to dynamically select the optimal expert path even in noisy or mixed-modal scenarios. This stage outputs structured extraction results. That is, the set of problems, the set of responsibilities, and the set of evidence.

[0130] Step 4: Integration and rectification closed-loop evaluation of rule-constrained dynamic spatiotemporal knowledge graph.

[0131] The output of step three Based on the structured information such as timestamps and responsibility identifiers retained in step one, graph nodes and edges are instantiated through entity alignment, coreference resolution, and relation merging. Includes physical identification, category, and rectification stage. timestamp Time Limit Node and responsibility transfer records ;side Attachment relationship type, occurrence time and state transition attributes This system depicts the evolution path from "pending rectification → in progress → pending review → completed → rebound warning." During graph updates, a state transition legality verification function Ω is introduced to determine the compliance of candidate state transitions. If preset rules such as "completed transitions cannot be automatically rolled back" or "unauthorized additions are not allowed after account cancellation" are violated, automatic updates are prevented and manual review is triggered. After a legal update, a time-aware graph neural network is used for node representation propagation, and a penalty term for state transition logic violations is added during training. L cons This allows the model to internalize legal state transition rules.

[0132] During the rectification and closed-loop evaluation phase, the problematic entities are linked to key performance indicators, constructing a multi-dimensional state vector that includes rectification completion rate, overdue rate, recurrence rate, evidence consistency, responsibility closure rate, and review pass rate. Predict the rectification status at the next time step using time-series prediction models (such as LSTM or GRU). The system employs a dual mechanism of risk scoring and state stagnation detection for early warning: a risk warning is triggered when the risk score exceeds a threshold θ, or when the state remains unchanged for LL consecutive timeframes during the rectification phase. The assessment results are updated to the knowledge graph via a write-back mechanism, forming a closed-loop management chain of "problem discovery—responsibility assignment—rectification progress—review and closure—rebound warning." Finally, the current knowledge graph subgraph, assessment results, and warning information are serialized into structured objects. .

[0133] Step 5: Automated generation of compliance documents for rigid constraints of the graph.

[0134] Using the structured object K output from step four as input, the prompt template constructor T() is called to organize the problem set P, evidence set E, responsibility set R, rectification status set S, and warning set W into a prompt text Prompt that conforms to the logic of an inspection report. The large language model generates a draft document Y based on this prompt. Subsequently, a four-dimensional consistency check is performed on the generated result, considering the problems, evidence, responsibilities, and rectification status. If the consistency score C(Y,K) is lower than a preset threshold θ, it indicates a risk of factual deviation or responsibility mismatch. The system automatically corrects the prompt template and triggers regeneration until the check passes. The final output includes a compliance document D comprising an inspection report, a problem list, rectification suggestions, and a summary.

[0135] The above five steps are interconnected, with the output of the previous step directly serving as the input for the next step. Furthermore, the warning and evaluation results generated in step four can be written back to the knowledge graph, forming a data-driven continuous optimization loop. This satisfies the business needs of heterogeneous document processing, strict accountability, full-cycle rectification management, and strong compliance document generation in the context of inspection and supervision.

[0136] This invention, through the aforementioned design, systematically solves the accuracy, efficiency, and integration challenges faced by existing intelligent analysis systems for inspection and supervision when processing multi-source heterogeneous data. Firstly, addressing the problem of broken connections between key elements due to the complex layout of inspection and supervision documents and the dispersion of evidence, this invention proposes a synergistic approach: "Layout-aware structured multimodal representation" and "anchor-constrained semantic alignment of text and images." On the one hand, during the feature extraction stage, page numbers, layout coordinates, evidence source identifiers, and responsible entity identifiers are explicitly encoded, enabling data representation to naturally carry evidence location and responsibility traceability. On the other hand, during the alignment stage, a triple anchor-point loss is introduced: institutional clause constraints, evidence association constraints, and responsible entity constraints. This ensures that the text-image matching results simultaneously meet the requirements of semantic consistency, evidence association, and responsibility location, fundamentally solving the problems of semantic deviation and broken responsibility connections caused by obscured official seals, multi-column layouts, and scanning noise. Based on this, the "Dynamic Adaptive Business Routing Based on Reinforcement Learning" designed in this invention replaces the traditional static gated routing with an expert sub-model pool customized for inspection and supervision tasks and a reward function driven by business indicators. This enables the system to dynamically select the optimal expert path based on modal complexity, evidence completeness, and risk level, significantly improving the processing efficiency and accuracy of multi-source supervision data fusion.

[0137] Secondly, while achieving high-precision and high-efficiency analysis, this invention constructs a rule-constrained closed-loop supervision and evaluation system, solving the problem of weak integration between knowledge graphs and rectification evaluation. By reconstructing traditional static triples into dynamic spatiotemporal graphs where nodes have rectification stage attributes, time limits, and responsibility flow records, and edges have state transition attributes, and embedding a state transition legality verification mechanism, illegal updates that violate government closed-loop rules are prevented, ensuring that the rectification process strictly follows the evolutionary norm of "pending rectification → in progress → pending review → completed → rebound warning." Based on this, combined with a multi-dimensional rectification state vector and a dual early warning mechanism, early warning of rectification overdue risks and automatic identification of rectification stagnation are achieved, forming a closed-loop management link of "problem discovery—responsibility assignment—rectification progress—review and closure—rebound warning." Finally, this rule-constrained dynamic knowledge graph drives a large language model, which takes structured facts as rigid input and constrains the generation process through a four-dimensional consistency verification mechanism of problems, evidence, responsibilities, and rectification status. This not only enables the automated and high-quality generation of documents such as inspection and supervision reports, problem lists, and rectification suggestions, but also effectively eliminates model illusions, ensuring that the output content is strictly anchored to traceable factual evidence. This meets the strict requirements of strong standardization, strong factuality, and strong compliance for inspection and supervision documents, providing solid technical support for the intelligent governance of the entire process of inspection and supervision work.

[0138] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0139] Based on the same inventive concept, this application also provides a device for generating compliance inspection documents using a fusion of text-image alignment and spatiotemporal knowledge graph, which implements the aforementioned method for generating compliance inspection documents using a fusion of text-image alignment and spatiotemporal knowledge graph. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in the embodiments of the device for generating compliance inspection documents using a fusion of text-image alignment and spatiotemporal knowledge graph provided below can be found in the limitations of the method for generating compliance inspection documents using a fusion of text-image alignment and spatiotemporal knowledge graph described above, and will not be repeated here.

[0140] Please see Figure 6 This invention provides a device for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs, comprising: The multimodal unified representation module is used to perform multimodal unified representation on the multi-source heterogeneous raw data obtained from inspections and patrols, and generate a unified multimodal representation. The image-text semantic alignment module is used to perform image-text semantic alignment on the unified multimodal representation and generate joint image-text features with alignment confidence. The dynamic adaptive multimodal routing module is used to determine the business characteristics of multi-source heterogeneous raw data based on the joint features of text and images with alignment confidence; select the expert model that best matches the business characteristics for targeted business processing, and obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses; The spatiotemporal knowledge graph construction and rectification evaluation module is used to map structured business processing results into nodes and edges of the knowledge graph, constructing a spatiotemporal knowledge graph containing rectification status attributes; constraining the dynamic evolution process of the spatiotemporal knowledge graph through state transition legality verification rules, and performing rectification closed-loop evaluation based on the evolved spatiotemporal knowledge graph to obtain structured factual data; rectification status attributes are used to describe the direction of evolution, and state transition legality verification rules are used to constrain the legality of evolution; The inspection document generation module is used to determine the results, rectification status, and early warning information of the inspection based on structured factual data, and to generate a draft of the inspection document; the draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] Reference Figure 7 This invention also provides a computer device, including: a memory and a processor, and a computer program stored in the memory. When the computer program is executed on the processor, it performs the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0143] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. They may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output devices, network access devices, etc.

[0144] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0145] In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the computer device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0146] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0147] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0148] This invention provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; Perform image-text semantic alignment on the unified multimodal representation to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of multi-source heterogeneous raw data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of the evolution. Based on structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The draft of the inspection document is then checked for factual consistency, and a compliant inspection document is obtained after passing the check.

[0149] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for patrol document compliance generation by fusing text-image alignment and spatio-temporal knowledge graph, characterized in that, Includes the following steps: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols to generate a unified multimodal representation; The unified multimodal representation is subjected to image-text semantic alignment to generate joint image-text features with alignment confidence. Based on the image-text joint features with alignment confidence, the business features of the multi-source heterogeneous original data are determined; the expert model that best matches the business features is selected for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses. The structured business processing results are mapped to nodes and edges of a knowledge graph to construct a spatiotemporal knowledge graph containing rectification status attributes. The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured fact data. The rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of evolution. Based on the structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated. The initial draft of the inspection document was checked for factual consistency, and a compliant inspection document was obtained after the check was passed.

2. The method of claim 1, wherein the method comprises: Multimodal unified representation is performed on the multi-source heterogeneous raw data obtained from inspections and patrols, generating a unified multimodal representation carrying business attributes, including: Text semantic features, image visual features, and table structure features are extracted from the multi-source heterogeneous raw data through a text encoder, a visual encoder, and a table encoder, respectively, and multimodal semantic features are constructed. The structured auxiliary information of the multi-source heterogeneous raw data is encoded into auxiliary information features; The multimodal semantic features are fused with the auxiliary information features to generate the unified multimodal representation.

3. The method of claim 1, wherein the method further comprises: The unified multimodal representation is subjected to image-text semantic alignment to generate joint image-text features with alignment confidence, including: Extract text features and image features from the unified multimodal representation; Based on the text features and image features, the basic semantic similarity is calculated using a pre-constructed similarity calculation function; Based on the basic semantic similarity, the basic semantic alignment loss is calculated using a pre-constructed contrastive learning loss function, and the business constraint loss is calculated using a pre-constructed anchor constraint loss function; the anchor constraint loss function is used to calculate the loss when violating institutional clause constraints, evidence association constraints, and responsible entity constraints. Using the sum of the basic semantic alignment loss and the business constraint loss as the overall optimization objective, semantic alignment optimization is performed on the unified multimodal representation to generate joint image and text features with alignment confidence.

4. The method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs according to claim 1, characterized in that, Based on the image-text joint features with alignment confidence, the business features of the multi-source heterogeneous original data are determined. An expert model optimally suited to these business features is selected for targeted business processing, resulting in structured business processing results, including: A business state vector is constructed based on the image-text joint features with alignment confidence and the auxiliary information of the multi-source heterogeneous original data; the business state vector is used to characterize the business features of the multi-source heterogeneous original data. Based on the business state vector, the matching probability of each expert model is calculated through a reinforcement learning policy network; the expert model whose matching probability satisfies the preset matching probability condition is taken as the optimally fitted expert model. The corresponding business tasks are processed by the optimally adapted expert model, and the business processing results of each expert model are obtained. Integrate all the aforementioned business processing results to generate a structured business processing result containing a set of issues, a set of responsibilities, and a set of evidence.

5. The method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs according to claim 1 or 4, characterized in that, The structured business processing results are mapped to nodes and edges of a knowledge graph, constructing a spatiotemporal knowledge graph containing rectification status attributes, including: Based on the problem set, responsibility set, and evidence set in the structured business processing results, instantiate knowledge graph nodes and generate relation edges; Add a rectification status attribute to each node; the rectification status attribute includes rectification stage, timestamp, time limit node, and responsibility transfer record; Add a relation attribute to each edge; the relation attribute includes relation type, relation occurrence time, and rectification state transition. Based on the nodes and edges, a spatiotemporal knowledge graph containing rectification status attributes is constructed.

6. The method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs according to claim 5, characterized in that, The dynamic evolution process of the spatiotemporal knowledge graph is constrained by state transition legality verification rules, and a rectification closed-loop evaluation is performed based on the evolved spatiotemporal knowledge graph to obtain structured factual data, including: During the evolution and update of the spatiotemporal knowledge graph, the legality of candidate rectification state transitions is verified through state transition legality verification rules. If the rectification state transition conforms to the preset rules, the evolution and update of the spatiotemporal knowledge graph is allowed. If the rectification state transition violates the preset rules, the evolution and update of the spatiotemporal knowledge graph is prevented and a manual intervention review process is triggered. The evolved spatiotemporal knowledge graph is input into a pre-constructed temporal graph neural network to update the node representations of the spatiotemporal knowledge graph; The rectification risk score of each problem entity in the updated spatiotemporal knowledge graph is calculated using a pre-constructed rectification risk scoring function; when the rectification risk score exceeds a preset threshold or the rectification status remains unchanged for a preset duration, an early warning message is generated. By integrating the issues, evidence, responsibilities, rectification status, and early warning information in the spatiotemporal knowledge graph, structured factual data is generated.

7. The method for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs according to claim 1, characterized in that, Based on the structured factual data, the results, rectification status, and early warning information of the inspection and supervision are determined, and a draft of the inspection document is generated, including: Based on the structured fact data, a prompt template is generated by constructing a pre-built prompt template constructor. The prompt template is input into a pre-built large language model to generate the first draft of the document.

8. A device for generating compliance inspection documents by integrating text-image alignment and spatiotemporal knowledge graphs, characterized in that, include: The multimodal unified representation module is used to perform multimodal unified representation on the multi-source heterogeneous raw data obtained from inspections and patrols, and generate a unified multimodal representation. The image-text semantic alignment module is used to perform image-text semantic alignment on the unified multimodal representation and generate joint image-text features with alignment confidence. The dynamic adaptive multimodal routing module is used to determine the business characteristics of the multi-source heterogeneous original data based on the image-text joint features with alignment confidence; select the expert model that best matches the business characteristics for targeted business processing to obtain structured business processing results; the expert model is a dedicated model for processing various inspection and patrol businesses; The spatiotemporal knowledge graph construction and rectification evaluation module is used to map the structured business processing results into nodes and edges of the knowledge graph, constructing a spatiotemporal knowledge graph containing rectification status attributes; constraining the dynamic evolution process of the spatiotemporal knowledge graph through state transition legality verification rules, and performing rectification closed-loop evaluation based on the evolved spatiotemporal knowledge graph to obtain structured fact data; the rectification status attributes are used to describe the direction of evolution, and the state transition legality verification rules are used to constrain the legality of evolution; The inspection document generation module is used to determine the inspection results, rectification status and early warning information based on the structured fact data, and to generate the first draft of the inspection document. The initial draft of the inspection document was checked for factual consistency, and a compliant inspection document was obtained after the check was passed.

9. A computer device, characterized in that, The device includes a processor and a memory: The memory is used to store computer programs and send the instructions of the computer programs to the processor; The processor executes, according to the instructions of the computer program, a method for generating compliance inspection documents that integrates text and image alignment with spatiotemporal knowledge graphs as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for generating compliance inspection documents that integrates text-image alignment and spatiotemporal knowledge graphs as described in any one of claims 1-6.

Images