A Multimodal Data Governance Method for Large Language Model Intelligent Agent Orchestration
By constructing a unified data asset model and a large language model intelligent agent orchestration method, the problem of the separation between structured data and multimodal data governance frameworks is solved, realizing unified management and efficient governance of cross-modal data, and supporting natural language interaction and unified retrieval.
Patent Information
- Application Number
- CN202611602858.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from several problems, including fragmented governance frameworks for structured and multimodal data, lack of intelligent orchestration and scheduling mechanisms in governance processes, inability of the technology stack to simultaneously support file storage management, lack of collaborative processing of vector retrieval and structured filtering on the same storage layer, lack of a unified natural language interaction interface, and disconnect between governance and retrieval usage.
A unified data asset model is constructed, and a large language model intelligent agent orchestration method is adopted. Multimodal data assets are managed through a five-stage fully automated processing pipeline, including document ingestion, two-stage deduplication, quality inspection, artificial intelligence annotation and vectorization, and cross-modal alignment. The orchestrator intelligent agent drives external sub-intelligent agents to collaboratively execute governance tasks, thereby achieving unified management and retrieval of cross-modal data.
It enables unified management of structured and multimodal data, improves governance efficiency, supports natural language interaction, eliminates the separation between governance and use, and provides unified quality auditing and retrieval capabilities for cross-modal data.
Smart Images

Figure CN123115122A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data governance method for orchestrating large language model intelligent agents. Background Technology
[0002] Existing data governance technologies can be mainly categorized as follows:
[0003] The first category is traditional ETL data governance platforms (represented by Apache Atlas, DataHub, etc.). These platforms primarily target structured data (relational database tables) and use batch processing scripts with fixed workflows to complete data access, metadata extraction, and quality checks. They have extremely limited capabilities in processing unstructured data such as images and videos, and the governance process relies on manual configuration. Data often takes several days to several weeks from registration to availability.
[0004] The second category consists of multimodal intelligent question-answering systems based on large language models (such as the multimodal intelligent question-answering system based on a large language model disclosed in CN121681747A). These systems focus on question-answering retrieval, but lack a complete pre-processing pipeline from document ingestion, deduplication, and quality checks to semantic vectorization and cross-modal alignment. User-uploaded documents must undergo manual pre-processing before they can be used by the search engine.
[0005] The third category: Integrated AI analysis and modeling platforms (such as the integrated artificial intelligence analysis and modeling platform supporting large-scale data processing disclosed in CN121478760A). These platforms employ traditional big data technology stacks (Spark + Elasticsearch) and do not support collaborative processing of file storage management, vector retrieval, and structured filtering within the same storage layer.
[0006] Furthermore, in existing technologies, quality issues of structured data are typically recorded in a separate rules engine database, while quality issues of multimodal data only exist as file tags. The two cannot be queried uniformly in the same view, resulting in fragmented quality tracking information for cross-modal data assets. After data governance is completed, retrieval and use often require switching systems—structured data is queried through SQL, while unstructured data is queried through a file browser or a separate search engine. Users cannot seamlessly switch between querying different types of data assets in the same conversation, resulting in a system disconnect between governance and use. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention provides a multimodal data governance method for orchestrating large language model intelligent agents. This method solves the technical problems in existing technologies, such as the separation of structured data and multimodal data governance frameworks, lack of intelligent orchestration and scheduling mechanisms in the governance process, inability of the technology stack to simultaneously support file storage management, collaborative processing of vector retrieval and structured filtering in the same storage layer, lack of a unified natural language interaction entry point, and disconnect between governance and retrieval usage.
[0008] This invention provides a multimodal data governance method for orchestrating large language model intelligent agents, comprising:
[0009] A unified data asset model is constructed, which includes a three-layer structure: data source layer, asset catalog layer, and dataset layer. This unified data asset model is used to manage both structured data assets and multimodal data assets simultaneously.
[0010] The five-stage fully automated processing pipeline is encapsulated as a set of utility functions that can be called by the orchestrator agent. The set of utility functions also includes an exponential backoff retry mechanism. The five-stage fully automated processing pipeline process includes: for multimodal data assets, performing five stages through a distributed data processing engine and columnar vector database: file ingestion, two-stage deduplication, quality inspection, artificial intelligence annotation and vectorization, and cross-modal alignment.
[0011] The orchestrator agent identifies the user's natural language governance needs and breaks down the tasks, generating a sequence of governance tasks. Through a task scheduling mechanism, it drives external sub-agents to call a set of tool functions to collaboratively execute the sequence of governance tasks. It also manages the processing progress of each data asset through a state machine and persistently stores the processing progress in a relational database to realize the governance process.
[0012] Furthermore, the asset catalog layer stores the metadata of the two types of assets in a unified manner through a unified asset index table and sub-tables divided by asset type.
[0013] Furthermore, the file ingestion stage specifically involves: when the file size is less than a preset storage threshold, inline storage mode is used; when the file size is greater than or equal to the preset storage threshold, only the object storage path reference is stored in the columnar vector database.
[0014] Furthermore, the document ingestion stage also includes: pre-building embedded vector columns in the columnar vector database for fixed-dimensional zero vector placeholders.
[0015] Furthermore, the two-stage deduplication stage includes the following: the first stage includes: immediately after file ingestion, performing MD5 hash-based precise deduplication, grouping according to hash values, retaining the first record, and marking the rest as duplicate files; the second stage includes: calculating the cosine similarity of the generated embedding vectors, performing semantic deduplication clustering on files with similarity values greater than a preset threshold, retaining the file with the best quality status in each cluster, and identifying duplicate files through an independent duplicate tag field without overwriting the pipeline status field.
[0016] Furthermore, the quality inspection stage specifically involves: performing damage detection, fuzzy detection, overexposure detection, and underexposure detection in parallel on the deduplicated files using custom functions in the distributed data processing engine; and synchronously writing the detection results into a columnar vector database and a relational database.
[0017] Furthermore, in the AI annotation and vectorization stage, AI annotation includes: generating structured tags through a large language model visual interface and storing the structured tags in a columnar vector database; extracting text information from the file through optical character recognition; extracting structured anchor points through regular pattern matching; and merging and storing them with spatiotemporal data in the exchangeable image file information.
[0018] The vectorization process includes: first, describing the image content in the image data as natural language text through the large language model visual interface, and then converting the natural language text into high-dimensional embedding vectors through the text embedding interface.
[0019] Furthermore, the cross-modal alignment stage specifically involves: aggregating all annotated optical character recognition text information and structured labels from the columnar vector database; extracting candidate alignment anchors through regular pattern matching; obtaining the semantic type of structured fields in the relational database; obtaining the matching value of the structured field corresponding to the semantic type matching the anchor type in the structured table through exact matching or fuzzy matching based on the anchor type of the candidate alignment anchors; and creating a cross-modal association record in the asset association table, wherein the record includes: confidence score and alignment evidence.
[0020] Furthermore, the external sub-agents include: access sub-agents, metadata sub-agents, tag sub-agents, standard sub-agents, quality sub-agents, and lineage sub-agents.
[0021] Furthermore, in the process of the orchestrator agent driving external sub-agents to call tool function sets to collaboratively execute the governance task sequence through the task scheduling mechanism, multiple independent and parallel governance tracks are established, each track independently schedules sub-agents, and supports independent retry or cancellation operations for a single governance track.
[0022] The beneficial effects of this invention are:
[0023] The three-layer unified data asset model of this invention eliminates the fragmentation of management standards for two types of data assets, and realizes unified management of cross-modal assets under a single directory;
[0024] The present invention uses an orchestrator agent to drive six sub-agents to achieve automated governance across the entire chain, which significantly improves governance efficiency compared to traditional batch processing solutions.
[0025] This invention encapsulates the five-stage pipeline as a utility function, allowing users to drive the entire governance process using natural language, significantly reducing the technical threshold.
[0026] This invention provides a columnar vector database that uniformly stores file content, metadata, tags, and vectors, supporting hybrid retrieval of vectors, full text, and structured filtering, eliminating the separation between governance and usage;
[0027] This invention uses a two-stage deduplication process to eliminate redundancy, saving API call costs and processing time in subsequent stages while preserving the complete processing trace.
[0028] This invention uses a dual-write columnar vector database, LanceDB, and a relational database, PostgreSQL, to manage structured and multimodal quality issues in the same table, forming a unified quality audit chain across modalities.
[0029] This invention employs a two-step embedding method, namely image-to-text and then embedding, which ensures consistency across modal vector spaces and avoids limitations of multimodal embedding interfaces. Pre-built columns and incremental updates avoid table structure changes and high write overhead.
[0030] This invention enables cross-modal alignment to automatically establish associations between structured assets and multimodal assets, achieving cross-modal data semantic interconnection;
[0031] This invention's UDF is local multi-core parallel and can be smoothly scaled to a distributed cluster, flexibly adapting to data scale; exponential backoff retries ensure fault tolerance for external interface failures. Attached Figure Description
[0032] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:
[0033] Figure 1 This is a flowchart of a specific embodiment of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] The present invention will be further illustrated below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications to the present invention in various equivalent forms all fall within the scope defined by the appended claims.
[0036] like Figure 1 As shown, this invention provides a multimodal data governance method for orchestrating large language model intelligent agents, including:
[0037] A unified data asset model is constructed, which includes a three-layer structure: data source layer, asset catalog layer, and dataset layer. This unified data asset model is used to manage both structured data assets and multimodal data assets simultaneously. The asset catalog layer stores the metadata of both types of assets in a unified manner through a unified asset index table and sub-tables divided by asset type.
[0038] The five-stage fully automated processing pipeline is encapsulated as a set of utility functions that can be called by the orchestrator agent. The set of utility functions also includes an exponential backoff retry mechanism. The five-stage fully automated processing pipeline process includes: for multimodal data assets, performing five stages through a distributed data processing engine and columnar vector database: file ingestion, two-stage deduplication, quality inspection, artificial intelligence annotation and vectorization, and cross-modal alignment.
[0039] The built-in exponential backoff retry mechanism works as follows: after an external interface call fails, the waiting time interval is incremented, such as 1 second, 2 seconds, 4 seconds, and the number of retries is preset, such as 3 times. All files that fail to be retried are marked as failed and skipped, without blocking the continuous execution of the overall pipeline.
[0040] The specific details of each stage are as follows:
[0041] In the file ingestion phase: the storage mode is automatically selected based on the comparison between the file size and the preset storage threshold. Specifically, when the file size is less than the preset storage threshold, it is directly stored in the columnar vector database using inline storage mode; when the file size is greater than or equal to the preset storage threshold, only the object storage path reference is stored in the columnar vector database. At the same time, during the file ingestion phase, when creating the table, i.e. when pre-creating the embedded vector column in the columnar vector database, a fixed-dimensional zero vector is used as a placeholder to prepare for incremental updates in the subsequent vectorization phase.
[0042] Two-stage deduplication strategy: In the first stage, MD5 hashing is performed immediately after file ingestion to accurately deduplicate the files. After grouping by content hash value, the first record is retained and the rest are marked as duplicate files. In the second stage, which is performed after the AI annotation and vectorization stage, the cosine similarity is calculated using the embedded vectors generated in the AI annotation and vectorization stage. Files with similarity values greater than a preset threshold are semantically deduplicated and clustered. The file with the best quality status is retained in each cluster. Duplicate files are identified by an independent duplicate tag field, which does not cover the pipeline status field.
[0043] Quality inspection phase: User-defined functions in the distributed data processing engine perform damage detection, blur detection, overexposure detection, and underexposure detection in parallel on deduplicated files; the detection results are synchronously written to the quality status field and quality problem field in the columnar vector database, as well as the quality problem table in the relational database, so that the SQL quality inspection results of structured data and the image quality inspection results of multimodal data can be uniformly managed and queried in the same quality problem table, forming a unified cross-modal quality tracking view.
[0044] Artificial intelligence annotation and vectorization stage: The artificial intelligence annotation process is as follows: calling the visual interface of the large language model to generate structured tags according to the configurable hierarchical tag system, and storing the structured tags in the columnar vector database in hierarchical JSON format; extracting text information from the file through optical character recognition technology, and extracting structured anchor points from it through regular pattern matching, and merging and storing them with the spatiotemporal data in the exchangeable image file information;
[0045] The vectorization process is as follows: the generation of embedding vectors adopts a two-step replacement method of image to text and then to embedding vector. First, the image content is described as natural language text through the visual interface of the large language model. Then, the natural language text is converted into high-dimensional embedding vectors through the text embedding interface. The text embedding interface is the same as the text embedding interface used in the structured data governance process, ensuring that the structured data embedding vector and the multimodal data embedding vector are in the same vector space. The embedding vector is written to the zero vector placeholder position of the pre-built embedding vector column in the ingestion stage through incremental update operations. No table structure change operation is required in the entire processing pipeline.
[0046] Cross-modal alignment stage: Aggregate optical character recognition text and artificial intelligence tags (i.e., structured tags) from all completed labeled files in the columnar vector database, and extract candidate alignment anchors through regular expression pattern matching; query the semantic types of structured fields in the relational database to locate the structured fields corresponding to the semantic types that match the anchor types; search for matching values in the corresponding structured tables through exact matching or fuzzy matching; after a successful match, create a cross-modal association record in the asset association table, which includes confidence scores and alignment evidence.
[0047] After completing intent recognition and task decomposition, and generating a governance task sequence, the orchestrator agent maps each step of the governance task sequence to a call request for the corresponding tool function in the tool function set, based on the task type and the dependencies between each stage, through a task scheduling mechanism. External sub-agents, including: access sub-agent, metadata sub-agent, tagging sub-agent, standard sub-agent, quality sub-agent, and lineage sub-agent, are scheduled by the orchestrator agent. They pass the identifier and context of the data asset to be governed as function parameters according to the parameter conventions of the call request, and call the corresponding tool function. After the tool function completes execution, it returns a structured result, including processing status, quality result, structured tags, and embedding direction. The data is processed in multiple ways, and the orchestrator agent gathers the data to drive the state machine to advance the corresponding data assets to the next processing stage. When the external interface called by the tool function, such as the large language model visual interface or the text embedding interface, returns a failure, the exponential backoff retry mechanism built into the tool function set will automatically retry at an increasing waiting interval. If the number of retries is exhausted and the failure still occurs, the data asset will be marked as a failure and skipped, without blocking the continuous execution of the overall pipeline. During the process, multiple independent and parallel governance tracks are established. Each track independently schedules sub-agents and independently records the data asset processing status of each track in the relational database. It supports independent retry or cancellation operations for a single governance track without affecting the normal execution of other governance tracks.
[0048] The orchestrator agent manages the processing progress of each data asset through a state machine. The processing progress includes: ingested state, deduplication state, quality check completed state, annotation completed state, and alignment completed state. The processing progress is persistently stored in a relational database, supporting multiple governance tracks to be executed in parallel and each track to track its status independently, thus realizing the governance process.
[0049] A specific embodiment of the present invention also provides a multimodal data governance system for orchestrating large language model intelligent agents, including: a memory and a processor, wherein the processor includes: a unified data asset model module, an orchestrator intelligent agent module, a multimodal processing pipeline engine module, and a tool layer bridging module;
[0050] The unified data asset model module is used to establish a three-layer data asset model consisting of a data source layer, an asset catalog layer, and a dataset layer. It manages the metadata of both structured data assets and multimodal data assets through a unified asset index table and sub-tables divided by asset type.
[0051] The orchestrator agent module is used to receive users' natural language governance requests, perform intent recognition and task decomposition, generate governance task sequences, drive six sub-agents to perform collaborative execution through a task scheduling mechanism, and manage the processing progress of each data asset through a state machine.
[0052] The multimodal processing pipeline engine module is used to perform five-stage fully automated processing on multimodal data assets through a distributed data processing engine and a columnar vector database. In the document ingestion stage, embedded vector columns are pre-built with zero vectors as placeholders. In the artificial intelligence annotation and vectorization stage, zero vectors are replaced with real embedded vectors row by row through incremental update operations.
[0053] The tool layer bridging module is used to encapsulate the five-stage fully automated processing pipeline into a set of tool functions that can be called by the orchestrator agent module, with a built-in exponential backoff retry mechanism.
[0054] The process and effects of this invention are further illustrated below through practical examples:
[0055] Example 1 uses the complete process of the method of the present invention; Examples 2 and 3 respectively demonstrate the implementation process and effects of the method of the present invention in typical scenarios such as multi-track parallel governance and cross-modal unified governance; Example 4 presents alternative variations of each module of the present invention. Each example provides corresponding processing results and effects to fully disclose the implementation process of the present invention.
[0056] Example 1: Full-process management of industrial inspection images (Complete implementation of the method of this invention)
[0057] In this embodiment, 5000 industrial inspection JPEG images are used as the governance object. A natural language command is used: "Help me access, deduplicate, check quality, tag, align, and publish these vehicle inspection images as searchable assets." This drives the orchestrator agent to sequentially execute five stages: file ingestion, two-stage deduplication, quality check, AI annotation and vectorization, and cross-modal alignment. The specific steps are as follows:
[0058] Step A1: File Ingestion: Generate thumbnails for each image, with a maximum width of 300px, and upload them using the MinIOthumbnails / prefix path; Extract EXIF data: shooting timestamp, GPS coordinates, and file metadata: filename, format, size, MD5; Adaptive storage based on a preset storage threshold of 10MB: 4800 images smaller than 10MB are directly stored in LanceDB using BlobV2 inline mode, and 200 images larger than or equal to 10MB are stored only using MinIO path references; When creating the table, a 1024-dimensional embedding column is pre-built, with a zero vector [0.0]*1024 as placeholders;
[0059] Step A2: Two-stage deduplication: MD5 precise deduplication: Grouping by content_md5 and keeping the first one, 120 duplicate images were found and marked: is_duplicate=True, an independent field that does not overwrite pipeline_state; After AI annotation, semantic deduplication calculates the cosine similarity of the embedded vectors, 35 similar frames with similarity >0.95 are selected, and the one with the best quality is kept, resulting in 4695 valid files;
[0060] Step A3: Quality Check: Four types of detection are performed in parallel using UDF: damaged (decoding failure), blurry (Laplacian variance <100), overexposed (mean brightness >240), and underexposed (mean brightness <15); 4730 images passed (80 blurry / 50 overexposed / 15 underexposed / 5 damaged), and dual writes were made to the LanceDB (quality_status, quality_issues) and PostgreSQL quality_issues tables (issue_source='image_check') to achieve a unified quality view across modalities;
[0061] Step A4: AI Annotation and Vectorization: qwen3.5-plus visual interface according to the five-level railway inspection labeling system:
[0062] L1 Business Domain → L2 Scenario → L3 Target Object → L4 Status / Defect → L5 Quantified Attributes, for example: {'L1':'Track Inspection','L2':'Main Line','L3':'Rail','L4':'Crack','L5':'Length≈5cm'}; Output hierarchical JSON tags, with the tag system managed by versioned YAML configuration; OCR extracts text, and regular expressions extract anchor points (mileage markers K\d+\+\d+, equipment number EQ-\d+, etc.) and merge them with EXIF spatiotemporal data into the location; Embedded vectors use a two-step method of image to text and then embedding: visual interface description → text-embedding-v3 embedding, with the same interface as structured data to ensure consistency across modal vector spaces; Zero vector placeholders are incrementally written via update, requiring no table structure changes throughout the process; IVF_PQ vector index is established;
[0063] Step A5: Cross-modal alignment: Aggregate OCR text and AI tags to extract anchor points, query the semantic type of PostgreSQL catalog_columns to locate matching fields, and perform exact matching or edit fuzzy matching with a distance less than or equal to 2 in the equipment register table. After a successful match, create an associated record in catalog_relationships (0.95 confidence for exact matching / 0.70 confidence for fuzzy matching + alignment evidence JSON), and advance the status to aligned.
[0064] Step A6: Search and publish: Perform a mixed search using search_similar_images with the search query: "semantic vector similarity + quality_status='passed' ANDL3='rail' ANDL4='crack'". Filter by structure and return Top-20 results, including thumbnail signature URLs (2-hour temporary links), hierarchical AI tags, quality status, and similarity distance. Key parameters: storage threshold 10MB, blur threshold 100, overexposure threshold 240, underexposure threshold 15, semantic similarity threshold 0.95, edit distance ≤2, exponential backoff with 3 retries in 1s / 2s / 4s, alignment confidence 0.95 / 0.70.
[0065] The results of this embodiment: After full-process processing of 5000 images, 4695 valid files were obtained, cross-modal association records were established, forming searchable assets; users can obtain target assets through a hybrid search of "semantic vector + structured filtering + quality filtering". This embodiment fully demonstrates the implementation process and effects of the method of the present invention.
[0066] Example 2: Multi-track parallel governance and real-time progress tracking
[0067] This embodiment illustrates the implementation process and effect of the method of the present invention in three types of multimodal asset parallel governance scenarios, highlighting the task scheduling mechanism and the state machine's ability to independently manage multiple governance tracks.
[0068] Scenario: Simultaneously initiate governance for three types of multimodal assets—equipment inspection image set (inspection-images bucket, approximately 5000 images), technical document set (tech-docs bucket, approximately 2000 PDFs), and surveillance video frame set (surveillance-frames bucket, approximately 10000 keyframes). Specific process:
[0069] The orchestrator identifies three governance tracks (track-1 / 2 / 3) and starts them in parallel. Each track independently schedules sub-agents and records its processing status independently via PostgreSQL, without blocking each other. Users can query the progress of each track, for example: track-1 alignment is completed at 4695 / 5000, with a completion rate of 93.9%; track-2 quality check is 60% complete; track-3 has been ingested at 30%. Individual tracks can be retried or canceled independently. For example, a batch of track-3 that fails due to API rate limiting can be retried separately without affecting other tracks.
[0070] The effect of this embodiment is that the three governance tracks are executed in parallel and their states are independent of each other. The failure or cancellation of a single track does not affect other tracks, which verifies the multi-track parallelism and real-time progress tracking capabilities of the present invention.
[0071] Example 3: Unified Parallel Governance of Structured and Multimodal Data
[0072] This embodiment illustrates the implementation process and effect of the method of the present invention in the scenario of parallel governance of structured data assets and multimodal data assets, with a focus on cross-modal unified quality audit.
[0073] Scenario: Parallel management of the PostgreSQL device register table `equipment_register` (120 fields, 100,000 devices) and the MinIO inspection image set. Specific process:
[0074] The structured track performs SQL quality rule checks (null value rate > 5% threshold, uniqueness constraints, value range), and writes the results to the quality_issues table (issue_source='sql_rule'); the multimodal track executes a five-stage pipeline, with each track creating asset records in the catalog_assets unified index table; when a user queries a quality issue, the SQL check results and image check results are simultaneously aggregated from the same quality_issues table (issue_source='image_check'), forming a cross-modal unified quality audit.
[0075] The effect of this embodiment is that the quality results of structured and multimodal data are managed and queried in the same quality_issues table, forming a unified quality audit chain across modalities, which verifies the ability of this invention to eliminate the fragmentation of two types of data quality tracking.
[0076] Example 4: Modified Implementation (Components and Parameters Can Be Replaced)
[0077] This embodiment illustrates the replaceable variations of each module of the present invention. None of these variations change the implementation logic or effect of the method of the present invention: the distributed data processing engine can be replaced with other frameworks that support UDF parallelism, such as DaftonRay, which supports smooth scaling from a single-machine multi-core system to a distributed cluster; the columnar vector database can be replaced with other vector databases that support hybrid queries of vector indexing + full-text + structured filtering and support Blob storage; the large language model visual interface can be replaced with other visual interfaces that support image description and OCR, while the two-step embedding logic remains unchanged; the storage threshold is not limited to 10MB; the tag level is not limited to five levels; the exponential backoff retry parameter is not limited to 1s / 2s / 4s retrying 3 times, and can be adjusted as needed.
[0078] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for multi-modal data governance of large language model agent orchestration, characterized in that, include: A unified data asset model is constructed, which includes a three-layer structure: data source layer, asset catalog layer, and dataset layer. This unified data asset model is used to manage both structured data assets and multimodal data assets simultaneously. The five-stage fully automated processing pipeline is encapsulated as a set of utility functions that can be called by the orchestrator agent. The set of utility functions also includes an exponential backoff retry mechanism. The five-stage fully automated processing pipeline process includes: for multimodal data assets, performing five stages through a distributed data processing engine and columnar vector database: file ingestion, two-stage deduplication, quality inspection, artificial intelligence annotation and vectorization, and cross-modal alignment. The orchestrator agent identifies the user's natural language governance needs and breaks down the tasks, generating a sequence of governance tasks. Through a task scheduling mechanism, it drives external sub-agents to call a set of tool functions to collaboratively execute the sequence of governance tasks. It also manages the processing progress of each data asset through a state machine and persistently stores the processing progress in a relational database to realize the governance process.
2. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The asset catalog layer stores the metadata of the two types of assets in a unified manner through a unified asset index table and sub-tables divided by asset type.
3. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The file ingestion stage specifically involves: when the file size is less than a preset storage threshold, inline storage mode is used; when the file size is greater than or equal to the preset storage threshold, only the object storage path reference is stored in the columnar vector database.
4. The method of claim 3, wherein the method is performed by the large language model agent. The document ingestion stage also includes: pre-building embedded vector columns in the columnar vector database for fixed-dimensional zero vector placeholders.
5. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The two-stage deduplication stage includes the following: the first stage includes performing MD5 hashing for precise deduplication immediately after file ingestion, grouping files according to hash values, retaining the first record, and marking the rest as duplicate files; the second stage includes calculating the cosine similarity of the generated embedding vectors, performing semantic deduplication clustering on files with similarity values greater than a preset threshold, retaining the file with the best quality status in each cluster, and identifying duplicate files through an independent duplicate tag field without overwriting the pipeline status field.
6. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The quality inspection stage specifically involves: performing damage detection, fuzzy detection, overexposure detection, and underexposure detection in parallel on the deduplicated files using custom functions in the distributed data processing engine; and synchronously writing the detection results into a columnar vector database and a relational database.
7. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The AI annotation and vectorization stage includes: generating structured tags through a large language model visual interface and storing the structured tags in a columnar vector database; extracting text information from the file through optical character recognition; extracting structured anchor points through regular pattern matching; and merging and storing them with spatiotemporal data from exchangeable image file information. The vectorization process includes: first, describing the image content in the image data as natural language text through the large language model visual interface, and then converting the natural language text into high-dimensional embedding vectors through the text embedding interface.
8. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The cross-modal alignment stage specifically involves: aggregating all annotated text information and structured labels of optical character recognition from the columnar vector database; extracting candidate alignment anchors through regular pattern matching; obtaining the semantic type of structured fields in the relational database; obtaining the matching value of the structured field corresponding to the semantic type matching the anchor type in the structured table through exact matching or fuzzy matching based on the anchor type of the candidate alignment anchors; and creating cross-modal association records in the asset association table, wherein the records include: confidence scores and alignment evidence.
9. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. The external sub-agents include: access sub-agent, metadata sub-agent, tag sub-agent, standard sub-agent, quality sub-agent, and lineage sub-agent.
10. The method of claim 1, wherein the method is performed by a large language model agent orchestration system. In the process of the orchestrator agent driving external sub-agents to call tool function sets to collaboratively execute the governance task sequence through the task scheduling mechanism, multiple independent and parallel governance tracks are established. Each track independently schedules sub-agents and supports independent retry or cancellation operations for a single governance track.
Citation Information
Patent Citations
Integrated artificial intelligence analysis and modeling platform supporting large-scale data processing
CN121478760A
Multi-modal intelligent question-answering system and interaction method based on privately deployed large language model
CN121681747A
