An Automated Data Governance Method and System Based on Multimodal Large Models

By constructing a semantic anchor mapping network for a multimodal large model and employing a hierarchical governance strategy, the problem of cross-modal semantic conflicts in multimodal data was solved, achieving efficient data consistency calibration and quality improvement.

CN122674882BActive Publication Date: 2026-11-10HANGZHOU JINCHENG INFORMATION SECURITY TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611185740.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-06
Publication Date
2026-11-10
Estimated Expiration
2046-08-06

AI Technical Summary

Technical Problem

Existing single-modal independent processing methods cannot effectively handle cross-modal semantic associations of multimodal data, leading to the accumulation of semantic conflicts and implicit quality defects between modalities such as text, images, and audio during cross-modal transmission, resulting in semantic inconsistencies and inference failures in business applications.

Method used

An automatic data governance method based on a multimodal large model is adopted. By constructing a semantic anchor mapping network between modalities, a unified semantic index sequence is generated, implicit quality deviations are identified, cross-modal deviation transmission links are traced, and hierarchical governance strategies are matched for correction and calibration to ensure semantic consistency.

Benefits of technology

It improves the searchability and relevance of cross-modal data, accurately identifies hidden quality deviations, reduces overall computing power and human resource investment costs, achieves cross-modal semantic consistency calibration, avoids resource waste, and improves data quality and application reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122674882B_ABST
    Figure CN122674882B_ABST
Patent Text Reader

Abstract

This invention relates to the field of data governance technology, and more particularly to an automatic data governance method and system based on a multimodal large model. The method includes: integrating isolated multi-source heterogeneous data into a semantically related, searchable whole by establishing cross-modal semantic anchor mapping and a unified index, providing a globally ordered data view for automated governance; generating a single-modal deviation distribution map by identifying latent quality deviations, achieving a leap from surface detection to intrinsic quality assessment; tracing the cross-modal deviation transmission links along the associated paths, identifying coupling amplification effects and classifying risk levels, achieving the tracking and quantification of the propagation path of quality risks; performing tiered governance and cross-modal semantic consistency calibration based on risk levels to perform targeted correction of deviations, ensuring that multimodal data maintains a unified semantic benchmark after correction; and verifying the effectiveness of governance and dynamically updating the anchor network by feeding the calibrated data back into the large model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to an automatic data governance method and system based on a multimodal large model. Background Technology

[0002] In intelligent applications driven by multi-source heterogeneous data, it has become the norm for multi-modal data such as text, images, audio, and structured tables to collaboratively describe the same business fact. Existing data governance methods generally adopt a single-modal independent processing mode, which can maintain basic data usability in scenarios with single data modalities and simple semantic relationships.

[0003] However, when multimodal data generates cross-modal semantic associations around the same entity, existing single-modal independent processing methods face fundamental limitations: an error in recording an attribute in text may lead to image understanding bias, audio event annotation errors, and abnormal field filling in structured tables through cross-modal correspondences. Single-modal governance, lacking semantic anchor mapping and bias transmission analysis mechanisms between modalities, cannot perceive the coupling amplification effect of biases during cross-modal transmission. This results in governance only correcting errors at the surface level of each modality, leading to the continuous accumulation of cross-modal semantic conflicts and implicit quality defects. Ultimately, this causes serious semantic inconsistencies and inference failures in business applications of multimodal fused datasets. Therefore, this invention proposes an automatic data governance method and system based on a large multimodal model. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the background art by proposing an automatic data governance method and system based on a multimodal large model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides an automatic data governance method based on a multimodal large model, comprising: S1. Access multi-source heterogeneous multimodal data, extract core semantic entities from each modality, construct a semantic anchor mapping network between modalities, and generate a unified semantic index sequence covering all modal data; S2. Based on the unified semantic index sequence, latent quality deviations in the data are identified modally by modality through a multimodal large model, and single-modal deviation distribution maps corresponding to each modality are generated. S3. Based on the single-modal deviation distribution map, trace the cross-modal deviation transmission links along the association path of the semantic anchor mapping network, analyze the coupling amplification effect of deviation in the cross-modal transmission process, and generate the transmission risk level of each cross-modal deviation transmission link. S4. Match a hierarchical governance strategy to the transmission risk level of each cross-modal deviation transmission link, perform directional deviation correction on the multimodal data, and perform cross-modal semantic consistency calibration; wherein, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy and global calibration strategy. S5. Feed the semantically consistent calibrated multimodal data back into the multimodal large model to perform semantic consistency verification. After the verification is passed, generate a standardized governance dataset and update the semantic anchor mapping network simultaneously.

[0006] Further, S1 includes: Semantic entity extraction is performed on multimodal data such as text, images, audio, and structured tables to obtain independent semantic entity sets for each modality; Using the attribute tags and spatiotemporal identifiers of semantic entities as the matching benchmark, a one-to-one or one-to-many correspondence between semantic entities of different modalities is established as a semantic anchor point. Extract the time order of occurrence of each semantic entity in the corresponding multimodal data as the occurrence sequence, and extract the logical belonging level of each semantic entity in its respective modal data as the subordinate level; A semantic anchor mapping network is constructed based on the association relationships of all semantic anchors, and a unified semantic index sequence covering all modal data is generated according to the occurrence sequence and subordinate hierarchy of semantic entities.

[0007] Further, S2 includes: The single-modal data is combined with the corresponding unified semantic index and imported into the multimodal big model. The multimodal big model identifies implicit quality biases such as semantic conflicts, logical breaks, context mismatches, and missing attributes in the data. The location and type of each latent quality deviation in the corresponding modal data are labeled, and a single-modal deviation distribution map is generated according to the influence range of the deviation; wherein, the influence range of the deviation is the number of semantic entities affected by a single latent quality deviation and the coverage length of the corresponding data segment.

[0008] Further, S3 includes: The cross-modal semantic entity association path formed by the concatenation of semantic anchors in the semantic anchor mapping network is defined as the association path. The association path in the semantic anchor mapping network is extended and traced to other modalities. Each deviation node in the single-modal deviation distribution map is taken as the starting deviation node. Other modal derivative deviations caused by the starting deviation node are identified, and the starting deviation node and all derivative deviation nodes are connected to form a cross-modal deviation transmission link. The influence range of each deviation node in the cross-modal deviation transmission link is analyzed. Deviation nodes with expanded influence ranges are identified as coupling amplification nodes, and the coupling amplification effect of deviation in cross-modal transmission is obtained.

[0009] Furthermore, S3 also includes: Calculate the total number of deviation nodes and the number of coupling amplification nodes in each cross-modal deviation propagation link; Based on the number of modalities covered by the cross-modal deviation transmission links and the severity of the deviation types, all transmission links are divided into three transmission risk levels: high, medium, and low. Among them, the deviation types correspond to four types of latent quality deviations identified: semantic conflict, logical discontinuity, context mismatch, and attribute loss. Different deviation types correspond to different degrees of severity.

[0010] Further, S4 includes: For low-risk transmission links, an in-situ correction strategy is adopted to independently correct each deviation node in the low-risk transmission link; For medium-risk transmission links, a link correction strategy is adopted, which uses the coupling amplification node as the core to correct the deviation of the entire transmission link in a coordinated manner; For high-risk transmission links, a global calibration strategy is adopted to suspend all data flow associated with the corresponding link. The semantic benchmark of the entire link data is unified based on the semantic anchor mapping network and then corrected point by point. Among them, all modal data involved in the entire transmission link are defined as the entire link data.

[0011] Furthermore, S4 also includes: In the semantic anchor mapping network, semantic entities with complete semantic information and no quality deviation are selected as the benchmark semantic entities. Using the benchmark semantic entities in the semantic anchor mapping network as a reference, semantic alignment is performed on the corrected modal data. Eliminate the differences in attribute descriptions and spatiotemporal information deviations of different modalities for the same semantic entity; where attribute description differences refer to the inconsistency in the feature representations of the same semantic entity by different modalities, and spatiotemporal information deviations refer to the mismatch in the time and location identifications of the same semantic entity by different modalities.

[0012] Further, S5 includes: The calibrated multimodal data is synchronously input into the multimodal large model, triggering the multimodal large model to carry out cross-modal semantic reasoning based on the unified semantic index, and then obtain the reasoning results; If there are no semantic contradictions or information conflicts in the reasoning results, the verification is deemed successful and a standardized governance dataset is generated; if there are semantic contradictions or information conflicts, the transmission link corresponding to the contradiction is located and the corresponding hierarchical governance strategy is returned for correction.

[0013] Furthermore, S5 also includes: Extract newly added semantic entities from the validated standardized governance dataset, supplement the semantic anchor mapping network with the newly added semantic entities, and establish their association with existing semantic entities; Weights and priorities are pre-assigned to each associated path in the semantic anchor mapping network, where the weights and priorities are used to characterize the probability and importance of deviation propagation in the corresponding path; Based on the actual associated paths traversed by the deviation nodes during this governance process across modalities, the weight priority of each associated path in the semantic anchor mapping network is optimized.

[0014] A second aspect of the present invention provides an automatic data governance system based on a multimodal large model, comprising: Multimodal semantic index construction module: It accesses multimodal data from multiple sources and heterogeneous sources, extracts the core semantic entities from each modality, constructs a semantic anchor mapping network between modalities, and generates a unified semantic index sequence covering all modal data; Single-modal quality deviation identification module: Based on a unified semantic index sequence, it identifies latent quality deviations in the data modally by modally through a multimodal large model, and generates single-modal deviation distribution maps corresponding to each modality; Cross-modal deviation link tracing module: Based on the single-modal deviation distribution map, it traces the cross-modal deviation transmission link along the association path of the semantic anchor mapping network, analyzes the coupling amplification effect of deviation in the cross-modal transmission process, and generates the transmission risk level of each cross-modal deviation transmission link; The hierarchical deviation correction and calibration module matches a hierarchical governance strategy based on the transmission risk level of each cross-modal deviation transmission link, performs directional deviation correction on multimodal data, and performs cross-modal semantic consistency calibration; among which, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy, and global calibration strategy; Dataset verification and network update module: The multimodal data after semantic consistency calibration is fed back into the multimodal large model to perform semantic consistency verification. After the verification is passed, a standardized governance dataset is generated and the semantic anchor mapping network is updated synchronously.

[0015] Compared with existing technologies, the present invention provides an automatic data governance method and system based on a multimodal large model, the advantages of which are: 1) By extracting core semantic entities and constructing a semantic anchor mapping network, a unified semantic index sequence is generated to provide a unified semantic reference benchmark for all-modal data, significantly improving the retrieval and relevance of cross-modal data; by relying on a multimodal large model to deeply analyze data content modally, implicit quality deviations such as semantic conflicts and logical breaks are accurately identified, and a single-modal deviation distribution map is generated to intuitively present the location, type, and scope of influence of deviations, achieving visualization and quantifiable traceability; by locating the deviation transmission link along the association path of the semantic anchor mapping network, the coupling amplification effect in the cross-modal transmission process of deviations is identified, and the transmission risk level is divided based on multi-dimensional indicators to prevent the possibility of small deviations evolving into a full-domain data quality incident in advance; 2) By constructing a hierarchical and adaptable precision governance system, three differentiated strategies—in-situ correction, link correction, and global calibration—are matched for transmission links of different risk levels (low, medium, and high). This avoids the waste of resources caused by over-governance in low-risk scenarios and enables the unification of semantic benchmarks across the entire link for high-risk links. Ultimately, cross-modal semantic consistency calibration is completed, significantly reducing the overall computing power and human resource investment costs while ensuring governance effectiveness. By feeding the calibrated data back into the multimodal large model to complete semantic consistency verification, and simultaneously updating the newly added semantic entities and actual deviation transmission paths to the semantic anchor mapping network, the weight and priority of the associated paths are dynamically optimized, allowing the entire governance system to continuously accumulate scenario experience during continuous operation. Attached Figure Description

[0016] Figure 1 This is a flowchart of an automatic data governance method based on a multimodal large model proposed in this invention.

[0017] Figure 2 This is a block diagram of an automatic data governance system based on a multimodal large model proposed in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This invention provides an automatic data governance method based on a multimodal large model, comprising: S1. Access multi-source heterogeneous multimodal data, extract core semantic entities from each modality, construct a semantic anchor mapping network between modalities, and generate a unified semantic index sequence covering all modal data; S2. Based on the unified semantic index sequence, latent quality deviations in the data are identified modally by modality through a multimodal large model, and single-modal deviation distribution maps corresponding to each modality are generated. S3. Based on the single-modal deviation distribution map, trace the cross-modal deviation transmission links along the association path of the semantic anchor mapping network, analyze the coupling amplification effect of deviation in the cross-modal transmission process, and generate the transmission risk level of each cross-modal deviation transmission link. S4. Match a hierarchical governance strategy to the transmission risk level of each cross-modal deviation transmission link, perform directional deviation correction on the multimodal data, and perform cross-modal semantic consistency calibration; wherein, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy and global calibration strategy. S5. Feed the semantically consistent calibrated multimodal data back into the multimodal large model to perform semantic consistency verification. After the verification is passed, generate a standardized governance dataset and update the semantic anchor mapping network simultaneously.

[0020] In this embodiment of the invention, the specific implementation of S1, which accesses multi-source heterogeneous multimodal data, extracts core semantic entities from each modality, constructs a semantic anchor mapping network between modalities, and generates a unified semantic index sequence covering all modalities, is as follows: S11. Semantic entity extraction is performed on the text, image, audio, and structured table multimodal data respectively to obtain independent semantic entity sets for each modality, specifically: After accessing multi-source heterogeneous multimodal data, semantic entity extraction is performed based on the physical characteristics and information carrying forms of different modalities; For text modal data, an entity extraction pipeline based on grammar rules and domain dictionary matching is adopted. After sentence segmentation and word segmentation, named entities are identified by combining domain vocabulary and part-of-speech tagging rules. Dependency parsing is used to extract objects, events, attribute values, relational expressions, etc. that express complete semantic concepts in the text as semantic entities, and their start and end positions in the text are recorded. For image modal data, the system combines color histogram and texture feature extraction to identify salient object regions in the image; extracts text regions through an optical character recognition engine; and uses spatial position relationship inference rules to generate spatial relationships and interaction behavior descriptions between objects. Each detected object instance, scene type, and text content is encapsulated as an independent semantic entity, while the bounding box coordinates and mask regions of the entity in the image are recorded. For audio modal data, after converting speech to text through automatic speech recognition, semantic entities are extracted using the same text entity extraction logic. At the same time, for non-speech audio signals, such as ambient sounds, alarm sounds, and equipment operation sounds, the audio signals are framed, and acoustic features such as short-time energy, zero-crossing rate, and Mel frequency cepstral coefficients are extracted. The events are classified and the start and end timestamps of the events are determined in the time domain. The specific acoustic events identified are used as semantic entities, and their start and end timestamps in the audio stream are labeled. For structured table data, the schema definition and data rows of the table are parsed, and field names, table names and discrete or continuous values ​​in each data cell are treated as independent semantic entities. The hierarchical relationship and row and column coordinates between semantic entities are recorded according to the primary and foreign key relationship and the table structure. The extraction results of each modality are deduplicated and synonymized. Specifically, entities with overlapping positions or completely identical content within the same modality are merged. The synonym merging process refers to using a pre-built thesaurus and entity linking knowledge base to uniformly map entity names that point to the same objective thing but have different textual descriptions to standardized semantic tags. After deduplication and synonym merging, a set of independent semantic entities for each modality is formed, and each semantic entity carries the source modality identifier, content description, and native positioning information.

[0021] S12. Using the attribute tags and spatiotemporal identifiers of semantic entities as the matching benchmark, establish one-to-one or one-to-many correspondences between semantic entities of different modalities as semantic anchors. After obtaining the independent semantic entity sets for each modality, cross-modal semantic associations are constructed. A set of multi-dimensional attribute labels is automatically generated for each semantic entity. The attribute labels are used to characterize the essential features of the semantic entity, including the semantic entity's category name, key feature vectors, numerical attributes, and text description summary. At the same time, a unified spatiotemporal identifier is extracted from the data of each modality. The time identifier comes from the timestamps attached to the data collection, the presentation time embedded in the audio and video, or the normalized value of the time expression in the text. The spatial identifier is obtained through the GPS coordinates of the image, the scene reference coordinate system, the channel position information of the audio, or the location field recorded in the structured data. Using attribute tags and spatiotemporal identifiers as the matching benchmark, the cross-modal semantic entity alignment process is initiated: First, the category tags of two different modal semantic entities are compared to determine whether they belong to a predefined set of compatible categories; if compatible, their key attribute values ​​are further compared. Numerical attributes must be consistent within the tolerance range, and text attributes must have the same core words. Simultaneously, the spatiotemporal identifiers of the two are checked. When the timestamp difference is less than a preset time window and the spatial coordinate distance is less than a preset spatial neighborhood radius, it is determined that they point to the same object or event in the real world, thus establishing a one-to-one correspondence. The time window is obtained by reading the synchronization accuracy parameter of the multimodal data acquisition device and the maximum allowable delay parameter required for data fusion, and taking the smaller value of the two as the time window. This window ensures that the time deviation of different modalities recording the same event is within an acceptable synchronization error range. The spatial neighborhood radius is obtained by reading the intrinsic and extrinsic parameter calibration data of the image acquisition device and the nominal accuracy of the positioning sensor, calculating the spatial resolution within the joint coverage area of ​​all acquisition devices, and taking three times the spatial resolution value as the spatial neighborhood radius to ensure that the spatial deviation caused by positioning error will not mistakenly sever the true correspondence. When a semantic entity in one modality covers multiple fine-grained semantic entities in another modality in terms of attribute features, for example, a unit malfunction in text may correspond to three semantic entities: smoke, sparks in an image, and abnormal noise in an audio file. Then, a one-to-many correspondence is established based on hierarchical semantic analysis; all established correspondences are abstracted as semantic anchors.

[0022] S13. Extract the time order of occurrence of each semantic entity in the corresponding multimodal data as the occurrence sequence, and extract the logical belonging level of each semantic entity in the modal data as the subordinate level. Specifically, for the extraction of occurrence time sequence, a deep analysis of the temporal location of each semantic entity in its modal data stream is conducted. For the text modality, based on the narrative order of the sentence containing the semantic entity and the logical progression of the text, combined with explicit or implicit time words in the text, the occurrence time of the fact described by the semantic entity is determined, and this time point is mapped to the occurrence time sequence position on the global time axis. For the image modality, for semantic entities within static images, the image acquisition time is directly used as the baseline occurrence time of all contained semantic entities. For semantic entities in video streams, the video time calculated from the frame number where the semantic entity first appears in the video frame is used as its occurrence time sequence. For the audio modality, the start timestamp of each speech or acoustic event semantic entity is its occurrence time sequence. For structured table-type data, the production time or insertion timestamp of the data record is used as the occurrence time sequence of the corresponding semantic entity. All time information is uniformly converted to Coordinated Universal Time (UTC) format and sorted to form the occurrence time sequence of each semantic entity in the global data sequence. For the extraction of subordinate levels, logical attribution levels are defined based on the inherent organizational structure of each modality of data. In text, nested levels of chapters, paragraphs, sentences, and phrases are identified, and each semantic entity belongs to the language unit that directly contains it. At the same time, based on semantic dependency relationships, it is determined whether a semantic entity is an attribute, component, or sub-event of another semantic entity, thereby marking its semantic subordinate level. In images, the level is determined by the inclusion relationship between scenes, foreground target groups, single targets, and target components. In audio, the subordinate level is reflected in the hierarchical structure of background sound tracks, mixed sound sources, and independent sound sources. In structured tables, the subordinate level is the tree structure and dimensional hierarchy of tables, records, and fields in the database schema definition. Thus, each semantic entity is assigned a descriptor with two dimensions: occurrence sequence and subordinate level.

[0023] S14. The specific implementation method for constructing a semantic anchor mapping network based on the association relationship of all semantic anchors, and generating a unified semantic index sequence covering the entire modality data according to the occurrence sequence and subordinate hierarchy of semantic entities is as follows: All generated semantic anchors are used as edges of the network, and semantic entities of each modality are used as nodes of the network to construct a cross-modal semantic anchor mapping network. In the semantic anchor mapping network, each node stores the content and meta-information of the corresponding semantic entity and marks its subordinate level label and global occurrence time value in its modality. The edges of the semantic anchor mapping network are associated with the type and credibility weight of the corresponding relationship. After the semantic anchor mapping network is constructed, a unified semantic index sequence covering all modal data is generated based on this network. The generation process follows a defined sorting rule: first, using the global timeline as the primary sorting key, semantic entities of all modalities are arranged in ascending order of their occurrence time. When multiple semantic entities have the same or nearly simultaneous occurrence time, the subordinate hierarchy is used as the secondary sorting key, and they are arranged from macroscopic to microscopic levels. This ensures that when describing the same event, scene-level semantic entities are arranged first, followed by object-level semantic entities, and finally attribute or component-level semantic entities. Semantic entities within the same level are sorted using modality priority and semantic entity identifiers. After being arranged according to this rule, each semantic entity in the sequence is assigned a unique hierarchical index code, which combines the time segment number, level depth, and sequence number to form a unified semantic index sequence. The unified semantic index sequence is an ordered list of semantic entities that covers all modal data and is semantically coherent. Each index item points to a specific node in the semantic anchor mapping network. All subsequent quality analysis, deviation tracking, and data repair steps can quickly locate and cross-reference multimodal information through this index. It should be noted that the multimodal large model used in this invention is built on the Transformer architecture, including a multimodal encoder layer, a cross-modal attention fusion layer, and a task decoder layer. The multimodal encoder layer uses a text encoder, a visual encoder, and an audio encoder to process the input sequences of the corresponding modalities, respectively. The text encoder uses a pre-trained large language model backbone network, the visual encoder uses a visual Transformer, and the audio encoder uses an audio Transformer based on Mel-spectral features. The feature sequences output by each encoder are fed into the cross-modal attention fusion layer, which is composed of multiple layers of cross-attention mechanisms and feedforward networks stacked alternately, enabling feature interaction and alignment between modalities. The output of the cross-modal attention fusion layer is connected to the task decoder layer, which includes a linear projection head for bias type classification and bias boundary localization. The training of the multimodal large model is divided into two stages: The first stage is cross-modal pre-training, which uses a large-scale image-text-speech alignment dataset to train the model through contrastive learning, masked modality modeling, and image-text-speech matching tasks, learning a unified cross-modal semantic representation; The second stage is fine-tuning for data governance tasks, which uses a manually constructed multimodal dataset containing known latent quality biases and their annotations to supervise the training of the model for bias identification and bias propagation path inference, optimizing the cross-entropy loss and cross-union ratio loss, enabling the model to master the ability to identify semantic conflicts, logical breaks, context mismatches, and attribute missingness; During fine-tuning, the input is constructed as a single-modal data fragment and its corresponding unified semantic index context representation, and the output is a joint prediction result of bias type, bias location, and bias influence range.

[0024] In this embodiment of the invention, the detailed implementation steps of S2, which is based on a unified semantic index sequence and identifies latent quality biases in the data modally by modally through a multimodal large model, to generate single-modal bias distribution maps corresponding to each modality, include: S21. The specific process of importing single-modal data with the corresponding unified semantic index into a multimodal large model, and having the multimodal large model identify implicit quality biases such as semantic conflicts, logical breaks, context mismatches, and missing attributes in the data, is as follows: In the actual execution of deviation identification, the single-modal data to be detected is combined with its corresponding index fragment in the unified semantic index sequence to construct the model input: the index items of the semantic entities contained in the current single-modal data fragment are extracted from the unified semantic index sequence, and the index items, together with the summary information of other modal semantic entities pointed to by the adjacent preceding and following index items, constitute the semantic context description text; the semantic context description text and the original representation data of the single-modal data are input together into the fine-tuned multimodal large model; inside the multimodal large model, the single-modal data extracts features through the corresponding encoder, the semantic context description text extracts features through the text encoder, and they interact in the cross-modal attention fusion layer; During inference, multimodal large models rely on their learned cross-modal knowledge to determine whether there are inconsistencies between the current single-modal data and the semantic context transmitted from other modalities. Specifically, when the model finds that the attribute description of a semantic entity in the current data contradicts the cross-modal reference attribute of the same semantic entity in the index context, the model identifies it as a semantic conflict; when the model detects a break in the chronological order of events or the causal chain, and the current data cannot connect with the context, it identifies it as a logical discontinuity; when the model determines that the terminology, style, or scene setting used in the current data does not match the cross-modal context, it identifies it as a context mismatch; when the model finds that a necessary attribute explicitly mentioned in the context is missing in the current modal data, it identifies it as an attribute missing. The identification results of these implicit quality biases are output as bias category labels and corresponding confidence scores through the decoder header.

[0025] S22. Label the location and type of each latent quality deviation in the corresponding modal data, and generate a single-modal deviation distribution map according to the influence range of the deviation; wherein, the influence range of the deviation is the number of semantic entities affected by a single latent quality deviation and the coverage length of the corresponding data segment; Specifically, after the multimodal large model outputs the deviation identification results, the position of each latent quality deviation in the corresponding modality data is marked according to the boundary information predicted by the decoder head. For text modality, the position is marked as the start and end character offsets of the deviation-related semantic entity and the sentence number. For image modality, the position is represented by the bounding box coordinates or pixel region mask corresponding to the deviation semantic entity. For audio modality, the position is defined by the start and end timestamps of the deviation semantic entity in the audio stream. For structured table data, the position is specified down to the table name, field name, and row number. Simultaneously, the bias category labels predicted by the model—namely, semantic conflict, logical discontinuity, context mismatch, or attribute loss—are attached to the corresponding positions. Association analysis is performed on semantic entities whose semantic changes or information failures are directly or indirectly caused by the bias, and the number of affected semantic entities is counted. Based on the extent of the bias's spread in the continuous data stream, the coverage length of the corresponding data segment is calculated: for the text modality, the total number of consecutive characters from the bias's starting entity to the bias's ending entity is used as the coverage length; for the image modality, the area of ​​the pixel region covered by the bias entity's bounding box is calculated, and if the bias affects multiple consecutive... For connected regions, the surfaces of each connected region are accumulated. For audio modalities, the duration between the start and end timestamps of the deviation is calculated as the coverage length. For structured tables, the coverage length is the product of the number of rows and fields affected by the deviation. Using the data space of this modality as a canvas, a single-modal deviation distribution map is constructed. This map depicts the spatial or temporal location of the deviation, the deviation type label, and the heat rendered by the influence range in a visual or structured data form, showing the distribution pattern of quality defects within the single modality, and providing initial nodes and basic vectors for subsequent cross-modal transmission analysis.

[0026] In this embodiment of the invention, the specific implementation of S3, which traces the cross-modal deviation propagation link along the association path of the semantic anchor mapping network based on the single-modal deviation distribution map, analyzes the coupling amplification effect of deviation in the cross-modal transmission process, and generates the transmission risk level corresponding to each cross-modal deviation propagation link, includes: S31. Define the cross-modal semantic entity association path formed by the chain of semantic anchors in the semantic anchor mapping network as the association path, and extend and trace to other modalities along the association path in the semantic anchor mapping network; take each deviation node in the single-modal deviation distribution map as the starting deviation node, identify other modal derived deviations caused by the starting deviation node, and connect the starting deviation node with all derived deviation nodes to form a cross-modal deviation transmission link. Specifically, in the semantic anchor mapping network, all continuous paths formed by connecting semantic anchors end to end are identified, and all continuous paths are defined as association paths. For example, if there exists a text entity A connected to an image entity B through semantic anchor 1, and the image entity B is connected to an audio entity C through semantic anchor 2, then the cross-node path from text entity A to image entity B and then to audio entity C constitutes an association path. Each association path represents the expression chain of a specific factual information in multimodal data. When tracing the propagation of cross-modal deviations, the deviation nodes in the single-modal deviation distribution map are used as the starting deviation nodes. These starting deviation nodes are located in the semantic anchor mapping network. Starting from the starting deviation node, the process extends outward along all semantic anchor edges connected to it to the adjacent nodes of other modalities, and recursively continues to traverse along the semantic anchor edges until the end node with no semantic anchor connection is reached, or all reachable cross-modal semantic entities are traversed. For each initial deviation node, examine the corresponding deviation nodes in other modalities reached along the associated path; re-invoke the multimodal large model to perform directional deviation detection on the corresponding semantic entities and their contextual data in other modalities, and determine whether a new deviation appears at the semantic entity that has a causal or logical connection with the initial deviation; if a context mismatch deviation is found at the corresponding semantic entity in the image modality, it is determined that the deviation is a derivative deviation caused by the initial deviation in the text modality; after identifying the derivative deviation, the initial deviation node is used as the head node of the propagation link, and all the identified derivative deviation nodes are connected in the order of their connection on the associated path to form a complete directed cross-modal deviation propagation link; this link clearly records the specific path of the deviation from a certain modality, propagating across modalities through semantic anchors, and inducing new deviations in other modalities; this structure is stored as a combined representation of node sequence and edge sequence.

[0027] S32. For the deviation influence range of each deviation node in the cross-modal deviation transmission link, identify the deviation node whose deviation influence range expands as the coupling amplification node, and analyze the process of obtaining the coupling amplification effect of deviation in cross-modal transmission as follows: After forming a cross-modal deviation propagation link, for each deviation node on the link, the pre-calculated deviation influence range value is extracted, including the number of affected semantic entities and the coverage length of the data segment; along the cross-modal deviation propagation link, starting from the initial deviation node, the influence range of two adjacent deviation nodes is compared one by one; when it is found that the influence range of a downstream deviation node is significantly expanded compared to its direct upstream deviation node, that is, the number of affected semantic entities increases, or the coverage length of the data segment increases, then the downstream deviation node is marked as a coupling amplification node; After identifying the coupling amplification node, we conducted an in-depth analysis of the underlying causes of the amplification. We extracted the modal data at the coupling amplification node and its associated preceding bias context. We found that when the bias is transmitted to the new modality corresponding to the coupling amplification node, due to the expressive characteristics of this modality, a single semantic error may interact and bind with a large number of originally normal semantic entities within the modality, causing the erroneous information to spread rapidly within the modality. For example, a missing attribute bias in text, when transmitted to an image, causes an overall misunderstanding of the image scene, thus affecting all objects related to the scene in the image. We recorded this phenomenon of the nonlinear expansion of the bias's influence range during cross-modal transmission as the coupling amplification effect, and bound the analysis results of this effect to the coupling amplification node for subsequent risk assessment.

[0028] S33. The process of calculating the total number of deviation nodes and the number of coupling amplification nodes in each cross-modal deviation propagation link is as follows: Perform parameter statistics on each cross-modal deviation propagation link; traverse the deviation node sequence of the cross-modal deviation propagation link, accumulate the length of the deviation node sequence, and obtain the total number of deviation nodes contained in the cross-modal deviation propagation link; at the same time, count the number of nodes in the cross-modal deviation propagation link that are explicitly marked as coupling amplification nodes; when counting, the calculation of the total number of deviation nodes is limited to all entity nodes with quality deviation markings in the cross-modal deviation propagation link, excluding semantic anchor nodes that are only used for concatenation but have no quality problems themselves.

[0029] S34. Combining the number of modalities covered by the cross-modal deviation transmission link with the severity of the deviation type, all transmission links are divided into three transmission risk levels: high, medium, and low. Among them, the deviation type corresponds to the four types of latent quality deviations identified: semantic conflict, logical discontinuity, context mismatch, and attribute loss. Different deviation types correspond to different degrees of severity. Understandably, the sequence of deviation nodes in each cross-modal deviation propagation link is read, the modality identifier of each deviation node is extracted, and after deduplication, the number of modalities covered by the cross-modal deviation propagation link is counted. The number of modalities includes the total number of modal types from which all deviation nodes in a cross-modal deviation propagation link originate. For example, if the deviation nodes in a cross-modal deviation propagation link only involve two modalities, namely text and image, then the number of modalities covered is 2; if they involve three modalities, namely text, image, and audio, then the number of modalities covered is 3. Then, a predefined deviation type severity mapping table is introduced, which quantifies four types—semantic conflict, logical discontinuity, context mismatch, and attribute missing—as a severity score. Semantic conflict and logical discontinuity have the highest severity scores, followed by attribute missing, and context mismatch has the lowest. For each cross-modal deviation transmission link, the highest severity score among all deviation nodes is taken as the type severity of that link. Next, a comprehensive risk score is calculated. The risk score is a weighted sum of the normalized values ​​of the number of covered modes, the normalized value of the total number of deviation nodes, the normalized value of the number of coupled amplification nodes, and the normalized value of the type hazard. Based on the range in which the risk score falls, all cross-modal deviation transmission links are divided into three transmission risk levels: high, medium, and low. Among them, the high risk level represents deviations that propagate across multiple modes, amplify multiple times, and have high-hazard deviation types; the medium risk level represents deviations that propagate across two modes or have a single amplification phenomenon; and the low risk level represents deviations that have a weak cross-modal impact or are limited to a slight spread within a single mode.

[0030] In this embodiment of the invention, the detailed implementation steps of S4, which involves matching a graded governance strategy to the transmission risk level of each cross-modal deviation transmission link, performing directional deviation correction on multimodal data, and calibrating cross-modal semantic consistency, include: S41. For low-risk transmission links, an in-situ correction strategy is adopted to independently correct each deviation node in the low-risk transmission link. Understandably, when a cross-modal deviation propagation link is assessed as low-risk, it indicates that the deviation coupling amplification effect of the link is weak and the dependence between deviation nodes is not strong. An in-situ correction strategy is adopted for low-risk propagation links. Each deviation node on the low-risk propagation link is traversed, with the local context of the single-modal data where the deviation node is located as the working area. The generative capabilities of the multimodal large model are used, allowing the model to utilize unbiased data within the modality and a few cross-modal reference semantic entity information pointed to by a unified semantic index as correction clues. For semantically conflicting deviation nodes, the multimodal large model is instructed to rewrite conflicting text fragments or replace contradictory attribute areas in the image to ensure consistency with the reference information. For deviation nodes with missing attributes, the missing attribute values ​​are inferred and filled in based on the context. The correction action is limited to the scope of the deviation's influence and does not propagate to modify other deviation node data that are not directly related to the deviation node's semantic anchor point. After each deviation node is corrected, local consistency verification is immediately performed on the deviation node to confirm that the correction result only changes the erroneous part and does not introduce new contradictions.

[0031] S42. For medium-risk transmission links, a link correction strategy is adopted, with the coupling amplification node as the core to correct the deviation of the entire transmission link in a coordinated manner. Understandably, the link correction strategy focuses on medium-risk transmission links where there are clearly defined coupling amplification nodes, and deviations are amplified due to the amplification effect, thus requiring coordinated correction. First, the marked coupling amplification nodes in the medium-risk transmission link are located. Using the modal data where the coupling amplification node is located as the core governance region, a joint correction view is constructed, containing all modal data fragments associated with the coupling amplification node and its upstream and downstream deviation nodes. Based on a semantic anchor mapping network, the initial deviation source semantics and the erroneous semantic patterns after coupling amplification in the medium-risk transmission link are extracted to guide the generation of correction schemes in the multimodal large model. The correction scheme prioritizes correcting the initial deviation from the source, while simultaneously adjusting the distorted semantic expressions at the coupling amplification nodes to prevent them from propagating errors downstream. Along the forward sequence of the medium-risk transmission link, each node is verified and fine-tuned to ensure semantic consistency among all deviation nodes after the amplification effect is eliminated.

[0032] S43. For high-risk transmission links, a global calibration strategy is adopted to suspend all data flow associated with the corresponding link, and then perform semantic benchmark unification on the entire link data based on the semantic anchor mapping network before making point-by-point corrections; wherein, all modal data involved in the entire transmission link are defined as the entire link data. Specifically, firstly, a pause command is issued to suspend all data flow associated with the high-risk transmission link. Data flow refers to the process of transferring relevant data in the pending, transmitted, and stored states across the various modalities covered by the high-risk transmission link, preventing defective data from further contaminating downstream businesses. Subsequently, all modal-related data involved in the entire high-risk transmission link, i.e., the full-link data, are included in the governance scope. In the semantic anchor mapping network of the high-risk transmission link, a set of unbiased, high-quality semantic entities are selected as semantic benchmarks. Using these semantic benchmarks as the core, semantic benchmark unification is performed on the full-link data: all semantic entities are forcibly aligned across modal references to the attribute descriptions, spatiotemporal identifiers, and relational definitions of the benchmark semantic entities, resetting any offset semantic coordinates. After benchmark unification is completed, each deviation node in the high-risk transmission link is traversed and corrected according to a point-by-point correction process.

[0033] S44. Select semantic entities with complete semantic information and no quality deviation in the semantic anchor mapping network as the benchmark semantic entities. Using the benchmark semantic entities in the semantic anchor mapping network as a reference, perform semantic alignment on the corrected modal data, including: A baseline semantic entity is selected in the semantic anchor mapping network; the entire semantic anchor mapping network is scanned, and the quality of all semantic entities is evaluated. Semantic entities with complete attribute labels, complete feature vectors, high mutual information with adjacent anchor semantic entities, and not marked as any deviation nodes in the single-modal deviation distribution map are selected as the baseline semantic entities; after selection, the baseline semantic entity is established as the reference origin for semantic alignment. Subsequently, targeting the semantic entities in each modal data after correction, the semantic offset between them and the corresponding baseline semantic entities is calculated. The semantic offset is obtained by a weighted sum of three factors: attribute difference, spatiotemporal deviation, and hierarchical misalignment. The attribute difference is calculated by comparing the intersection-union ratio of the attribute label sets of the two entities and the normalized value of the difference in numerical attributes. The spatiotemporal deviation is calculated by summing the timestamp difference and the spatial Euclidean distance after normalization. The hierarchical misalignment is calculated by comparing and normalizing the depth difference of the subordinate levels of the two entities. At the same time, the weights of the three components are preset according to the application scenario. The calculation process measures the difference in attribute vectors, spatiotemporal coordinate offset, and the degree of inconsistency in description granularity. Based on the calculated semantic offset components, specific transformation functions are generated for each component. The attribute transformation function defines a direct assignment mapping from the current attribute value to the base attribute value. For numerical attributes, a linear scaling mapping is used, with the scaling ratio determined by the attribute difference, maintaining the dimensionless value. The spatiotemporal transformation function consists of a time translation and a spatial affine transformation matrix. The time translation is directly taken from the time difference portion of the spatiotemporal deviation, and the spatial affine transformation matrix contains rotation, scaling, and translation parameters, derived from the spatial deviation components, ensuring that the dimensions before and after the spatial coordinate transformation are consistent with the unit of length. The hierarchical transformation function is a reattachment operation that adjusts the entity's subordinate hierarchy depth to the base hierarchy depth. The physical meaning of all transformation operations directly corresponds to the elimination of attribute descriptions. Differences and spatiotemporal information deviations are addressed by explicitly deriving transformation parameters from offsets, avoiding the misuse of dimensionless parameters. Three transformation functions are applied sequentially to the semantic entities to be aligned, respectively performing attribute value replacement, spatiotemporal coordinate transformation, and subordinate hierarchy relocation. After transformation, each transformed entity is verified against the baseline semantic entity to ensure they meet preset alignment tolerance thresholds in attribute labels, spatiotemporal stamps, and hierarchy depth. These tolerance thresholds are determined based on the nominal accuracy of each modal data acquisition device and the stringency of semantic consistency required by the business scenario: attribute label alignment requires complete consistency with a tolerance of 0; spatiotemporal stamp alignment tolerance is half the root mean square value of the acquisition device's synchronization error; and hierarchy depth alignment tolerance is set to 0. If these conditions are not met, the transformation parameters are iteratively adjusted until the tolerance conditions are satisfied.

[0034] S45. Eliminate the differences in attribute descriptions and spatiotemporal information deviations of different modal data for the same semantic entity, and ensure that the semantic expression of the same semantic entity is consistent in all associated modalities; where, attribute description differences refer to the inconsistency of feature descriptions of the same semantic entity in different modalities, and spatiotemporal information deviations refer to the mismatch of time and location identifiers of the same semantic entity in different modalities. In the semantic alignment process, the focus is on eliminating differences in attribute descriptions and spatiotemporal information deviations. Regarding attribute description differences—that is, inconsistencies in the feature representations of the same semantic entity across different modalities—attributes of corresponding semantic entities in other modalities are overridden or merged based on the attribute set of the baseline semantic entity. For example, if the text modal describes a device as red, while the image modal extracts a dark red color feature through object detection, the attribute label in the image modal is unified to red, based on the red color attribute of the baseline semantic entity. Alternatively, a description can be added explaining that dark red is a specific lighting condition, thus eliminating feature representation discrepancies at their source. Regarding spatiotemporal information deviations—that is, mismatches in the time and location identifiers of the same semantic entity across different modalities—the timestamps of all semantic entities are uniformly converted to the same baseline time zone and time precision, and corrected based on the spatiotemporal stamp of the baseline semantic entity. For spatial coordinates, coordinate transformation is performed using calibration parameters and geographic references from multimodal acquisition to eliminate positional deviations caused by GPS drift or image positioning errors.

[0035] In this embodiment of the invention, the specific implementation of step S5, which involves feeding back the semantically consistent calibrated multimodal data into a large multimodal model for semantic consistency verification, generating a standardized governance dataset after successful verification, and synchronously updating the semantic anchor mapping network, is as follows: S51. The specific steps for synchronously inputting the calibrated multimodal data into the multimodal large model, triggering the multimodal large model to perform cross-modal semantic reasoning based on the unified semantic index, and then obtaining the reasoning results are as follows: All modal data from the same batch, after correction and alignment, are synchronously fed into the multimodal large model according to their unified semantic index sequence. A cross-modal semantic reasoning instruction is issued to the multimodal large model, which requires the model to use each semantic anchor fact specified in the unified semantic index as a reasoning unit, jointly extract information from text, images, audio, and structured data, and deduce and generate a complete and self-consistent set of semantic propositions. When the model is performing reasoning, its cross-modal attention fusion layer will force the features corresponding to the same semantic index items in different modalities to be mapped to the same semantic subspace and check whether there is any mutually exclusive information. The form of the reasoning result is a set of structured semantic statement triples. Each triple describes an entity, attribute, and relationship, and is accompanied by evidence supporting the modality extraction. For example, the model may infer that device A is in an overheated state, and its evidence comes from text alarm records, thermal imaging highlighted areas in images, and alarm voices in audio.

[0036] S52. If there are no semantic contradictions or information conflicts in the reasoning results, the verification is deemed successful and a standardized governance dataset is generated; if there are semantic contradictions or information conflicts, the transmission link corresponding to the contradiction is located and the corresponding hierarchical governance strategy is returned to be corrected. Understandably, after the multimodal large model outputs inference results, it automatically scans the set of semantic statement triples in the inference results; it checks whether there are two or more triples that make logically mutually exclusive statements about the same semantic entity or event under the same semantic index. For example, one triple states that device A is in a normal state, while another triple states that device A is in an abnormal state. If no such semantic contradictions or information conflicts are found, the verification is considered successful. At this point, the multimodal data that has undergone a complete governance process and is semantically consistent is packaged according to a preset standard format to generate a standardized governance dataset. Each data in this dataset... Each item carries a reliable unified semantic index and complete cross-modal reference information. If a semantic contradiction or information conflict is detected, the semantic entity associated with the contradiction triple is traced back to the semantic anchor mapping network to determine the cross-modal deviation transmission link where the semantic entity is located. The governance record of the specific transmission link is retrieved, it is remarked as not repaired, and the relevant data is automatically returned to the hierarchical governance strategy link corresponding to the risk level of the link. That is, low-risk links return to in-situ correction, medium-risk links return to link correction, and high-risk links return to global calibration for iterative repair until the verification is passed.

[0037] S53. Extract newly added semantic entities from the validated standardized governance dataset, supplement the semantic anchor mapping network with the newly added semantic entities, and establish their association with existing semantic entities. The specific method is as follows: After generating the standardized governance dataset, the semantic entity sets before and after governance are compared to identify semantic entities newly generated during the governance process due to the completion of missing attributes and the resolution of conflicts, or semantic entities newly discovered during cross-modal alignment but not originally extracted. The newly added semantic entities are extracted, and their deviation nodes are added to the semantic anchor mapping network. When adding, based on the semantic context and unified semantic index position of the new semantic entity in the standardized governance dataset, existing semantic entities with attribute co-referentiality, spatiotemporal similarity, or hierarchical inclusion relationship with it are searched. The attribute label and spatiotemporal identifier matching method is reused to automatically establish a one-to-one or one-to-many semantic anchor association relationship between the new semantic entity and the existing semantic entities, and the newly generated edges are added to the semantic anchor mapping network.

[0038] S54. Pre-set weight priorities for each associated path in the semantic anchor mapping network, where the weight priorities are used to characterize the probability and importance of deviation propagation of the corresponding path. Specifically, after the initial construction or each update of the semantic anchor mapping network, a weight priority is pre-calculated and set for each associated path in the semantic anchor mapping network. The weight priority is used to characterize the probability and importance of deviation propagation of the corresponding path. The setting process is divided into initial assignment and dynamic calibration. During the initial assignment, the semantic coupling strength between modalities is quantified. For example, the forced correspondence between text and structured table data usually has a high propagation probability and is given a higher initial weight; while the indirect association between images and audio inferred from the scene has a relatively low propagation probability and is given a lower initial weight. Meanwhile, the weights are modulated based on the criticality of semantic entities on the associated paths, such as whether they are high-frequency core entities. During dynamic calibration, historical data governance records are used to count the frequency of cumulative deviation propagation on each associated path, and the initial weights are corrected using the frequency values. This weight priority is stored as an attribute of the associated path.

[0039] S55. Based on the actual associated paths traversed by the cross-modal propagation of deviation nodes during this governance process, optimize the weight priority of each associated path in the semantic anchor mapping network. Understandably, the actual path trajectories of all identified cross-modal deviation propagation links during this governance process are extracted, i.e., the associated paths that deviation nodes actually traverse during cross-modal propagation; the path identifiers are recorded in the governance log; subsequently, for each associated path where deviation propagation actually occurs, a weight enhancement operation is performed, specifically increasing its weight priority value, with the increase being positively correlated with the severity of the propagation and the risk level of the propagation link; while for associated paths that did not carry any deviation propagation in this governance but were previously predicted to be high priority, a small weight decay is performed to prevent outdated empirical weights from becoming fixed.

[0040] Please see Figure 2 This invention provides an automatic data governance system based on a multimodal large model, comprising: Multimodal semantic index construction module: It accesses multimodal data from multiple sources and heterogeneous sources, extracts the core semantic entities from each modality, constructs a semantic anchor mapping network between modalities, and generates a unified semantic index sequence covering all modal data; Single-modal quality deviation identification module: Based on a unified semantic index sequence, it identifies latent quality deviations in the data modally by modally through a multimodal large model, and generates single-modal deviation distribution maps corresponding to each modality; Cross-modal deviation link tracing module: Based on the single-modal deviation distribution map, it traces the cross-modal deviation transmission link along the association path of the semantic anchor mapping network, analyzes the coupling amplification effect of deviation in the cross-modal transmission process, and generates the transmission risk level of each cross-modal deviation transmission link; The hierarchical deviation correction and calibration module matches a hierarchical governance strategy based on the transmission risk level of each cross-modal deviation transmission link, performs directional deviation correction on multimodal data, and performs cross-modal semantic consistency calibration; among which, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy, and global calibration strategy; Dataset verification and network update module: The multimodal data after semantic consistency calibration is fed back into the multimodal large model to perform semantic consistency verification. After the verification is passed, a standardized governance dataset is generated and the semantic anchor mapping network is updated synchronously.

[0041] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0042] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0043] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An automatic data governance method based on a multimodal large model, characterized in that, include: S1. Access multi-source heterogeneous multimodal data, extract core semantic entities from each modality, construct a semantic anchor mapping network between modalities, and generate a unified semantic index sequence covering all modalities, including: Semantic entity extraction is performed on multimodal data such as text, images, audio, and structured tables to obtain independent semantic entity sets for each modality; Using the attribute tags and spatiotemporal identifiers of semantic entities as the matching benchmark, a one-to-one or one-to-many correspondence between semantic entities of different modalities is established as a semantic anchor point. Extract the time order of occurrence of each semantic entity in the corresponding multimodal data as the occurrence sequence, and extract the logical belonging level of each semantic entity in its respective modal data as the subordinate level; A semantic anchor mapping network is constructed based on the association relationship of all semantic anchors, and a unified semantic index sequence covering the entire modality data is generated according to the occurrence time and subordinate hierarchy of semantic entities. S2. Based on the unified semantic index sequence, latent quality deviations in the data are identified modally by modality through a multimodal large model, and single-modal deviation distribution maps corresponding to each modality are generated. S3. Based on the single-modal deviation distribution map, trace the cross-modal deviation propagation links along the association paths of the semantic anchor mapping network, analyze the coupling amplification effect of deviations in the cross-modal transmission process, and generate the transmission risk level of each cross-modal deviation propagation link, including: The cross-modal semantic entity association path formed by the concatenation of semantic anchors in the semantic anchor mapping network is defined as the association path. The association path in the semantic anchor mapping network is extended and traced to other modalities. Each deviation node in the single-modal deviation distribution map is taken as the starting deviation node. Other modal derivative deviations caused by the starting deviation node are identified, and the starting deviation node and all derivative deviation nodes are connected to form a cross-modal deviation transmission link. The influence range of each deviation node in the cross-modal deviation transmission link is analyzed, and the deviation node with the expanded influence range is identified as the coupling amplification node. The coupling amplification effect of deviation in cross-modal transmission is then analyzed. Calculate the total number of deviation nodes and the number of coupling amplification nodes in each cross-modal deviation propagation link; Based on the number of modalities covered by the cross-modal deviation transmission links and the severity of the deviation types, all transmission links are divided into three transmission risk levels: high, medium, and low. Among them, the deviation types correspond to four types of latent quality deviations identified: semantic conflict, logical discontinuity, context mismatch, and attribute loss. Different deviation types correspond to different degrees of severity. S4. Match a hierarchical governance strategy to the transmission risk level of each cross-modal deviation transmission link, perform directional deviation correction on the multimodal data, and perform cross-modal semantic consistency calibration; wherein, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy and global calibration strategy. S5. Feed the semantically consistent calibrated multimodal data back into the multimodal large model to perform semantic consistency verification. After the verification is passed, generate a standardized governance dataset and update the semantic anchor mapping network simultaneously.

2. The automatic data governance method based on a multimodal large model according to claim 1, characterized in that, S2 includes: The single-modal data is combined with the corresponding unified semantic index and imported into the multimodal big model. The multimodal big model identifies implicit quality biases such as semantic conflicts, logical breaks, context mismatches, and missing attributes in the data. The location and type of each latent quality deviation in the corresponding modal data are labeled, and a single-modal deviation distribution map is generated according to the influence range of the deviation; wherein, the influence range of the deviation is the number of semantic entities affected by a single latent quality deviation and the coverage length of the corresponding data segment.

3. The automatic data governance method based on a multimodal large model according to claim 1, characterized in that, S4 includes: For low-risk transmission links, an in-situ correction strategy is adopted to independently correct each deviation node in the low-risk transmission link; For medium-risk transmission links, a link correction strategy is adopted, which uses the coupling amplification node as the core to correct the deviation of the entire transmission link in a coordinated manner; For high-risk transmission links, a global calibration strategy is adopted to suspend all data flow associated with the corresponding link. The semantic benchmark of the entire link data is unified based on the semantic anchor mapping network and then corrected point by point. Among them, all modal data involved in the entire transmission link are defined as the entire link data.

4. The automatic data governance method based on a multimodal large model according to claim 3, characterized in that, S4 further includes: In the semantic anchor mapping network, semantic entities with complete semantic information and no quality deviation are selected as the benchmark semantic entities. Using the benchmark semantic entities in the semantic anchor mapping network as a reference, semantic alignment is performed on the corrected modal data. Eliminate the differences in attribute descriptions and spatiotemporal information deviations of different modal data for the same semantic entity; where attribute description differences refer to the inconsistency in the feature representations of the same semantic entity by different modalities, and spatiotemporal information deviations refer to the mismatch in the time and location identifications of the same semantic entity by different modalities.

5. The automatic data governance method based on a multimodal large model according to claim 1, characterized in that, S5 includes: The calibrated multimodal data is synchronously input into the multimodal large model, triggering the multimodal large model to carry out cross-modal semantic reasoning based on the unified semantic index, and then obtain the reasoning results; If there are no semantic contradictions or information conflicts in the reasoning results, the verification is deemed successful and a standardized governance dataset is generated; if there are semantic contradictions or information conflicts, the transmission link corresponding to the contradiction is located and the corresponding hierarchical governance strategy is returned for correction.

6. The automatic data governance method based on a multimodal large model according to claim 1, characterized in that, The S5 also includes: Extract newly added semantic entities from the validated standardized governance dataset, supplement the semantic anchor mapping network with the newly added semantic entities, and establish their association with existing semantic entities; Weights and priorities are pre-assigned to each associated path in the semantic anchor mapping network, where the weights and priorities are used to characterize the probability and importance of deviation propagation in the corresponding path; Based on the actual associated paths traversed by the deviation nodes during this governance process across modalities, the weight priority of each associated path in the semantic anchor mapping network is optimized.

7. An automatic data governance system based on a multimodal large model, characterized in that, An automatic data governance method based on a multimodal large model, as described in any one of claims 1-6, comprises: Multimodal semantic index construction module: It accesses multimodal data from multiple sources and heterogeneous sources, extracts the core semantic entities from each modality, constructs a semantic anchor mapping network between modalities, and generates a unified semantic index sequence covering all modal data; Single-modal quality deviation identification module: Based on a unified semantic index sequence, it identifies latent quality deviations in the data modally by modally through a multimodal large model, and generates single-modal deviation distribution maps corresponding to each modality; Cross-modal deviation link tracing module: Based on the single-modal deviation distribution map, it traces the cross-modal deviation transmission link along the association path of the semantic anchor mapping network, analyzes the coupling amplification effect of deviation in the cross-modal transmission process, and generates the transmission risk level of each cross-modal deviation transmission link; The hierarchical deviation correction and calibration module matches a hierarchical governance strategy based on the transmission risk level of each cross-modal deviation transmission link, performs directional deviation correction on multimodal data, and performs cross-modal semantic consistency calibration; among which, the hierarchical governance strategy includes in-situ correction strategy, link correction strategy, and global calibration strategy; Dataset verification and network update module: The multimodal data after semantic consistency calibration is fed back into the multimodal large model to perform semantic consistency verification. After the verification is passed, a standardized governance dataset is generated and the semantic anchor mapping network is updated synchronously.

Citation Information

Patent Citations

  • Automatic data management method and system based on large model

    CN121166662A

  • Data annotation platform based on cross-domain multi-modal data governance

    CN122414302A