Asset semantic automatic labeling method and system for multi-modal data

Abstract

The application discloses a kind of asset semantic automatic labeling method and system for multi-modal data.The method is preprocessed to multi-source heterogeneous data and adaptively key frame sampling;Multi-modal features are fused using multi-head cross-modal attention mechanism of causal mask, and are mapped to unified semantic space through prototype contrast learning, to enhance the distinguishing degree of representation;A differentiable ontology reasoning network is constructed, label matching and confidence enhancement are carried out in combination with tree-shaped long short-term memory network and neural symbolic reasoning rules, and confidence is bidirectionally propagated along the semantic label tree;Based on Shapley value weighted multi-agent, collaborative verification is carried out, and decision-making strategy is optimized in combination with self-play reinforcement learning;Finally, a semantic knowledge graph is constructed, and elastic weight is used to consolidate and realize continuous learning of the model.The application effectively improves the causal logic, knowledge reasoning generalization ability and multi-agent decision fairness of multi-modal semantic labeling.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for automatic semantic annotation of assets for multimodal data. Background Technology

[0002] With the deepening of enterprise digital transformation, data asset management has become one of the core tasks of enterprise informatization construction. Currently, enterprise data assets exhibit significant characteristics of being multi-source, heterogeneous, and multimodal. They not only include traditional structured data such as relational database tables, but also widely involve various unstructured or semi-structured data forms such as text reports, images, audio and video, and semi-structured documents. To improve the usability and understandability of data assets, asset semantic annotation technology has emerged. Its purpose is to assign machine-readable and human-understandable semantic tags to various data assets, thereby supporting upper-level applications such as data retrieval, data governance, and data sharing and exchange.

[0003] In existing technologies, asset labeling for multimodal data mainly adopts the following methods: First, rule-based methods, which design feature extraction rules and label mapping rules for different modalities, and then simply merge the labeling results of each modality; Second, methods based on single-modal deep learning models, which train independent classification or labeling models for text, image, audio and other modalities, and generate unified labels through post-concatenation or voting strategies; Third, methods based on keyword matching or template filling, which extract explicit words from metadata or data content as labels.

[0004] However, existing technologies, which rely on independent model annotation followed by later fusion, fail to establish deep semantic connections between cross-modal data in practical applications. This leads to new information silos among multimodal assets, making it difficult to achieve true semantic-level unified retrieval and association analysis. The semantic descriptions generated by existing annotation methods lack hierarchical structures and business knowledge guidance, often only outputting flat keyword lists. This fails to establish an effective mapping with the ontology system of the business domain to which the data assets belong, causing the annotation results to deviate from business semantics. Although the assets are "annotated," they are difficult for business personnel to understand and use. Furthermore, the semantic extraction capabilities for unstructured and semi-structured data are insufficient. A large amount of hidden dark data in documents, images, and audio / video remains outside the asset management system, making it difficult to obtain high-quality, interpretable semantic annotations. The accuracy of automatic annotation results lacks hierarchical verification and interpretability guarantees. While pure black-box deep learning methods have strong generalization capabilities, annotation errors are difficult to trace and correct, while pure rule-based methods cannot adapt to complex and ever-changing real-world business scenarios.

[0005] Therefore, how to automatically annotate multimodal data assets in an efficient, consistent, interpretable manner with business semantic relevance has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to perform efficient, consistent, interpretable and business semantic association automatic annotation of multimodal data assets. In view of the above-mentioned defects of the prior art, this invention provides an automatic asset semantic annotation method and system for multimodal data.

[0007] on the one hand An automatic semantic annotation method for assets in multimodal data includes the following steps: S1. Access and preprocessing of multi-source heterogeneous data assets: Collect multimodal data assets within the enterprise and perform standardized modal content parsing to obtain preprocessed multimodal datasets. S2. Construct a unified semantic representation space for modalities, and use a multi-head cross-modal attention mechanism with causal masking to fuse features from various modalities. Then, through cross-modal prototype contrastive learning, map the fused features to the unified semantic space to generate a unified semantic representation vector. The multi-head cross-modal attention mechanism with causal masking ensures that the attention weight of the i-th modality can only calculate features of the current and previous modalities. The cross-modal prototype contrastive learning uses learnable prototype vectors and a loss function that includes prototype attraction terms and prototype dispersion regularization terms. S3. Construct a hierarchical semantic reasoning network guided by a differentiable ontology. Embed concept nodes and relationships in the business ontology model as vectors. Use a tree-like long short-term memory network (TreeLSTM) to generate the embedding vector of each label node in the hierarchical semantic label tree. Calculate the matching score between the unified semantic representation vector and the embedding vector of each label node. The matching score introduces a depth decay factor. For candidate labels with matching scores below a first threshold but above a second threshold, apply a set of neural symbolic reasoning rules to enhance knowledge-guided reasoning. The reasoning rules are implemented using differentiable tensor operations. Iterate until convergence and record the reasoning path. S4. Perform bidirectional propagation of confidence from bottom to top and from top to bottom along the hierarchical semantic label tree structure, and determine the adopted label, rejected label, or label to be verified in the refinement step based on the final confidence after propagation. S5. A multi-agent chain reasoning refinement based on Shapley weighting is adopted. Four agents are set up to perform semantic consistency verification, hierarchical structure verification, cross-modal association verification and business rule verification respectively. The Shapley value of each agent for the labeling quality is calculated as the weight. The final labeling result is generated by weighted voting, and the policy network of each agent is updated by self-play reinforcement learning. S6. Associate the refined semantic annotation results with the business ontology model, construct a multimodal asset semantic knowledge graph, and store the reasoning path in an interpretable form; S7. Incremental continuous learning is performed on the models from S2 to S5 based on the labeled feedback data.

[0008] Preferably, the multi-head cross-modal attention mechanism of the causal mask in S2 is specifically as follows: For each modality's feature vector, a linear transformation matrix of query, key, and value is learned. Attention weights are obtained by calculating the similarity between the query and the key and applying a causal mask matrix. In the causal mask matrix, the mask value is negative infinity when the index of the target modality is less than the index of the source modality, and the mask value is zero otherwise. The value vectors are weighted and summed using the attention weights, and then normalized by layers to obtain the fused feature vectors of each modality.

[0009] Preferably, the cross-modal prototype comparison loss function in S2 includes a first term and a second term; the first term is used to make the unified semantic representation vector of the data asset closer to the corresponding semantic prototype vector; the second term is used to force the Euclidean distance between different semantic prototype vectors to be no less than the preset minimum spacing, and the weighted sum of the two terms constitutes the total loss.

[0010] Preferably, the method by which the TreeLSTM network in S3 generates the embedded vector of the label node is as follows: for each node in the hierarchical semantic label tree, the embedded vector of the parent node and the embedded vector of the corresponding business ontology concept are input into the TreeLSTM unit, and the embedded vector of the node is recursively calculated; the method for calculating the matching score is as follows: calculate the cosine similarity between the unified semantic representation vector and the embedded vector of the label node, and then multiply it by the depth decay factor, wherein the depth decay factor is a negative exponential function with the depth of the label node in the tree as the independent variable.

[0011] Preferably, each rule in the neural symbol reasoning rule set in S3 has the form of deriving a conclusion if the premise condition is true. Specifically, the confidence of the current candidate label is represented as an activation vector. The premise condition of each rule is calculated by the minimum value of the activation values ​​of multiple premise concepts to determine the rule satisfaction level. The activation value of the conclusion concept is updated to the larger of the original activation value and the rule satisfaction level. All rules are applied iteratively until the activation vector converges. New candidate labels and corresponding reasoning paths are generated based on the converged activation vector.

[0012] The bidirectional propagation of confidence in S4 specifically includes: during bottom-up propagation, the confidence of the child node is multiplied by a preset upward propagation weight and compared with the current confidence of the parent node, and the larger value is used to update the parent node's confidence; during top-down propagation, the confidence of the parent node is multiplied by a preset downward propagation weight and compared with the current confidence of the child node, and the larger value is used to update the child node's confidence; the upward propagation weight and the downward propagation weight are preset according to the ontology relationship type.

[0013] Preferably, the multi-agent decision-making based on Shapley value weighting in S5 specifically involves: treating the four agents as participants in a cooperative game; pre-defining a value function to evaluate the comprehensive labeling performance of any subset of agents on the validation set; for each agent, calculating the average of its marginal contribution across all possible subsets as the agent's Shapley value; and finally, the labeling result is the set of labels that maximizes the weighted sum of the Shapley values ​​of each agent's approval level within all candidate label sets.

[0014] Preferably, the self-play reinforcement learning step in S5 is as follows: In a simulated environment, four agents take turns playing the roles of labeler and discriminator. The discriminator introduces conflicts and errors into the existing labeled results according to preset rules, and the labeler attempts to correct and generate new labeled results. The reward function of reinforcement learning is composed of the product of precision, recall and inference path completeness. The policy network parameters of each agent are updated using a proximal policy optimization algorithm.

[0015] Preferably, the construction of the multimodal asset semantic knowledge graph in S6 includes: treating each data asset as a node, treating the semantic label corresponding to the asset as another type of node, using resource description framework triples to represent the label relationship between the asset node and the label node, and attaching confidence scores and explanatory links pointing to inference path nodes to the relationship edges; the inference path nodes record rule triggering sequences, attention weight distribution, and confidence propagation trajectories.

[0016] Preferably, the incremental continuous learning in S7 adopts the elastic weight consolidation method, and a regularization term is added when optimizing the loss function of new samples. The regularization term is: for each model parameter, the Fisher information value of the parameter in the old task is multiplied by the square of the difference between the current parameter and the old task parameter, and then multiplied by a preset regularization coefficient. The sum of all parameters is used as the regularization term.

[0017] on the other hand An automatic semantic annotation system for assets in multimodal data, for performing any of the methods described herein, comprising: The data access and preprocessing module is configured to collect multimodal data assets, perform format standardization, noise reduction, and modal content parsing on the multimodal data assets, and output a preprocessed multimodal dataset. The causal cross-modal coding module, connected to the data access and preprocessing module, is configured to perform shallow feature extraction on each modality data in the preprocessed multimodal dataset, and use a causal mask multi-head cross-modal attention mechanism to adaptively weight and fuse the features of each modality to output cross-modal fused features; wherein, the causal mask ensures that the attention calculation of the current modality can only access the features of the current and previous modalities; The prototype contrastive learning module, connected to the causal cross-modal coding module, is configured to project the cross-modal fusion features onto a unified semantic space through a linear mapping to generate a unified semantic representation vector. The prototype contrastive learning module internally maintains a set of learnable semantic prototype vectors and constrains the spatial distribution of the unified semantic representation vectors through a contrastive learning loss function that includes prototype attraction terms and prototype dispersion regularization terms, so that semantically similar data assets are close to each other in the unified semantic space, and semantically dissimilar data assets are far apart from each other. A differentiable ontology reasoning module, connected to the prototype comparison learning module, is configured to store a business ontology model. This business ontology model includes embedding vectors of domain concept nodes, embedding vectors of concept attributes, and embedding vectors of concept relationships. The differentiable ontology reasoning module employs a tree-structured long short-term memory network, using the parent node embeddings and corresponding concept embeddings of each node in a hierarchical semantic label tree as input, to recursively generate embedding vectors for each label node. The differentiable ontology reasoning module is also configured to calculate the matching score between the unified semantic representation vector and the embedding vectors of each label node, and generate a candidate semantic label set based on the relationship between the matching score and a preset threshold. For fuzzy candidate labels with matching scores between a first threshold and a second threshold, the differentiable ontology reasoning module applies differentiable neural symbolic reasoning rules to enhance knowledge-guided reasoning, iteratively updating the label confidence until convergence, and recording the reasoning path. The hierarchical confidence propagation module, connected to the differentiable ontology reasoning module, is configured to propagate the confidence of the candidate semantic label set bidirectionally from bottom to top and from top to bottom along the hierarchical semantic label tree structure. During bottom-up propagation, the confidence of child nodes is weighted and merged into the parent node. During top-down propagation, the confidence of the parent node is weighted and passed to the child nodes. The final confidence after propagation determines the adopted label, rejected label, or label to be verified. The Shapley value multi-agent verification module, connected to the hierarchical confidence propagation module, is configured to include a semantic consistency verification agent, a hierarchical structure verification agent, a cross-modal association verification agent, and a business rule verification agent. The Shapley value multi-agent verification module predefines a value function to evaluate the labeling performance of any subset of agents, calculates the average marginal contribution of each agent across all subsets as the Shapley value weight for that agent, and performs weighted voting based on the degree of agreement of each agent with the candidate label set. It then performs refined verification on the labels to be verified and outputs the final semantic labeling result. The Shapley value multi-agent verification module is also configured with a self-playing reinforcement learning unit, used to alternately play the roles of labeler and discriminator in a simulated environment to update the policy network of each agent. The knowledge graph construction module is connected to the Shapley value multi-agent verification module and is configured to associate the final semantic annotation result with the business ontology model. It constructs a multimodal asset semantic knowledge graph in the form of resource description framework triples. The knowledge graph uses data asset nodes and semantic tag nodes as nodes and semantic relations as edges, and adds confidence scores and explanatory links pointing to the reasoning path to the edges. The continuous learning module is connected to the causal cross-modal coding module, the prototype contrastive learning module, the differentiable ontology reasoning module, and the Shapley value multi-agent verification module, respectively. It is configured to collect labeled feedback data and use the elastic weight consolidation method to incrementally update the model parameters of each module, thereby optimizing the loss of new tasks while constraining the change range of important parameters.

[0018] The beneficial effects of this invention are as follows: 1. A multi-head cross-modal attention mechanism based on causal masking forces the attention of the current modality to focus only on features of the current and previous modalities, achieving causal and interpretable fusion of cross-modal features. Combined with cross-modal prototype contrastive learning, data from different modalities is mapped to a unified semantic space. Using learnable semantic prototype vectors as anchors, prototype attraction and prototype dispersion regularization constrain the spatial distribution of unified semantic representations, enabling different modal representations of the same business object to be tightly clustered in the semantic space. This fundamentally breaks the traditional paradigm of separate annotation followed by later fusion, enabling true semantic-level associations between multimodal assets and achieving unified semantic retrieval across text, images, audio, and video, effectively solving the technical challenge of new information silos forming between multimodal assets.

[0019] 2. A hierarchical semantic tagging system driven by business ontology was constructed. TreeLSTM was used to deeply integrate ontology concept embedding with the hierarchical structure of the tag tree, ensuring that the embedding of each tag node contains complete path semantic information from the root node to that node. The introduction of a depth decay factor requires stronger semantic evidence to support the lower-level fine-grained tags, aligning with the cognitive principle that "the finer the annotation, the more sufficient the evidence required." The annotation results are automatically associated with the conceptual system of the ontology model, solving the deficiency of existing technologies where flat keyword lists are divorced from business semantics, enabling business personnel to directly understand and use the annotation results.

[0020] 3. By integrating symbolic reasoning rules with neural networks in a differentiable manner, and for fuzzy samples with insufficient confidence, knowledge-guided reasoning enhancement is achieved using predefined conceptual relationships and reasoning rules from the ontology model. Differentiable execution of symbolic logic is realized through iterative updates of label confidence using fuzzy logic AND and maximization operations until convergence. This mechanism enables high-quality, interpretable semantic annotation of dark data hidden in documents, images, and audio / video, improving the effective annotation coverage of unstructured data and allowing a large amount of unstructured data to be incorporated into the asset management system.

[0021] 4. A two-way confidence propagation process is performed along the hierarchical semantic tag tree structure, proceeding from bottom to top and from top to bottom. The confidence scores of child nodes are weighted and passed to the parent node, while the parent node's confidence score simultaneously adjusts the scores of its child nodes, ensuring a consistent and stable overall confidence distribution across the tag tree. This mechanism avoids hierarchical conflicts caused by isolated confidence judgments, guarantees the logical consistency of the annotation results within the hierarchical structure, and improves the completeness of the hierarchical structure in the annotation results.

[0022] 5. Four agents perform semantic consistency, hierarchical structure, cross-modal association, and business rule verification, respectively. Shapley value theory is used to quantify the marginal contribution of each agent to the final annotation quality, enabling differentiated allocation and dynamic optimization of agent weights. A self-play reinforcement learning mechanism allows agents to continuously learn and handle various complex conflict scenarios through adversarial game between the annotator and discriminator. Each annotation result is accompanied by a complete inference path—including the rule trigger sequence, attention weight distribution, and confidence propagation trajectory—making annotation errors traceable.

[0023] 6. The refined semantic annotation results are registered to the ontology model in the form of RDF triples to construct a multimodal asset semantic knowledge graph with data assets and domain concepts as nodes. Confidence scores and interpretable links are attached to the relation edges. This knowledge graph realizes multi-dimensional semantic interconnection between data assets and between data assets and business concepts, providing a structured semantic infrastructure for intelligent semantic retrieval, data governance, and cross-departmental data sharing. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart illustrating the steps of the automatic semantic annotation method for assets based on multimodal data in this application embodiment.

[0025] Figure 2This is a schematic diagram illustrating the structure of an automatic asset semantic annotation system for multimodal data according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0027] Example 1 Preferred embodiments of the present invention, for example Figure 1 As shown, an automatic semantic annotation method for assets in multimodal data includes the following steps: S1. Multi-source heterogeneous data asset access and preprocessing: This involves collecting multimodal data assets from within the enterprise and parsing their modal content in a standardized format to obtain a preprocessed multimodal dataset. Specifically, multimodal data assets include structured data (such as relational database tables), semi-structured data (such as JSON and XML logs), and unstructured data (such as video, audio, images, and text). The collected multimodal data assets undergo format standardization, noise reduction, and modal content parsing.

[0028] For video modalities, audio and video stream separation and temporal alignment are performed, and an adaptive keyframe sampling strategy based on the inter-frame optical flow rate of change is adopted. Specifically, the rate of change of optical flow amplitude between pixels in adjacent frames is calculated, and an optical flow rate of change index is defined. When the optical flow amplitude exceeds a preset threshold (e.g., the rate of change of amplitude exceeds 15%), it is determined to be a region with dense dynamic semantics. At this time, the sampling density is increased (e.g., the sampling interval is reduced to 1 / 3 of the original interval) to preserve complete dynamic semantic information. When the optical flow amplitude is below the threshold, it is determined to be a static or slowly changing region, and the sampling density is reduced (e.g., the sampling interval is increased to 3 times the original interval) to reduce redundancy in subsequent calculations.

[0029] For the audio modality, a learnable front-end filter bank is used instead of a fixed Mel spectrum for noise-resistant feature extraction. Specifically, a set of trainable filter banks is initialized, with its center frequency and bandwidth as learnable parameters. These parameters are adaptively adjusted during training via backpropagation to extract more robust acoustic features from noisy frequency signals. For the image modality, standard size normalization (e.g., scaling to 224×224 pixels) and color normalization are performed. For the text modality, word segmentation, stop word removal, and character encoding cleaning are performed. After these processes, a preprocessed multimodal dataset with a uniform output structure is generated.

[0030] S2. Construct a unified semantic representation space for modalities, use a multi-head cross-modal attention mechanism with causal masking to fuse features from various modalities, and map the fused features to the unified semantic space through cross-modal prototype contrastive learning to generate a unified semantic representation vector.

[0031] The multi-head cross-modal attention mechanism of causal masking forces the model to fuse causal dependencies according to a preset modal order (e.g., text → image → audio → video), ensuring that the attention weights of the i-th modality can only compute features from the current and previous modalities. Cross-modal prototype contrastive learning uses learnable prototype vectors and a loss function that includes prototype attraction and prototype dispersion regularization terms. Specifically, the multi-head cross-modal attention mechanism of causal masking is as follows: For each modality's feature vector, linear transformation matrices for query, key, and value are learned. Attention weights are obtained by calculating the similarity between the query and key and applying a causal mask matrix. In the causal mask matrix, the mask value is zero when the index of the target modality is less than or equal to the index of the source modality; otherwise, the mask value is negative infinity. The attention weights are used to perform a weighted summation of the value vectors, and after layer normalization, the fused feature vectors for each modality are obtained. Specifically, assuming there are N modalities, their feature vector sequence is as follows: Then, for target mode i and source mode j, the attention weights are calculated as follows:

[0032] in and Modal and modality Query, key transformation matrix, Let be the dimension of the key vector. Then apply a causal mask matrix. Where the target mode index Less than the source modality index Time (i.e.) ), ,otherwise The final attention weights are: The value vectors are weighted and summed using this weight, and then processed by layer normalization and a feedforward network to obtain the fused feature vectors for each modality.

[0033] Cross-modal prototype contrastive learning uses a set of learnable prototype vectors. Where K is the preset number of semantic prototypes. The cross-modal prototype contrast loss function includes a first term and a second term; the first term is used to bring the unified semantic representation vector of the data asset closer to its corresponding semantic prototype vector; the second term is used to force the Euclidean distance between different semantic prototype vectors to be no less than a preset minimum distance, and the weighted sum of the two terms constitutes the total loss. Specifically, the first term is the prototype attraction term, which is used to bring the unified semantic representation vector z of the data asset closer to its corresponding semantic prototype vector. The specific form is ,in For temperature coefficient, This represents the true semantic category or pseudo-label corresponding to vector z.

[0034] The second term is the prototype dispersion regularization term, which is used to ensure that the Euclidean distance between different semantic prototype vectors is not less than the preset minimum spacing. The specific form is The total loss is calculated by weighting the two factors together. ,in This is the balancing coefficient. By minimizing this loss, semantically similar data assets are brought closer together in a unified semantic space, while semantically dissimilar data assets are kept apart.

[0035] S3. Construct a hierarchical semantic reasoning network guided by a differentiable ontology. Embed concept nodes and relationships in the business ontology model as vectors. Use a tree-like long short-term memory network (TreeLSTM) to generate the embedding vector of each label node in the hierarchical semantic label tree. Calculate the matching score between the unified semantic representation vector and the embedding vector of each label node. Introduce a depth decay factor into the matching score. For candidate labels with matching scores below the first threshold but above the second threshold, apply a set of neural symbolic reasoning rules to enhance knowledge-guided reasoning. The reasoning rules are implemented using differentiable tensor operations. Iterate until convergence and record the reasoning path.

[0036] Specifically, the TreeLSTM network generates the embedding vectors of labeled nodes as follows: For each node in the hierarchical semantic labeling tree, the embedding vector of the parent node and the corresponding business ontology concept embedding vector are input into the TreeLSTM unit, and the embedding vector of that node is recursively calculated. A top-down recursive approach is used; after initializing the root node embedding, the embedding vectors of child nodes are generated layer by layer using the parent node embedding and the current node's business ontology concept embedding as input. Specifically, for each node v in the hierarchical semantic labeling tree, the embedding vector of its parent node p is... and the corresponding business ontology concept embedding vector The common input is the TreeLSTM unit. Based on the standard LSTM, the TreeLSTM unit pools the hidden states of multiple child nodes and then participates in the gating computation, recursively obtaining the embedding vector of that node. Its core calculation formula can be summarized as follows: .

[0037] The matching score is calculated by: calculating the uniform semantic representation vector z and the tag node embedding vector. cosine similarity Multiply by the depth decay factor ,in The attenuation coefficient is... The depth of the tag node v in the tree. Match score. The introduction of this depth decay factor gives higher matching priority to higher-level (more abstract) tags, which aligns with the cognitive logic of semantic hierarchical reasoning.

[0038] When matching score At the second threshold With the first threshold In between (i.e.) The presence of a value indicating insufficient confidence in the current label suggests the need for knowledge-guided reasoning enhancement. Each rule in the neural symbolic reasoning rule set has the form "if the premise is true, then the conclusion is drawn." Specifically, the confidence of the current candidate label is represented as an activation vector A. The premise of each rule r is calculated by the minimum value of the activation values ​​of multiple premise concepts to determine the degree of rule satisfaction. Then, the activation value of the conclusion concept is updated to the larger of the original activation value and the degree of rule satisfaction: This operation is implemented using differentiable tensor operations. The rule application process can be executed iteratively until the change in activation vector A is less than a preset convergence threshold. After the iteration, new candidate labels and corresponding inference paths (i.e., which rules were triggered and the dependencies between rules) are generated based on the converged activation vector.

[0039] S4. Perform bidirectional propagation of confidence from bottom to top and from top to bottom along the hierarchical semantic label tree structure, and determine the adopted label, rejected label, or label to be verified in the refinement step based on the final confidence after propagation.

[0040] When propagating from bottom to top, the confidence level of the child nodes is... Multiply by the preset uppropagation weight Then, compare the current confidence level with that of the parent node. Compare the values ​​and update the parent node's confidence score by taking the larger value: .

[0041] When propagating from top to bottom, the confidence level of the parent node is... Multiplied by the preset downpropagation weight Then, the current confidence level of the child nodes. Compare the values ​​and update the confidence scores of the child nodes with the larger value: Among them, the upward propagation weight and downward propagation weights The ontology relationship type (such as "is-a", "part-of", etc.) is preset. For example, for the "is-a" relationship, the uppropagation weight is set to 0.9 and the downpropagation weight is set to 0.7 to reflect the strong evidence support of the subclass for the parent class.

[0042] After bidirectional propagation, if the node's final confidence level is higher than the first threshold... If it is below the second threshold, it is determined as an "adoption tag"; If the value is zero, it is determined as a "rejected label"; if it falls between the two, it is determined as a "label to be verified" and proceeds to the next refinement step. If there are multiple child nodes, traverse each child node, multiply its confidence by the upward propagation weight of the corresponding ontology relationship, and then take the maximum value of the current confidence of the parent node to update it; the same applies to top-down propagation, traversing each child node to perform the update.

[0043] S5. A multi-agent chain reasoning refinement based on Shapley weighting is adopted. Four agents are set up to perform semantic consistency verification, hierarchical structure verification, cross-modal association verification and business rule verification respectively. The Shapley value of each agent for the labeling quality is calculated as the weight. The final labeling result is generated by weighted voting, and the policy network of each agent is updated by self-play reinforcement learning.

[0044] The Shapley-weighted multi-agent decision-making method specifically involves treating four agents as participants in a cooperative game and pre-defining a value function. This function is used to evaluate the overall labeled performance (e.g., F1 score) of any subset S of agents on the validation set. For each agent... The average of the marginal contributions of the agent across all possible subsets is calculated as the Shapley value of the agent. : , Where N is the set of all agents. The final labeling result is: the set of labels that maximizes the weighted sum of Shapley values ​​for each agent's approval level among all candidate label sets T, i.e. . For intelligent agents The confidence score for the candidate label set T ranges from [0,1].

[0045] The self-play reinforcement learning process involves four agents taking turns acting as labelers and discriminators in a simulated environment. The discriminator introduces conflicts and errors into existing labels according to preset rules (such as randomly modifying labels or adding noisy labels), constructing adversarial labeling scenarios. The labelers then attempt to correct these errors and generate new labels. The reward function for reinforcement learning consists of the product of precision, recall, and inference path completeness, calculated as either a normalized value of the trigger rule chain length (longer chains indicate higher completeness) or the proportion of triggered rules to the total number of rules that should be triggered. The Proximal Policy Optimization (PPO) algorithm is used to update the policy network parameters of each agent to improve its annotation robustness in complex adversarial scenarios.

[0046] S6. Associate the refined semantic annotation results with the business ontology model to construct a multimodal asset semantic knowledge graph, and store the inference paths in an interpretable form. Constructing the multimodal asset semantic knowledge graph includes: creating each data asset as an asset node, creating the corresponding semantic label as a label node, and using a Resource Description Framework (RDF) triple (asset node, label, label node) to represent the semantic relationship between assets and labels. Attach a confidence score attribute and an explanation link pointing to the inference path node to the relationship edge; the inference path node records the rule trigger sequence (e.g., which inference rules were triggered), the attention weight distribution (the weights of cross-modal attention in S2), and the confidence propagation trajectory (recording the iterative change process of the confidence of each node in S4) to support the complete traceability and interpretability of the annotation results.

[0047] S7. Incremental continuous learning is performed on the models from S2 to S5 based on labeled feedback data. Incremental continuous learning employs an elastic weight consolidation method, incorporating a regularization term when optimizing the loss function for new samples. The regularization term is: for each model parameter... Use this parameter in the Fisher information value of the old task. Multiply by the current parameter Compared with old task parameters The square of the difference, multiplied by the preset regularization coefficient. The sum of all parameters is used as the regularization term: This regularization term constrains the update magnitude of important parameters, effectively preventing the forgetting of existing knowledge when learning new annotation tasks.

[0048] Example 2 An automatic semantic annotation system for assets in multimodal data, used to perform the method of any one of Embodiment 1, with reference to Figure 2It includes a data access and preprocessing module, a causal cross-modal coding module, a prototype comparison learning module, a differentiable ontology reasoning module, a hierarchical confidence propagation module, a Shapley value multi-agent verification module, a knowledge graph construction module, and a continuous learning module.

[0049] The data access and preprocessing module is configured to collect multimodal data assets, perform format standardization, noise reduction, and modal content parsing on the multimodal data assets, and output a preprocessed multimodal dataset.

[0050] The causal cross-modal coding module is connected to the data access and preprocessing module. It is configured to perform shallow feature extraction on each modality data in the preprocessed multimodal dataset, and use a causal mask multi-head cross-modal attention mechanism to adaptively weight and fuse the features of each modality to output cross-modal fused features. Among them, the causal mask ensures that the attention calculation of the current modality can only access the features of the current and previous modalities.

[0051] The prototype contrastive learning module is connected to the causal cross-modal coding module and is configured to project cross-modal fusion features onto a unified semantic space through linear mapping to generate a unified semantic representation vector. The prototype contrastive learning module maintains a set of learnable semantic prototype vectors and constrains the spatial distribution of the unified semantic representation vectors through a contrastive learning loss function that includes prototype attraction terms and prototype dispersion regularization terms, so that semantically similar data assets are close to each other in the unified semantic space, while semantically dissimilar data assets are far apart.

[0052] The differentiable ontology reasoning module is connected to the prototype comparison learning module and configured to store the business ontology model. The business ontology model includes the embedding vectors of domain concept nodes, the embedding vectors of concept attributes, and the embedding vectors of concept relationships. The differentiable ontology reasoning module adopts a tree-structured long short-term memory network, taking the parent node embedding and the corresponding concept embedding of each node in the hierarchical semantic label tree as input, and recursively generates the embedding vector of each label node. The differentiable ontology reasoning module is also configured to calculate the matching score between the unified semantic representation vector and the embedding vector of each label node, and generate a candidate semantic label set according to the relationship between the matching score and a preset threshold. For fuzzy candidate labels with matching scores between the first threshold and the second threshold, the differentiable ontology reasoning module applies differentiable neural symbol reasoning rules to enhance knowledge-guided reasoning, iteratively updates the label confidence until convergence, and records the reasoning path.

[0053] The hierarchical confidence propagation module is connected to the differentiable ontology reasoning module and is configured to propagate the confidence of the candidate semantic label set in both bottom-up and top-down directions along the hierarchical semantic label tree structure. When propagating from bottom to top, the confidence of the child nodes is weighted and merged into the parent node. When propagating from top to bottom, the confidence of the parent node is weighted and passed to the child nodes. The final confidence after propagation determines the adopted label, rejected label, or label to be verified.

[0054] The Shapley value multi-agent verification module is connected to the hierarchical confidence propagation module and is configured to include semantic consistency verification agents, hierarchical structure verification agents, cross-modal association verification agents, and business rule verification agents. The Shapley value multi-agent verification module predefines a value function to evaluate the labeling performance of any subset of agents, calculates the average marginal contribution of each agent on all subsets as the Shapley value weight of that agent, and performs weighted voting based on the degree of approval of each agent for the candidate label set. It then performs refined verification on the labels to be verified and outputs the final semantic labeling results. The Shapley value multi-agent verification module is also equipped with a self-playing reinforcement learning unit, which is used to alternately play the roles of labeler and discriminator in a simulated environment to update the policy network of each agent.

[0055] The knowledge graph construction module is connected to the Shapley value multi-agent verification module and configured to associate the final semantic annotation results with the business ontology model. The multimodal asset semantic knowledge graph is constructed in the form of resource description framework triples. The knowledge graph uses data asset nodes and semantic label nodes as nodes and semantic relations as edges, and adds confidence scores and explanatory links pointing to reasoning paths to the edges.

[0056] The continuous learning module is connected to the causal cross-modal coding module, the prototype contrastive learning module, the differentiable ontology reasoning module, and the Shapley value multi-agent verification module, respectively. It is configured to collect labeled feedback data and use the elastic weight consolidation method to incrementally update the model parameters of each module, thereby optimizing the loss of new tasks while constraining the change range of important parameters.

[0057] The implementation principle of an automatic asset semantic annotation method for multimodal data in this application embodiment is as follows: First, structured multimodal data is obtained through unified access and preprocessing of multi-source heterogeneous data; then, multimodal features are fused and mapped to a unified semantic space using a causal cross-modal attention mechanism and prototype contrastive learning to generate a representation vector with semantic discriminativeness; next, under the guidance of the business ontology, candidate label matching and symbolic reasoning enhancement are performed on a hierarchical semantic label tree through a differentiable reasoning network; then, confidence is propagated bidirectionally along the label tree to fully utilize hierarchical structural information; subsequently, Shapley value from cooperative game theory is introduced to make weighted decisions on the verification results of multiple agents, and each agent is continuously optimized through self-play reinforcement learning; finally, the annotation results are associated with a knowledge graph, and an elastic weight consolidation method is used to achieve incremental continuous learning of the model.

[0058] The system employs a causal masking mechanism to ensure the temporal causality of multimodal feature fusion, constructs a discriminative unified semantic space through prototype contrastive learning, combines symbolic knowledge with neural networks using a differentiable ontology reasoning network to improve the semantic accuracy of annotations, achieves optimal decision combinations for each verification dimension through Shapley value-weighted multi-agent verification, and ensures long-term effectiveness in real-world business scenarios through continuous learning. The entire solution forms a closed-loop chain from data access, semantic understanding, knowledge reasoning, multi-dimensional verification to knowledge accumulation and continuous learning.

[0059] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for automatic semantic annotation of assets for multimodal data, characterized in that, Includes the following steps: S1. Access and preprocessing of multi-source heterogeneous data assets: Collect multimodal data assets within the enterprise and perform modal content parsing in a standardized format to obtain a preprocessed multimodal dataset. S2. Construct a unified semantic representation space for modalities, and use a multi-head cross-modal attention mechanism with causal masking to fuse features from various modalities. Then, through cross-modal prototype contrastive learning, map the fused features to the unified semantic space to generate a unified semantic representation vector. The multi-head cross-modal attention mechanism with causal masking ensures that the attention weight of the i-th modality can only calculate features of the current and previous modalities. The cross-modal prototype contrastive learning uses learnable prototype vectors and a loss function that includes prototype attraction terms and prototype dispersion regularization terms. S3. Construct a hierarchical semantic reasoning network guided by a differentiable ontology. Embed concept nodes and relationships in the business ontology model as vectors. Use a tree-like long short-term memory network (TreeLSTM) to generate the embedding vector of each label node in the hierarchical semantic label tree. Calculate the matching score between the unified semantic representation vector and the embedding vector of each label node. The matching score introduces a depth decay factor. For candidate labels with matching scores below a first threshold but above a second threshold, apply a set of neural symbolic reasoning rules to enhance knowledge-guided reasoning. The reasoning rules are implemented using differentiable tensor operations. Iterate until convergence and record the reasoning path. S4. Perform bidirectional propagation of confidence from bottom to top and from top to bottom along the hierarchical semantic label tree structure, and determine the adopted label, rejected label, or label to be verified in the refinement step based on the final confidence after propagation. S5. A multi-agent chain reasoning refinement based on Shapley weighting is adopted. Four agents are set up to perform semantic consistency verification, hierarchical structure verification, cross-modal association verification and business rule verification respectively. The Shapley value of each agent for the labeling quality is calculated as the weight. The final labeling result is generated by weighted voting, and the policy network of each agent is updated by self-play reinforcement learning. S6. Associate the refined semantic annotation results with the business ontology model, construct a multimodal asset semantic knowledge graph, and store the reasoning path in an interpretable form; S7. Incremental continuous learning is performed on the models from S2 to S5 based on the labeled feedback data.

2. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The multi-head cross-modal attention mechanism of the causal mask in S2 is specifically as follows: For each modality's feature vector, a linear transformation matrix of query, key, and value is learned. Attention weights are obtained by calculating the similarity between the query and the key and applying a causal mask matrix. In the causal mask matrix, the mask value is negative infinity when the index of the target modality is less than the index of the source modality, and the mask value is zero otherwise. The value vectors are weighted and summed using the attention weights, and then normalized by layers to obtain the fused feature vectors for each modality.

3. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The cross-modal prototype contrast loss function in S2 includes a first term and a second term; the first term is used to make the unified semantic representation vector of the data asset closer to the corresponding semantic prototype vector; the second term is used to force the Euclidean distance between different semantic prototype vectors to be no less than the preset minimum spacing, and the weighted sum of the two terms constitutes the total loss.

4. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The method by which the TreeLSTM network in S3 generates the embedding vector of the label node is as follows: for each node in the hierarchical semantic label tree, the embedding vector of the parent node and the embedding vector of the corresponding business ontology concept are input into the TreeLSTM unit, and the embedding vector of the node is recursively calculated; the matching score is calculated by: calculating the cosine similarity between the unified semantic representation vector and the embedding vector of the label node, and then multiplying it by the depth decay factor, which is a negative exponential function with the depth of the label node in the tree as the independent variable.

5. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, Each rule in the neural symbol reasoning rule set in S3 has the form of deriving a conclusion if the premise condition is true. Specifically, the confidence of the current candidate label is represented as an activation vector. The premise condition of each rule is calculated by the minimum value of the activation values ​​of multiple premise concepts to determine the rule satisfaction level. The activation value of the conclusion concept is updated to the larger of the original activation value and the rule satisfaction level. All rules are applied iteratively until the activation vector converges. Based on the converged activation vector, new candidate labels and corresponding reasoning paths are generated. The confidence bidirectional propagation in S4 specifically includes: during bottom-up propagation, multiplying the confidence of the child node by a preset upward propagation weight and comparing it with the current confidence of the parent node, taking the larger value to update the parent node's confidence; During top-down propagation, the confidence of the parent node is multiplied by a preset downward propagation weight and compared with the current confidence of the child node. The larger value is used to update the confidence of the child node. The upward propagation weight and downward propagation weight are preset according to the ontology relationship type.

6. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The S5-based multi-agent decision-making based on Shapley value weighting is as follows: four agents are regarded as participants in a cooperative game; a value function is predefined to evaluate the comprehensive labeling performance of any subset of agents on the validation set; for each agent, the average of its marginal contribution on all possible subsets is calculated as the agent's Shapley value; the final labeling result is the set of labels in all candidate labels that maximizes the weighted sum of the Shapley values ​​of each agent's approval level.

7. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The self-play reinforcement learning steps in S5 are as follows: In a simulated environment, four agents take turns playing the roles of labeler and discriminator. The discriminator introduces conflicts and errors into the existing labeled results according to preset rules, and the labeler attempts to correct and generate new labeled results. The reward function of reinforcement learning consists of the product of precision, recall, and inference path completeness. The policy network parameters of each agent are updated using a proximal policy optimization algorithm.

8. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, The construction of a multimodal asset semantic knowledge graph in S6 includes: treating each data asset as a node, treating the semantic label corresponding to the asset as another type of node, using resource description framework triples to represent the label relationship between asset nodes and label nodes, and attaching confidence scores and explanatory links pointing to inference path nodes to the relationship edges; the inference path nodes record rule triggering sequences, attention weight distribution, and confidence propagation trajectories.

9. The method for automatic semantic annotation of assets for multimodal data according to claim 1, characterized in that, In S7, incremental continuous learning employs an elastic weight consolidation method. When optimizing the loss function for new samples, a regularization term is added. The regularization term is calculated as follows: for each model parameter, the Fisher information value of that parameter in the old task is multiplied by the square of the difference between the current parameter and the old task parameter, and then multiplied by a preset regularization coefficient. The summation of all parameters is then used as the regularization term.

10. An automatic semantic annotation system for assets oriented to multimodal data, used to execute the method of any one of claims 1-9, characterized in that, include: The data access and preprocessing module is configured to collect multimodal data assets, perform format standardization, noise reduction, and modal content parsing on the multimodal data assets, and output a preprocessed multimodal dataset. The causal cross-modal coding module, connected to the data access and preprocessing module, is configured to perform shallow feature extraction on each modality data in the preprocessed multimodal dataset, and use a causal mask multi-head cross-modal attention mechanism to adaptively weight and fuse the features of each modality to output cross-modal fused features; wherein, the causal mask ensures that the attention calculation of the current modality can only access the features of the current and previous modalities; The prototype contrastive learning module, connected to the causal cross-modal coding module, is configured to project the cross-modal fusion features onto a unified semantic space through a linear mapping to generate a unified semantic representation vector. The prototype contrastive learning module internally maintains a set of learnable semantic prototype vectors and constrains the spatial distribution of the unified semantic representation vectors through a contrastive learning loss function that includes prototype attraction terms and prototype dispersion regularization terms, so that semantically similar data assets are close to each other in the unified semantic space, and semantically dissimilar data assets are far apart from each other. A differentiable ontology reasoning module, connected to the prototype comparison learning module, is configured to store a business ontology model. This business ontology model includes embedding vectors of domain concept nodes, embedding vectors of concept attributes, and embedding vectors of concept relationships. The differentiable ontology reasoning module employs a tree-structured long short-term memory network, using the parent node embeddings and corresponding concept embeddings of each node in a hierarchical semantic label tree as input, to recursively generate embedding vectors for each label node. The differentiable ontology reasoning module is also configured to calculate the matching score between the unified semantic representation vector and the embedding vectors of each label node, and generate a candidate semantic label set based on the relationship between the matching score and a preset threshold. For fuzzy candidate labels with matching scores between a first threshold and a second threshold, the differentiable ontology reasoning module applies differentiable neural symbolic reasoning rules to enhance knowledge-guided reasoning, iteratively updating the label confidence until convergence, and recording the reasoning path. The hierarchical confidence propagation module, connected to the differentiable ontology reasoning module, is configured to propagate the confidence of the candidate semantic label set bidirectionally from bottom to top and from top to bottom along the hierarchical semantic label tree structure. During bottom-up propagation, the confidence of child nodes is weighted and merged into the parent node. During top-down propagation, the confidence of the parent node is weighted and passed to the child nodes. The final confidence after propagation determines the adopted label, rejected label, or label to be verified. The Shapley value multi-agent verification module, connected to the hierarchical confidence propagation module, is configured to include a semantic consistency verification agent, a hierarchical structure verification agent, a cross-modal association verification agent, and a business rule verification agent. The Shapley value multi-agent verification module predefines a value function to evaluate the labeling performance of any subset of agents, calculates the average marginal contribution of each agent across all subsets as the Shapley value weight for that agent, and performs weighted voting based on the degree of agreement of each agent with the candidate label set. It then performs refined verification on the labels to be verified and outputs the final semantic labeling result. The Shapley value multi-agent verification module is also configured with a self-playing reinforcement learning unit, used to alternately play the roles of labeler and discriminator in a simulated environment to update the policy network of each agent. The knowledge graph construction module is connected to the Shapley value multi-agent verification module and is configured to associate the final semantic annotation result with the business ontology model. It constructs a multimodal asset semantic knowledge graph in the form of resource description framework triples. The knowledge graph uses data asset nodes and semantic tag nodes as nodes and semantic relations as edges, and adds confidence scores and explanatory links pointing to the reasoning path to the edges. The continuous learning module is connected to the causal cross-modal coding module, the prototype contrastive learning module, the differentiable ontology reasoning module, and the Shapley value multi-agent verification module, respectively. It is configured to collect labeled feedback data and use the elastic weight consolidation method to incrementally update the model parameters of each module, thereby optimizing the loss of new tasks while constraining the change range of important parameters.

Images