A multi-modal metadata alignment fusion method, device, equipment and storage medium

Abstract

The application discloses a multimodal metadata alignment fusion method and device, equipment and storage medium, the method comprises: respectively carrying out metadata extraction to structured data, unstructured text data and image data respectively, generating multimodal metadata with semantic annotation; a shared semantic embedding space is established, the multimodal metadata is mapped to the shared semantic embedding space for coding, and a unified space vector is obtained; the alignment candidate pair is determined from the unified space vector through similarity calculation, and the semantic relationship of the alignment candidate pair is identified; the alignment candidate pair and the corresponding semantic relationship are subjected to conflict detection, and a unified metadata system is obtained by weight weighted fusion and conflict resolution. The application improves and optimizes from multiple aspects such as multimodal data processing, semantic understanding, alignment accuracy and conflict resolution, overcomes the shortcomings of the prior art, and can provide more accurate and comprehensive multimodal metadata alignment fusion results.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, device, and storage medium for multimodal metadata alignment and fusion. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and the Internet of Things (IoT) technologies, multimodal data is widely used in government affairs, finance, healthcare, and the industrial internet. However, significant differences exist between data modalities in metadata structure, semantic representation, and naming conventions, posing a major challenge to the unified management and collaborative analysis of cross-modal data. Therefore, metadata alignment and fusion technologies have become crucial for improving the availability of multi-source heterogeneous data and supporting intelligent analysis and decision-making.

[0003] Current mainstream multimodal metadata processing technologies can be broadly categorized into two types: the first is rule-based mapping, which aligns metadata between structured data through predefined field mapping rules; the second is semantic similarity-based methods, which utilize natural language processing techniques to calculate the similarity between textual metadata. In the field of image metadata, existing solutions often employ pre-trained visual models to generate feature vectors, followed by simple matching using cosine similarity.

[0004] However, existing methods lack a unified processing mechanism for multimodal data and cannot simultaneously handle metadata information between structured, text, and image data. Relying on rules, literal similarity, or lightweight models for metadata alignment lacks an understanding of deep semantic relationships, and is particularly limited in cross-modal semantic alignment between images and text. Summary of the Invention

[0005] To address at least one deficiency or improvement need in the prior art, this invention provides a multimodal metadata alignment and fusion method, apparatus, device, and storage medium. This addresses the problem that the prior art lacks a unified processing mechanism for multimodal data, cannot simultaneously process metadata information between structured, text, and image multimodal data, and relies on rules, literal similarity, or lightweight models for metadata alignment, lacking an understanding of deep semantic relationships, resulting in inaccurate alignment results.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a multimodal metadata alignment and fusion method is provided, comprising: Metadata is extracted from structured data, unstructured text data, and image data respectively, generating multimodal metadata with semantic annotations; A shared semantic embedding space is established, and multimodal metadata is mapped to the shared semantic embedding space for encoding to obtain a unified space vector; Alignment candidate pairs are determined from a unified spatial vector by similarity calculation, and the semantic relationships between the alignment candidate pairs are identified. Conflict detection is performed on alignment candidate pairs and their corresponding semantic relationships, and a unified metadata system is obtained by resolving conflicts based on weighted fusion.

[0007] In one possible implementation, metadata is extracted from structured data, unstructured text data, and image data separately to generate multimodal metadata with semantic annotations, and the implementation also includes: Parse structured data from database metadata tables to extract table structure field-level metadata; Pre-trained language models are used to perform entity recognition and extract text keywords from unstructured text data; Image content labels are extracted from image data using a pre-trained image object detection model.

[0008] In one possible implementation, a shared semantic embedding space is established, and multimodal metadata is mapped to the shared semantic embedding space for encoding to obtain a unified space vector. This also includes: A shared semantic embedding space is constructed through a cross-modal contrastive learning framework, and optimization is performed with the goal of maximizing the cosine similarity of cross-modal data with the same semantics in the space. Modal projection matrices are used to map multimodal metadata to an optimized shared semantic embedding space to obtain modal features; Modal features are subjected to layer normalization and dimension alignment to output a spatial vector with uniform dimensions.

[0009] In one possible implementation, the process of determining alignment candidate pairs from a unified space vector through similarity calculation and identifying the semantic relationships between the alignment candidate pairs further includes: Construct a cross-modal similarity matrix and calculate the similarity metric between vectors in a unified space. Sort the data based on similarity metrics and select a preset number of target metadata pairs as initial candidate pairs; The initial candidate pairs are filtered by preset filtering rules, and the initial candidate pairs that meet the preset requirements are identified as alignment candidate pairs. The semantic relationships of alignment candidate pairs are identified based on preset semantic rules and semantic relationship classification models.

[0010] In one possible implementation, identifying the semantic relationships of alignment candidate pairs based on preset semantic rules and a semantic relationship classification model further includes: Preliminary semantic relationship determination is performed on the alignment candidate pairs based on preset semantic rules; The initial semantic relations are input into the semantic relation classification model to calculate the semantic association strength of the aligned candidate pairs; The semantic relationships of alignment candidate pairs are identified based on the semantic association strength and a preset relationship threshold.

[0011] In one possible implementation, conflict detection is performed on alignment candidate pairs and their corresponding semantic relationships, and conflicts are resolved through weighted fusion to obtain a unified metadata system. This also includes: Conflicts are detected by comparing metadata descriptions of the same semantic relationship in different alignment candidate pairs; Based on a predefined conflict pattern library, the detected conflicts are classified and analyzed to determine the conflict type; Weights are assigned to metadata of different modalities based on historical conflict records and conflict types, and a unified metadata system is obtained by weighted fusion to resolve conflicts.

[0012] In one possible implementation, weights are assigned to metadata of different modalities based on historical conflict records and conflict types. A unified metadata system is obtained by resolving conflicts through weighted fusion. This also includes: The reliability score of each mode is calculated based on historical conflict records, and the initial weight of each mode is dynamically adjusted according to the reliability score. The dynamically adjusted weights are adaptively corrected according to the conflict type to obtain the target weights for each modality. By combining the target weights of each modality and their corresponding metadata, a unified fusion result is generated through weighted calculation.

[0013] According to a second aspect of the present invention, a multimodal metadata alignment and fusion apparatus is also provided, comprising: The data extraction module is configured to extract metadata from structured data, unstructured text data, and image data respectively, and generate multimodal metadata with semantic annotations. The mapping and encoding module is configured to establish a shared semantic embedding space, map multimodal metadata to the shared semantic embedding space for encoding, and obtain a unified space vector; The relationship recognition module is configured to determine alignment candidate pairs from a uniform space vector through similarity calculation and to identify the semantic relationship between the alignment candidate pairs; The conflict resolution module is configured to perform conflict detection on alignment candidate pairs and their corresponding semantic relationships, and resolve conflicts based on weighted fusion to obtain a unified metadata system.

[0014] According to a third aspect of the present invention, a multimodal metadata alignment and fusion apparatus is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the above-described multimodal metadata alignment and fusion methods.

[0015] According to a fourth aspect of the present invention, a storage medium is also provided, which stores a computer program executable by a multimodal metadata alignment and fusion device, wherein when the computer program is run on the multimodal metadata alignment and fusion device, the multimodal metadata alignment and fusion device performs the steps of any of the above-described multimodal metadata alignment and fusion methods.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: This invention provides a multimodal metadata alignment and fusion method. By extracting metadata from structured data, unstructured text data, and image data separately, and generating multimodal metadata with semantic annotations, it overcomes the limitations of unified multimodal data processing mechanisms and can simultaneously process metadata information from multiple different types of data. A shared semantic embedding space is established, and multimodal metadata is mapped and encoded within it to obtain a unified spatial vector. Compared to methods relying on rules, literal similarity, or lightweight models for metadata alignment, this method can delve deeper into the underlying semantic information of the data, better understand the semantic relationships between different modalities, and thus uncover these deep semantic relationships. Based on the unified spatial vector, similarity calculations determine alignment candidate pairs, and their semantic relationships are identified. Conflict detection and resolution are then performed on the alignment candidate pairs and semantic relationships, ultimately resulting in a unified metadata system. This helps to more accurately find the correspondences between different modalities, reduce alignment errors, improve the accuracy of metadata alignment, and overcome the inaccurate alignment results of existing technologies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the multimodal metadata alignment and fusion method provided by the present invention; Figure 2 Provided by the present invention Figure 1 A schematic flowchart of an embodiment of step S101; Figure 3 Provided by the present invention Figure 1 A schematic flowchart of an embodiment of step S102; Figure 4 Provided by the present invention Figure 1 A schematic flowchart of an embodiment of step S103; Figure 5 Provided by the present invention Figure 4 A schematic flowchart of an embodiment of step S404; Figure 6 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S104; Figure 7 Provided by the present invention Figure 6 A schematic flowchart of an embodiment of step S603; Figure 8 This is a schematic diagram of a structure of an embodiment of the multimodal metadata alignment and fusion device provided by the present invention; Figure 9 This is a schematic diagram of the structure of the multimodal metadata alignment and fusion device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] This invention provides a method, apparatus, device, and storage medium for aligning and fusing multimodal metadata, which will be described below.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multimodal metadata alignment and fusion method provided by the present invention. In a specific embodiment of the present invention, a multimodal metadata alignment and fusion method is disclosed, comprising: S101. Extract metadata from structured data, unstructured text data and image data respectively, and generate multimodal metadata with semantic annotation; S102. Establish a shared semantic embedding space, map multimodal metadata to the shared semantic embedding space for encoding, and obtain a unified space vector; S103. Determine alignment candidate pairs from the unified space vector through similarity calculation, and identify the semantic relationships of the alignment candidate pairs; S104. Conflict detection is performed on the alignment candidate pairs and their corresponding semantic relationships. Conflicts are resolved by weighted fusion to obtain a unified metadata system.

[0023] In the above embodiments, structured data is typically stored in relational databases or specific data tables, with clearly defined fields and formats. The metadata extraction process can directly extract key information, such as data type, value range, and relationships, based on the database's table structure and field definitions. Simultaneously, by combining this metadata with an ontology knowledge base specific to the business domain, corresponding semantic tags are added to give it clear semantic meaning.

[0024] Unstructured text data has a free-form content and lacks a fixed structure, requiring natural language processing techniques for metadata extraction. Preliminary processing, such as word segmentation and part-of-speech tagging, is performed to identify entities and keywords. Then, dependency parsing is used to mine semantic relationships and construct a semantic structure. Finally, metadata such as topic and sentiment is extracted and labeled.

[0025] Image data metadata extraction requires a combination of computer vision and deep learning. On the one hand, feature extraction algorithms are used to obtain low-level feature vectors such as color and texture; on the other hand, object detection and image classification models are used to identify objects, scenes, and other information and extract them as metadata, while adding semantic annotations to generate multimodal metadata with semantic annotations.

[0026] Different modal metadata features and semantics differ, necessitating the establishment of a shared semantic embedding space to achieve a unified representation. A modal projection matrix is ​​used to uniformly map metadata from different modalities to the shared semantic embedding space, yielding modal features. Modality-specific feature extractors are employed to process different data types, including a text encoder based on a hierarchical attention mechanism, a visual encoder combining region and global features, and a joint field encoder for structured data. This ensures that different modal representations of the same semantic content maintain close proximity in the vector space while separating irrelevant content, ultimately resulting in a unified spatial vector.

[0027] After obtaining the unified spatial vectors of multimodal metadata, it is necessary to calculate the similarity between these vectors to identify candidate pairs that may have alignment relationships and further identify their semantic relationships. A cross-modal similarity matrix is ​​constructed by calculating the similarity metric between the unified spatial vectors. Based on a preset similarity threshold, metadata pairs with similarity scores higher than the threshold are selected as alignment candidates.

[0028] To further clarify the semantic relationships between alignment candidate pairs, the candidate pairs are analyzed and judged according to preset semantic rules and semantic relationship classification models. For example, in a multimodal dataset containing product information, if alignment candidate pairs are formed between the "product category" metadata in the structured data, the statement "this product belongs to electronic products" in the unstructured text data, and the appearance features of electronic devices shown in the image data, then the semantic relationship between them can be determined to be "belonging to the same category" through semantic relationship recognition.

[0029] Real-world multimodal data comes from diverse sources, and alignment candidate pairs and their semantic relationships may conflict. For example, different data sources may label the same product category differently. Therefore, conflict detection is necessary to analyze the consistency and inconsistencies of alignment candidate pairs and semantic relationships. Conflict resolution is achieved through weighted fusion, typically by assigning weights to different modalities based on factors such as data source reliability and data update time. By combining candidate pairs and semantic relationships through weighted calculation, a unified metadata system is obtained, eliminating conflicts and improving metadata quality and consistency.

[0030] Compared with existing technologies, this embodiment provides a multimodal metadata alignment and fusion method. By extracting metadata from structured data, unstructured text data, and image data separately, and generating multimodal metadata with semantic annotations, it overcomes the limitations of unified multimodal data processing mechanisms and can simultaneously process metadata information of multiple different types of data. A shared semantic embedding space is established, and multimodal metadata is mapped and encoded within it to obtain a unified spatial vector. Compared with methods relying on rules, literal similarity, or lightweight models for metadata alignment, this method can deeply mine the underlying semantic information of the data, better understand the semantic relationships between different modalities, and thus uncover these deep semantic relationships. Based on the unified spatial vector, similarity calculations are used to determine alignment candidate pairs and identify their semantic relationships. Conflict detection and resolution are then performed on the alignment candidate pairs and semantic relationships to ultimately obtain a unified metadata system. This helps to more accurately find the correspondence between different modalities, reduce alignment errors, improve the accuracy of metadata alignment, and overcome the inaccurate alignment results of existing technologies.

[0031] Please see Figure 2 , Figure 2 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S101. In some embodiments of the present invention, metadata extraction is performed on structured data, unstructured text data, and image data respectively to generate multimodal metadata with semantic annotations, and the method further includes: S201. Parse the database metadata table of structured data to extract table structure field-level metadata; S202. Use a pre-trained language model to perform entity recognition and extract text keywords from unstructured text data; S203. Extract image content labels from image data using a pre-trained image target detection model.

[0032] In the above embodiments, for metadata extraction from structured data sources, complete table structure definition information is obtained by accessing the database system directory, including basic metadata attributes such as table name, field name, data type, and constraints. Based on the distribution pattern, uniqueness characteristics, and format regularity of field values, the business semantic type of the fields is inferred. For complex business scenarios, the accuracy of field annotation can be enhanced by custom semantic rules, and the final output structured metadata contains a complete semantic description that is machine-readable.

[0033] It should be noted that database metadata tables refer to system directory tables built into a database management system, used to store the definition and structure information of database objects. In relational databases, these system tables typically include table information tables describing the data table structure, column information tables defining record fields, index tables storing index information, and foreign key constraint tables maintaining relationships between tables. Different types of database management systems (such as Oracle, MySQL, SQL Server, etc.) have their own metadata table implementations, but they all follow similar information architecture principles, and this invention does not impose further limitations on them.

[0034] Metadata extraction from unstructured text data employs a pre-trained language understanding model to perform multi-level semantic analysis on the input text. First, fine-grained named entity recognition is performed to locate entity fragments with specific semantic types in the text, such as names of people, organizations, and time expressions. Simultaneously, keyword extraction is performed to identify core concepts, terms, and keywords in the text. Through contextual semantic analysis, a network of relationships between entities and keywords is established. Furthermore, domain knowledge graphs can be used to semantically normalize the extracted entities and terms, mapping diverse expressions in the text to a standard concept system, ensuring that the generated metadata accurately reflects the deep semantic content of the text.

[0035] As a preferred embodiment, the pre-trained language model adopts the Transformer architecture and captures long-distance dependencies in text through a self-attention mechanism. This includes models based on bidirectional Transformers, autoregressive language models, and state-of-the-art large-scale pre-trained models. Through pre-training on massive amounts of text data, it can understand the contextual semantics of words and identify entities and relationships in the text. In practical applications, the pre-trained model can be fine-tuned according to specific needs to better adapt it to text processing tasks in specific domains.

[0036] Metadata extraction for image-type data employs a pre-trained object detection model to perform comprehensive visual content analysis on the input image, identifying specific objects and their spatial locations, extracting global semantic features, and recognizing the overall scene category and atmosphere. Through multi-level feature fusion technology, local object information is organically combined with global scene information to generate a comprehensive visual content description. The detected visual content is transformed into standardized semantic labels, accurately describing the visual content and implicit semantics of the image.

[0037] As we can understand, a pre-trained image object detection model refers to a computer vision model pre-trained on a large-scale image dataset. It is capable of recognizing objects in an image and their locations. It can use deep convolutional neural networks or visual Transformers as its basic architecture, and achieve object localization and classification through region proposal networks or end-to-end detection frameworks. Typical image object detection models include two-stage detection models based on region proposal and single-stage detection models based on regression, capable of recognizing thousands of common object categories. In practical applications, transfer learning can be performed on these pre-trained models as needed to adapt them to the image analysis requirements of specific domains.

[0038] Please see Figure 3 , Figure 3 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S102. In some embodiments of the present invention, a shared semantic embedding space is established, and multimodal metadata is mapped to the shared semantic embedding space for encoding to obtain a unified space vector. The method further includes: S301. Construct a shared semantic embedding space through a cross-modal contrastive learning framework, and optimize it with the goal of maximizing the cosine similarity of cross-modal data with the same semantics in the space. S302. The modal projection matrix is ​​used to map the multimodal metadata to the optimized shared semantic embedding space to obtain modal features; S303. Perform layer normalization and dimension alignment on the modal features to output a spatial vector with uniform dimensions.

[0039] In the above embodiments, the construction of the shared semantic embedding space is implemented using a cross-modal contrastive learning framework. This framework optimizes the vector similarity of the same semantic content across different modalities using a specific loss function. During training, positive and negative sample pairs are constructed. Positive samples consist of data from different modalities describing the same semantic concept, while negative samples are randomly combined from unrelated cross-modal data. The training process can utilize a deep neural network as its underlying architecture, iteratively optimizing and learning to map different modalities of data to a unified vector space.

[0040] A specialized modal projection matrix architecture is designed for each data type: for structured data, a joint encoding method is used to process field names and sampled values ​​simultaneously; for text data, a hierarchical feature extractor is used to capture word-level and sentence-level semantic information; and for image data, a region feature aggregation network integrates local and global visual features. Each modal projection matrix needs to be optimized end-to-end within a contrastive learning framework. The projection process preserves the semantic features of the original data while eliminating modality-specific expression differences to obtain modal features.

[0041] Layer normalization standardizes the projected modal features, eliminating differences in vector scale and distribution among different modalities and ensuring that all modal features fall within the same numerical range. Dimension alignment uses a learnable linear transformation to unify the feature vectors of different modalities to the same dimension for subsequent similarity calculations. A grouping normalization strategy is employed, grouping features according to semantic meaning to preserve important semantic association information, ultimately outputting a spatial vector with a unified dimension.

[0042] Please see Figure 4 , Figure 4 Provided by the present invention Figure 1 A flowchart illustrating one embodiment of step S103. In some embodiments of the present invention, the process of determining alignment candidate pairs from a unified spatial vector through similarity calculation and identifying the semantic relationships of the alignment candidate pairs further includes: S401. Construct a cross-modal similarity matrix and calculate the similarity metric between unified space vectors; S402. Sort the target metadata based on similarity metrics and select a preset number of target metadata pairs as initial candidate pairs; S403. Filter the initial candidate pairs according to the preset filtering rules, and determine the initial candidate pairs that meet the preset requirements as alignment candidate pairs; S404. Identify the semantic relationships of alignment candidate pairs based on preset semantic rules and semantic relationship classification models.

[0043] In the above embodiments, the construction of the cross-modal similarity matrix adopts a vector comparison method based on a unified semantic space. All metadata vectors to be aligned are standardized to ensure that vectors from different modalities are on a comparable scale. The similarity metric employs an improved cosine similarity algorithm, which not only considers the angle between vectors but also introduces a semantic relevance correction factor to enhance the recognition of domain-specific terms. To improve computational efficiency, the calculation process uses a block matrix operation strategy. First, sub-matrices are calculated by modality, and then combined into a complete cross-modal similarity matrix to calculate the similarity metric between vectors in the unified space.

[0044] The similarity metrics are sorted in descending order. A higher similarity metric indicates greater similarity between the two pairs. Therefore, by sorting in descending order, a predetermined number of initial candidate pairs can be determined. Specifically, the predetermined number is automatically adjusted based on the data distribution density. The candidate range is appropriately broadened in sparse areas, while the screening criteria are raised in dense areas. The predetermined number can be adjusted according to actual needs, and this invention does not impose further limitations on this.

[0045] The preset filtering rules employ a multi-level filtering mechanism. The first level of filtering is based on a hard threshold (which can be set according to actual needs; this invention does not impose further restrictions on this), eliminating candidate pairs that clearly do not meet the basic similarity requirements. The second level of filtering performs semantic consistency checks, verifying the conceptual compatibility of candidate pairs through a pre-built thesaurus and domain ontology. The third level of filtering uses contextual association analysis to check whether the semantic roles of candidate pairs match in their respective data environments, avoiding contradictory alignments caused by local filtering. The final output set of aligned candidate pairs maximizes coverage of potential semantic associations while ensuring high accuracy.

[0046] Please see Figure 5 , Figure 5 Provided by the present invention Figure 4 A flowchart illustrating an embodiment of step S404. In some embodiments of the present invention, identifying the semantic relationship of alignment candidate pairs based on preset semantic rules and a semantic relationship classification model further includes: S501. Perform preliminary semantic relationship determination on alignment candidate pairs based on preset semantic rules; S502. Input the preliminary semantic relations into the semantic relation classification model and calculate the semantic association strength of the aligned candidate pairs; S503. Identify the semantic relationships of alignment candidate pairs based on semantic association strength and preset relationship threshold.

[0047] In the above embodiments, the preset semantic rules are based on a hierarchical rule engine architecture, including a basic rule layer, a synonym rule layer, a structural rule layer, and a context rule layer. Specifically, the basic rule layer uses string matching technology to handle cases where field names are completely identical, ensuring rapid determination of explicit matches. The synonym rule layer loads semantic dictionaries customized for each specific domain, identifying standardized synonym expressions, including term abbreviations, multilingual comparisons, and variations of professional terms. The structural rule layer analyzes naming patterns and data type characteristics to infer potential hierarchical relationships and attribute associations, such as identifying naming patterns containing parent-child structures or value ranges with inclusion relationships. The context rule layer examines the usage scenarios of alignment candidate pairs in their respective data environments, verifies the rationality of relationships through contextual semantic analysis, and thus achieves preliminary semantic relationship determination.

[0048] The semantic relation classification model employs a multimodal fusion deep neural network architecture. Its input layer receives preliminary semantic relations, and a cross-attention mechanism is used to establish fine-grained semantic association mappings. A multi-head attention structure captures relation features at different levels of abstraction, constructing a joint representation of candidate pairs. The output layer uses a multi-task learning framework to simultaneously predict relation type and association strength, ensuring consistency between type determination and confidence assessment.

[0049] The preset relationship thresholds were determined through statistical analysis of a large number of labeled samples. Based on positive and negative samples collected from historical alignment tasks, the similarity score distributions corresponding to various semantic relationships were calculated. Based on these score distributions, a basic threshold was set for each type of relationship. Understandably, these basic thresholds will be continuously optimized and adjusted as the number of samples accumulates.

[0050] The semantic relationship recognition process employs a multi-stage decision-making framework. First, the semantic association strength vector is analyzed from multiple dimensions to extract key features, including basic similarity scores, cross-modal consistency indices, and contextual matching degrees. A comprehensive confidence assessment is then generated using a weighted fusion function. This comprehensive confidence score is compared to a preset threshold: for high-confidence candidate pairs significantly exceeding the threshold, the defined relationship type is directly output; for candidate pairs in the pending review range, auxiliary verification is performed using knowledge graph reasoning and contextual consistency checks; and candidate pairs below the rejection threshold are ignored.

[0051] Please see Figure 6 , Figure 6 Provided by the present invention Figure 1 A flowchart illustrating an embodiment of step S104. In some embodiments of the present invention, conflict detection is performed on alignment candidate pairs and their corresponding semantic relationships, and a unified metadata system is obtained by resolving conflicts based on weighted fusion. The method further includes: S601. Detect whether there is a conflict by comparing the metadata descriptions of the same semantic relationship in different alignment candidate pairs; S602. Based on a predefined conflict pattern library, classify and analyze the detected conflicts to determine the conflict type; S603. Assign weights to metadata of different modalities based on historical conflict records and conflict types, and resolve conflicts through weighted fusion to obtain a unified metadata system.

[0052] In the above embodiments, during the construction of a unified metadata system, multiple alignment candidate pairs may exist for the same semantic relationship. To detect whether a conflict exists, it is first necessary to locate all alignment candidate pairs involving the same semantic relationship. Alignment candidate pairs with the same semantic relationship identifier can be grouped together based on the identifier of the semantic relationship.

[0053] Metadata descriptions typically contain information across multiple dimensions, such as data type, value range, unit, and semantic meaning. This information is compared dimension by dimension. For example, in terms of data type, if one alignment candidate pair's metadata is described as an integer, while another alignment candidate pair's metadata corresponding to the same semantic relationship is described as a string, a conflict may exist. In terms of value range, if one description specifies values ​​within a specific interval, while another description's value range does not overlap, this will also be considered a potential conflict. Furthermore, even if the data type and value range appear identical, differences in semantic meaning can still lead to conflict. For instance, one description interprets "temperature" as Celsius, while another interprets it as Fahrenheit; although both are numerical types, their semantics differ, constituting a conflict.

[0054] By comprehensively comparing the results across various dimensions, a conflict is determined. If an inconsistency is found in any key dimension, a conflict is identified among these alignment candidate pairs, and they are marked for further processing.

[0055] Once a conflict is detected, its detailed information is matched against a predefined conflict pattern library. Key features of the conflict are extracted, such as the semantic relationships involved, the differences in metadata descriptions (data type, value range, semantic connotation, etc.), and the contextual information of the conflict. These features are then compared with the features of each pattern in the conflict pattern library. If the features of a conflict highly match those of a particular pattern, the conflict is considered to belong to the conflict type defined by that pattern.

[0056] Specifically, conflict detection of alignment candidate pairs and their corresponding semantic relations includes, but is not limited to: Type conflict detection: Identifies conflicts between numerical and textual descriptions of the same concept in different modalities; Unit conflict detection: Identifies inconsistencies in units of measurement across modal data; Relationship conflict detection: Verifies the consistency of relation declarations for the same pair of metadata in different modalities; Semantic conflict detection: Identifying logical contradictions in the descriptions of the same concept by different modalities.

[0057] Please see Figure 7 , Figure 7 Provided by the present invention Figure 6 A flowchart illustrating an embodiment of step S603 is provided. In some embodiments of the present invention, weights are assigned to metadata of different modalities based on historical conflict records and conflict types. A unified metadata system is obtained by resolving conflicts through weighted fusion. The method further includes: S701. Calculate the reliability score of each mode based on historical conflict records, and dynamically adjust the initial weight of each mode according to the reliability score; S702. Adaptively correct the dynamically adjusted weights according to the conflict type to obtain the target weights for each mode. S703. Combining the target weights of each modality and their corresponding metadata, a unified fusion result is generated through weighted calculation.

[0058] In the above embodiments, the historical conflict record contains details of different modal metadata in past conflicts, such as the time when the conflict occurred, the metadata modality involved, the type of conflict, and the method of conflict resolution.

[0059] For each modality, the number of conflicts that occur within a certain time frame is counted. Modalities with fewer conflicts generally indicate higher data quality and greater reliability. Simultaneously, the severity of each conflict is analyzed, which can be measured by factors such as the impact of the conflict on subsequent business processes and the resources spent resolving the conflict.

[0060] To comprehensively consider both the number and severity of conflicts, a weighted approach is used to calculate the reliability score. A weighting coefficient is assigned to both the number and severity of conflicts, which can be adjusted based on actual business needs and data characteristics. Then, based on the number and severity of conflicts for each modality, a weighted calculation is performed according to the weighting coefficients to obtain the reliability score for each modality.

[0061] The initial weights were pre-set based on factors such as the importance and volume of each modality in the overall data, without considering historical conflicts. The dynamic adjustment logic is based on the proportional relationship between the reliability score and the initial weight. If a modality has a high reliability score, it indicates that its data quality is good, and it should be given a larger weight during the fusion process to fully leverage its data advantages. Conversely, if a modality has a low reliability score, it indicates that its data may have many problems, and its weight should be appropriately reduced to minimize its adverse impact on the fusion results.

[0062] In specific adjustments, an adjustment coefficient can be set, which is proportional to the reliability score. The initial weight of each mode is multiplied by the corresponding adjustment coefficient to obtain the adjusted weight. The adjustment coefficient can be calculated linearly or non-linearly, dynamically and reasonably adjusting the initial weights based on the reliability of each mode.

[0063] Type conflict detection primarily focuses on the consistency of data types among different modal metadata. When a type conflict is detected, it indicates a difference in data type definitions between different modalities, which can affect data fusion. For modalities experiencing type conflicts, their weights are adjusted based on the severity of the conflict. If the type conflict is due to data entry errors or non-standard data sources, and the modality has low importance in the overall data, its weight can be appropriately reduced. For example, in a dataset containing multiple modalities such as text, images, and numerical values, if the text modality and the numerical modality have a type conflict in a semantic relationship, and the text modality has poor data quality and low reliability, while the semantic relationship is highly dependent on the numerical modality, then the weight of the text modality will be reduced. Conversely, if the type conflict is due to changes in business requirements or updates to data definitions, and the modality plays a crucial role in the business, then its weight will be maintained or appropriately increased.

[0064] Unit conflict detection is used to determine whether the units of metadata from different modalities are consistent. Inconsistent units can lead to errors in numerical comparisons and fusion, affecting the accuracy of the fusion results. When a unit conflict is detected, the conversion relationships and difficulty of the units between different modalities are analyzed. If the units of a certain modality can be converted to a unified unit through simple calculations, and the data quality of that modality is high, then its weight will remain unchanged or be appropriately increased. For example, in the fusion of length data, one modality uses meters as the unit, and another uses centimeters. Since the conversion from centimeters to meters is very simple, and the data of the modality using centimeters is accurate, its weight can be maintained or increased. If the unit conversion relationships are complex, or the units of a certain modality cannot be unified with other modalities, and the importance of that modality in the overall data is low, then the weight of that modality will be reduced.

[0065] Relationship conflict detection primarily examines the consistency of semantic relationships between metadata from different modalities. Inconsistencies in semantic relationships can lead to logical errors after data fusion, affecting data usability. For modalities with relationship conflicts, weights are adjusted based on the degree of impact of the conflict on business logic. If a relationship conflict causes serious errors in business logic, and the conflicting modality is not a critical aspect of the business, its weight will be significantly reduced. For example, in a supply chain management system, conflicting data from different modalities in describing cargo transportation relationships can lead to inaccurate cargo tracking. If the data source for this conflicting modality is of low reliability and is not a core monitoring component, its weight will be reduced. If the relationship conflict has a minor impact on business logic, or if the conflicting modality contains important business information, its weight will be maintained or appropriately adjusted.

[0066] Semantic conflict detection is used to identify differences in semantic expression between metadata from different modalities, which can lead to ambiguity in the meaning of fused data, affecting data understanding and use. When a semantic conflict is detected, the similarity and degree of difference in semantics between different modalities are analyzed. If the semantics of a certain modality are highly similar to those of other modalities, and the data quality of that modality is good, then its weight will be maintained or appropriately increased. For example, in a text classification task, different modalities may have semantic differences in their descriptions of the same topic, but if the semantics of one modality are more accurate and richer, then its weight will be increased. If the semantic conflict is significant, and the semantic expression of a certain modality is unclear or inaccurate, and the importance of that modality in the overall data is low, then the weight of that modality will be decreased.

[0067] By comprehensively considering different types of conflicts such as type conflict, unit conflict, relational conflict, and semantic conflict, the dynamically adjusted weights are adaptively corrected to obtain the target weights for each modality, making the weights more consistent with the actual data situation and business needs.

[0068] After obtaining the target weights for each modality, a weighted calculation can be performed using the corresponding metadata for each modality to generate a unified fusion result. First, the metadata for each modality is standardized. Since the metadata of different modalities may have different data types, value ranges, and distribution characteristics, standardization makes them comparable. Then, based on the target weights for each modality, a weighted calculation is performed on the standardized metadata. Each modality's metadata is multiplied by its corresponding target weight, and then the weighted results for all modalities are summed.

[0069] By employing a weighted calculation method, the differences in data quality, reliability, and business importance among various modalities are considered, resulting in a more reasonable synthesis of information from different modalities. The final unified fusion result can then be used for subsequent data analysis, decision support, and other business scenarios.

[0070] To better implement the multimodal metadata alignment and fusion method in the embodiments of the present invention, based on the multimodal metadata alignment and fusion method, please refer to the corresponding... Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the multimodal metadata alignment and fusion device provided by the present invention. The embodiment of the present invention provides a multimodal metadata alignment and fusion device 800, comprising: The data extraction module 810 is configured to extract metadata from structured data, unstructured text data and image data respectively, and generate multimodal metadata with semantic annotation. The mapping and encoding module 820 is configured to establish a shared semantic embedding space, map multimodal metadata to the shared semantic embedding space for encoding, and obtain a unified space vector; The relation recognition module 830 is configured to determine alignment candidate pairs from a uniform space vector through similarity calculation and to identify the semantic relations of the alignment candidate pairs; The conflict resolution module 840 is configured to perform conflict detection on alignment candidate pairs and their corresponding semantic relationships, and resolve conflicts based on weighted fusion to obtain a unified metadata system.

[0071] It should be noted that the device 800 provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding content in the above method embodiments, and will not be repeated here.

[0072] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a multimodal metadata alignment and fusion device provided in an embodiment of the present invention. Based on the above-described multimodal metadata alignment and fusion method, the present invention also provides a multimodal metadata alignment and fusion device, which can be a computing device such as a mobile terminal, desktop computer, laptop, handheld computer, or server. The multimodal metadata alignment and fusion device 900 includes a processor 910, a memory 920, and a display 930. Figure 9 Only a portion of the components of the multimodal metadata alignment and fusion device are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead.

[0073] In some embodiments, the memory 920 may be an internal storage unit of the multimodal metadata alignment and fusion device 900, such as a hard disk or memory of the multimodal metadata alignment and fusion device 900. In other embodiments, the memory 920 may be an external storage device of the multimodal metadata alignment and fusion device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the multimodal metadata alignment and fusion device 900. Furthermore, the memory 920 may include both internal storage units and external storage devices of the multimodal metadata alignment and fusion device 900. The memory 920 is used to store application software and various types of data installed on the multimodal metadata alignment and fusion device 900, such as the program code installed on the multimodal metadata alignment and fusion device 900. The memory 920 may also be used to temporarily store data that has been output or will be output. In one embodiment, a multimodal metadata alignment and fusion program 940 is stored in the memory 920. The multimodal metadata alignment and fusion program 940 can be executed by the processor 910 to implement the multimodal metadata alignment and fusion method of the various embodiments of this application.

[0074] In some embodiments, processor 910 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 920 or process data, such as executing a multimodal metadata alignment and fusion method.

[0075] In some embodiments, display 930 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 930 is used to display information from the multimodal metadata alignment and fusion device 900 and to display a user interface for visualization. Components 910-930 of the multimodal metadata alignment and fusion device 900 communicate with each other via a system bus.

[0076] In one embodiment, the processor 910 implements the steps of the multimodal metadata alignment and fusion method described above when executing the multimodal metadata alignment and fusion program 940 in the memory 920.

[0077] This embodiment also provides a computer-readable storage medium storing a multimodal metadata alignment and fusion program, which, when executed by a processor, performs the following steps: Metadata is extracted from structured data, unstructured text data, and image data respectively, generating multimodal metadata with semantic annotations; A shared semantic embedding space is established, and multimodal metadata is mapped to the shared semantic embedding space for encoding to obtain a unified space vector; Alignment candidate pairs are determined from a unified spatial vector by similarity calculation, and the semantic relationships between the alignment candidate pairs are identified. Conflict detection is performed on alignment candidate pairs and their corresponding semantic relationships, and a unified metadata system is obtained by resolving conflicts based on weighted fusion.

[0078] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0079] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0080] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0085] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0086] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for aligning and fusing multimodal metadata, characterized in that, include: Metadata is extracted from structured data, unstructured text data, and image data respectively, generating multimodal metadata with semantic annotations; A shared semantic embedding space is established, and the multimodal metadata is mapped to the shared semantic embedding space for encoding to obtain a unified space vector; Alignment candidate pairs are determined from the unified spatial vector by similarity calculation, and the semantic relationships of the alignment candidate pairs are identified. Conflict detection is performed on the alignment candidate pairs and their corresponding semantic relationships. A unified metadata system is obtained by resolving conflicts through weighted fusion. This includes: detecting conflicts by comparing metadata descriptions of the same semantic relationship in different alignment candidate pairs; classifying and analyzing detected conflicts based on a predefined conflict pattern library to determine conflict types; calculating reliability scores for each modality based on historical conflict records, and dynamically adjusting the initial weights of each modality according to the reliability scores; adaptively correcting the dynamically adjusted weights according to the conflict types to obtain target weights for each modality; and generating a unified fusion result by combining the target weights of each modality and their corresponding metadata through weighted calculation. The step of determining alignment candidate pairs from the unified spatial vector through similarity calculation and identifying the semantic relationships of the alignment candidate pairs further includes: Construct a cross-modal similarity matrix and calculate the similarity metric between the unified space vectors; Based on the similarity metric, the target metadata corresponding to a preset number is selected as the initial candidate pairs. The initial candidate pairs are filtered using preset filtering rules to determine the initial candidate pairs that meet the preset requirements as alignment candidate pairs; The semantic relationships of the alignment candidate pairs are identified based on preset semantic rules and a semantic relationship classification model; The preset filtering rules are a multi-level filtering mechanism; The first-level filtering is based on a hard threshold, which removes initial candidate pairs that do not meet the basic similarity requirements. The second-level filtering implements semantic consistency checks, which verify the conceptual compatibility of the initial candidate pairs through a pre-built thesaurus and domain ontology. The third-level filtering uses contextual analysis to check whether the semantic roles of candidate pairs match in their respective data environments. The final output maximizes the coverage of aligned candidate pairs with potential semantic associations while meeting the preset precision.

2. The multimodal metadata alignment and fusion method as described in claim 1, characterized in that, The step of extracting metadata from structured data, unstructured text data, and image data respectively to generate multimodal metadata with semantic annotations also includes: Parse structured data from database metadata tables to extract table structure field-level metadata; Pre-trained language models are used to perform entity recognition and extract text keywords from unstructured text data; Image content labels are extracted from image data using a pre-trained image object detection model.

3. The multimodal metadata alignment and fusion method as described in claim 1, characterized in that, The step of establishing a shared semantic embedding space, mapping the multimodal metadata to the shared semantic embedding space for encoding, and obtaining a unified space vector, further includes: A shared semantic embedding space is constructed through a cross-modal contrastive learning framework, and optimization is performed with the goal of maximizing the cosine similarity of cross-modal data with the same semantics in the space. The multimodal metadata is mapped to an optimized shared semantic embedding space using a modal projection matrix to obtain modal features; The modal features are subjected to layer normalization and dimension alignment to output a spatial vector with uniform dimensions.

4. The multimodal metadata alignment and fusion method as described in claim 1, characterized in that, The step of identifying the semantic relationship of the alignment candidate pair according to the preset semantic rules and semantic relationship classification model further includes: A preliminary semantic relationship determination is performed on the alignment candidate pairs based on preset semantic rules; The preliminary semantic relationships are input into the semantic relationship classification model to calculate the semantic association strength of the aligned candidate pairs; The semantic relationship of the alignment candidate pair is identified based on the semantic association strength and the preset relationship threshold.

5. A multimodal metadata alignment and fusion apparatus, used to implement the multimodal metadata alignment and fusion method as described in any one of claims 1 to 4, characterized in that, include: The data extraction module is configured to extract metadata from structured data, unstructured text data, and image data respectively, and generate multimodal metadata with semantic annotations. The mapping and encoding module is configured to establish a shared semantic embedding space, map the multimodal metadata to the shared semantic embedding space for encoding, and obtain a unified space vector; A relationship recognition module is configured to determine alignment candidate pairs from the unified space vector through similarity calculation and to identify the semantic relationship of the alignment candidate pairs; The conflict resolution module is configured to perform conflict detection on the alignment candidate pairs and their corresponding semantic relationships, and resolve conflicts based on weighted fusion to obtain a unified metadata system. The step of determining alignment candidate pairs from the unified spatial vector through similarity calculation and identifying the semantic relationships of the alignment candidate pairs further includes: Construct a cross-modal similarity matrix and calculate the similarity metric between the unified space vectors; Based on the similarity metric, the target metadata corresponding to a preset number is selected as the initial candidate pairs. The initial candidate pairs are filtered using preset filtering rules to determine the initial candidate pairs that meet the preset requirements as alignment candidate pairs; The semantic relationships of the alignment candidate pairs are identified based on preset semantic rules and a semantic relationship classification model; The preset filtering rules are a multi-level filtering mechanism; The first-level filtering is based on a hard threshold, which removes initial candidate pairs that do not meet the basic similarity requirements. The second-level filtering implements semantic consistency checks, which verify the conceptual compatibility of the initial candidate pairs through a pre-built thesaurus and domain ontology. The third-level filtering uses contextual analysis to check whether the semantic roles of candidate pairs match in their respective data environments. The final output maximizes the coverage of aligned candidate pairs with potential semantic associations while meeting the preset precision.

6. A multimodal metadata alignment and fusion device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of the multimodal metadata alignment and fusion method according to any one of claims 1 to 4.

7. A storage medium, characterized in that, It stores a computer program executable by a multimodal metadata alignment and fusion device, which, when run on the multimodal metadata alignment and fusion device, causes the multimodal metadata alignment and fusion device to perform the steps of the multimodal metadata alignment and fusion method according to any one of claims 1 to 4.

Images