Multi-modal data semantic alignment method and system based on knowledge graph embedding

Abstract

The invention discloses a multi-modal data semantic alignment method and system based on knowledge graph embedding, relates to the technical field of inspection and detection, and solves the problems of low data accuracy and interpretability after multi-modal data fusion. According to the embodiment of the invention, through a knowledge graph embedding and semantic alignment mechanism, the semantic gap problem of cross-modal data is solved, and accurate association of multi-modal data in an inspection and detection scene is realized; the optimized fusion vector obtained by fusion enhances the interpretability and consistency of the data, and improves the detection accuracy and robustness of a subsequent detection model.

Description

Technical Field

[0001] This invention relates to the field of testing and inspection technology, and in particular to a multimodal data semantic alignment method and system based on knowledge graph embedding. Background Technology

[0002] In the modern field of inspection and testing, with the rapid development of the Internet of Things, sensor technology, and artificial intelligence, the evaluation dimensions of the tested objects are becoming increasingly rich, and multimodal data has become the core foundation supporting accurate testing and intelligent decision-making. How to effectively integrate these heterogeneous multimodal data and fully explore the complementary information between different modalities is a key technical bottleneck for improving the accuracy, efficiency, and intelligence level of inspection and testing.

[0003] Currently, in the application of traditional multimodal data fusion methods in the field of inspection and testing, feature-level concatenation is commonly used. This typically involves first extracting features independently from each modality, and then directly concatenating the extracted feature vectors into a single high-dimensional feature vector, which is then input into a subsequent classification or regression model to complete the detection task.

[0004] The implementation of the aforementioned feature-level concatenation method relies on manual feature engineering or general-purpose deep learning models. At the manual feature engineering level, technicians need to design feature extraction operators empirically based on specific detection scenarios. This approach is not only time-consuming and labor-intensive, but also difficult to adapt to complex and ever-changing detection objects and environments. While general-purpose deep learning models can automatically extract features, their network structure and parameter optimization are not customized for the specific semantics of the inspection and detection domain, resulting in a lack of explicit modeling of domain knowledge in the feature learning process.

[0005] However, the field of inspection and testing exhibits strong semantic relationships, with different modalities often corresponding to specific quality attributes or defect characteristics of the inspected object. Traditional methods ignore these domain-specific semantic relationships, leading to significant semantic gaps in the feature space of cross-modal data: each modal feature only reflects the surface information of its own data, and the deep semantic complementarity between modalities cannot be effectively mined, resulting in low accuracy and interpretability of the fused data.

[0006] Therefore, a method and system for semantic alignment of multimodal data based on knowledge graph embedding is needed. Summary of the Invention

[0007] To address the issues of low accuracy and interpretability of multimodal data fusion in existing technologies, this invention provides a multimodal data semantic alignment method and system based on knowledge graph embedding, which can improve the accuracy and interpretability of fused data. The specific technical solution is as follows: In a first aspect, embodiments of this application provide a multimodal data semantic alignment method based on knowledge graph embedding, including: A knowledge graph in the field of inspection and testing is constructed; multimodal feature vectors are extracted from heterogeneous inspection and testing data, including spectral images, real-time sensor data, and text reports; these multimodal feature vectors are projected onto the semantic space corresponding to the entity embeddings of the knowledge graph to obtain multimodal projection vectors; the semantic similarity between the multimodal projection vectors and the entity embeddings is calculated, and based on this semantic similarity, the multimodal projection vectors and the entity embeddings are semantically aligned; the aligned multimodal projection vectors and the entity embeddings are fused to generate an optimized fusion vector for downstream inspection and testing tasks.

[0008] Preferably, fusing the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks includes: performing weighted fusion of the aligned multimodal projection vector with the entity embedding to obtain a basic fusion vector; performing intra-modal attention operations and inter-modal attention operations based on the basic fusion vector and the aligned multimodal projection vector to obtain an enhanced fusion vector; and generating the optimized fusion vector based on the enhanced fusion vector, the entity embedding, and the downstream inspection and testing task.

[0009] Preferably, the weighted fusion of the aligned multimodal projection vector and the entity embedding to obtain a basic fusion vector includes: concatenating and pooling the aligned multimodal projection vector to obtain a context vector; calculating the modal feature weights and entity embedding weights using a gated neural network based on the multimodal feature vector, the entity embedding, and the context vector; and weighted summing the aligned multimodal projection vector and the entity embedding based on the modal feature weights and the entity embedding weights to obtain the basic fusion vector.

[0010] Preferably, the construction of the knowledge graph in the field of inspection and testing includes: extracting entities from structured data and unstructured documents in the field of inspection and testing, wherein the categories of the entities include test objects, material components, test indicators, instruments and equipment, instrument parameters, test conditions, defects and phenomena, and test results and judgments; identifying the relationships between the entities based on predefined domain relationship types; constructing the knowledge graph based on the entities and the relationships; and before projecting the multimodal feature vectors onto the semantic space corresponding to the entity embeddings of the knowledge graph, the method further includes: using a knowledge graph embedding model to map different categories of entities to different semantic subspaces.

[0011] Preferably, the step of calculating the semantic similarity between the multimodal projection vector and the entity embedding, and semantically aligning the multimodal projection vector and the entity embedding based on the semantic similarity, includes: calculating the semantic similarity between the multimodal projection vector and the entity embedding; calculating the positive attention weight from the multimodal projection vector to the entity embedding and the negative attention weight from the entity embedding to the multimodal projection vector based on the semantic similarity; and adjusting the multimodal projection vector based on the comprehensive attention weight to align the multimodal projection vector and the entity embedding; wherein the comprehensive attention weight is calculated based on the positive attention weight and the negative attention weight, and the adjustment rate of the multimodal projection vector is calculated based on the distribution entropy of the comprehensive attention weight.

[0012] Preferably, before adjusting the multimodal projection vector based on the comprehensive attention weight, the method further includes: resetting the comprehensive attention weight between the multimodal projection vector and the entity embedding that does not conform to the inspection and detection logic to 0 based on a preset domain constraint rule; wherein the domain constraint rule is used to indicate the correct inspection and detection logic.

[0013] Preferably, the multimodal feature vector includes a visual feature vector, a temporal feature vector, and a textual feature vector; the extraction of multimodal feature vectors from heterogeneous data for testing and detection includes: extracting the visual feature vector from the spectral image using a convolutional neural network; extracting the temporal feature vector from the real-time sensor data using a recurrent neural network with an integrated temporal attention mechanism; extracting the textual feature vector from the text report using a natural language processing model; and performing L2 normalization on the visual feature vector, the temporal feature vector, and the textual feature vector.

[0014] Secondly, embodiments of this application provide a multimodal data semantic alignment system based on knowledge graph embedding, applied to the method described in the first aspect, the system comprising: The knowledge graph module constructs a knowledge graph for the field of testing and inspection. The feature engineering module is used to extract multimodal feature vectors from heterogeneous inspection and testing data, including spectral images, real-time sensor data, and text reports. The projection module is used to project the multimodal feature vector onto the semantic space corresponding to the entity embedding of the knowledge graph, so as to obtain the multimodal projection vector. An alignment module is used to calculate the semantic similarity between the multimodal projection vector and the entity embedding, and to perform semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity. The generation module is used to fuse the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks.

[0015] Thirdly, embodiments of this application provide a computing device, including: a memory for storing a program; and a processor for loading the program to execute the method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: by using knowledge graph embedding and semantic alignment mechanisms, the semantic bias problem after cross-modal data fusion is solved, and accurate association of multimodal data in inspection and detection scenarios is achieved; the optimized fusion vector obtained by fusion enhances the interpretability and consistency of the data, and improves the accuracy and robustness of subsequent detection models. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0019] Figure 1 A flowchart illustrating a multimodal data semantic alignment method based on knowledge graph embedding, provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a multimodal data semantic alignment system based on knowledge graph embedding provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] To address the issues of low accuracy and interpretability of multimodal data fusion in existing technologies, this invention provides a multimodal data semantic alignment method and system based on knowledge graph embedding, which can improve the accuracy and interpretability of fused data.

[0025] Please see Figure 1 , Figure 1 This application provides a flowchart illustrating a multimodal data semantic alignment method based on knowledge graph embedding, which is applied to a computing device. Figure 1 As shown, the method includes: Step 101: Calculate the knowledge graph in the field of inspection and testing using computing equipment.

[0026] The computing device can be a server or a terminal, specifically a server or terminal in an inspection and testing scenario. As an upstream device of the inspection and testing equipment, it provides semantic information of multimodal data to the inspection and testing equipment, enabling the inspection and testing equipment to analyze the collected multimodal data and output accurate inspection and testing results. The computing device can also be the inspection and testing equipment itself.

[0027] The computing device can obtain data from authoritative databases in the field of testing and inspection, including testing standard libraries, instrument manuals, material property libraries, and historical test reports. These data sources may exist in structured or unstructured formats.

[0028] The computing device can then preprocess this acquired data. Specifically, for unstructured data, the computing device can use existing document parsing tools to extract text content, removing formatting marks and irrelevant characters; then, for this text content, general text preprocessing techniques are applied, including word segmentation, stop word removal, lowercase unification, and punctuation removal, to standardize the text; the text is then segmented into independent sentences for subsequent entity and relation extraction.

[0029] For the testing and inspection field, computing devices can introduce field-specific stop word lists and reserved word lists during the preprocessing stage. For example, key unit symbols (such as "ppm" and "℃") and testing terms (such as "calibration" and "deviation") can be retained to avoid accidental deletion during cleaning and ensure the integrity of the field's semantics.

[0030] Among them, the computing device can predefine entity categories according to the needs of the inspection and testing field.

[0031] Preferably, the entity category includes the object of detection, material composition, detection indicators, instruments and equipment, instrument parameters, detection conditions, defects and phenomena, as well as detection results and judgments.

[0032] For example, entities for detection indicators include "pH value", "concentration", and "spectral peak value"; entities for instrument parameters include "wavelength", "accuracy", and "sampling rate"; entities for material composition include "water", "sodium ion", and "polymer"; entities for detection conditions include "temperature", "humidity", and "pressure"; and entities for defect types include "crack", "corrosion", and "bubble".

[0033] Among them, computing devices can use Named Entity Recognition (NER) models to identify entities from preprocessed text.

[0034] Specifically, the computing device can first use labeled data from the field of inspection and testing to fine-tune the NER model and improve the accuracy of entity recognition in the field; secondly, for rare entities (such as specific instrument parameters), an active learning strategy is adopted to manually label unlabeled samples through uncertainty sampling, so as to improve the generalization ability of the NER model in the professional field and ensure the coverage and accuracy of entity extraction.

[0035] Among them, the computing device can predefine relationship types specific to the field of inspection and testing, such as "affected by temperature", "corresponding to defect type", "belonging to the testing index", "has parameters", "leads to result deviation", etc.

[0036] Specifically, computing devices can use relation extraction models to identify relationships between entities in sentences, such as rule-based methods based on dependency parsing or deep learning models. Consistency checks can then be performed using domain-specific rules (such as temporal constraints in the detection process), for example, "sample preprocessing" must precede "metric detection" to correct contradictory relationships. This improves the reliability of relation extraction and overcomes the problems of relation sparsity and conflict.

[0037] The computing device can then organize the extracted entities and relations into a graph structure, where nodes represent entities and edges represent relations. Specifically, this graph structure can be stored as a graph database or as Resource Description Framework (RDF) triples.

[0038] Step 102: The computing device extracts the multimodal feature vectors of the heterogeneous data from the inspection and testing.

[0039] The heterogeneous testing data includes spectral images of the tested objects, real-time sensor data during the testing process, and text reports after testing. Specifically, this heterogeneous testing data can correspond to the same or the same type of tested object. It is understood that when the heterogeneous testing data corresponds to multiple types of tested objects, the computing device can group the heterogeneous data based on the tested objects.

[0040] Preferably, the multimodal feature vector includes a visual feature vector, a temporal feature vector, and a text feature vector; the computing device can use a convolutional neural network to extract the visual feature vector from the spectral image; use a recurrent neural network with an integrated temporal attention mechanism to extract the temporal feature vector from the real-time sensor data; use a natural language processing model to extract the text feature vector from the text report; and then perform L2 normalization processing on the visual feature vector, the temporal feature vector, and the text feature vector.

[0041] This convolutional neural network is trained on scientific image datasets, such as those from materials science or biomedical imaging, in the field of inspection and testing, and can more accurately capture the spectral features of the spectral image. Specifically, the convolutional neural network includes convolutional layers, activation layers, pooling layers, and fully connected layers.

[0042] The recurrent neural network includes multiple Long Short-Term Memory (LSTM) layers. Each LSTM layer controls the information flow through a gating mechanism. The input real-time sensor data passes through multiple LSTM layers, and the hidden state is output at each time step. Finally, the time-series feature vector is generated by global average pooling or by taking the last hidden state.

[0043] The natural language processing model used is the BERT model. The computing device can first pre-train and fine-tune the BERT model using text corpora from the inspection and detection domain to learn the semantics of domain-specific vocabulary. Then, key phrases are identified from the text using a domain terminology database or rules. These phrases are encoded separately by BERT and weighted and fused with the overall text features, enhancing the text features' ability to capture the core semantics of the detection scenario and addressing BERT's insufficient understanding of specialized domain terminology.

[0044] Specifically, the computing device can first obtain the word embedding vector corresponding to each word in the text through the BERT model. Each of these vectors contains the semantic information of a single word. Then, average pooling is used to calculate the average value of all word embedding vectors according to their positions, resulting in a vector of uniform length, which serves as the text feature vector representing the overall text.

[0045] In this process, L2 normalization is performed on the feature vectors of each modality to ensure that they are at similar or the same scale, which facilitates subsequent semantic alignment.

[0046] Step 103: The computing device projects the multimodal feature vector onto the semantic space corresponding to the entity embedding of the knowledge graph to obtain the multimodal projection vector.

[0047] The computing device can employ a knowledge graph embedding model to map entities and relationships in the knowledge graph to a low-dimensional vector space. Preferably, the computing device can use the knowledge graph embedding model to map different categories of entities to different semantic subspaces.

[0048] The knowledge graph embedding model is the TransE model. The basic assumption of the TransE model is that the sum of the head entity vector and the relation vector should approximately equal the tail entity vector. For each triple (head entity h, relation r, tail entity t), its scoring function is: ; in, These are learnable importance weights used to indicate the degree of importance of different relations r.

[0049] The scoring function is the training objective of the knowledge graph embedding model; it quantifies the probability that a triple is true. The smaller the score of a triple (h, r, t), the stronger and more reasonable the semantic association between the head entity h, relation r, and tail entity t. Through an alignment process, the scoring function can transfer its learned semantic associations to multimodal data, enabling the multimodal data to correctly associate with the correct entities and relations in the knowledge graph.

[0050] The computing device can implement projection using a fully connected neural network, projecting multimodal feature vectors onto the same vector space as the knowledge graph embedding, i.e., the aforementioned low-dimensional vector space. The projection calculation formula includes: ; in, It is the eigenvector of the i-th mode. It is the projection vector of the i-th mode. and These are modality-specific learnable projection parameters. Different modalities share the same target dimension d, ensuring spatial consistency. The set of projection vectors for each modality constitutes the multimodal projection vector.

[0051] The loss function of this fully connected neural network includes cross-modal consistency loss, which forces different modal projection vectors of the same sample to be close to each other in the semantic space, laying the foundation for subsequent alignment. Simultaneously, domain-adaptive initialization is used, initializing the projection parameters with the statistical properties of knowledge graph entity embeddings (such as mean and variance) to accelerate convergence.

[0052] Step 104: The computing device calculates the semantic similarity between the multimodal projection vector and the entity embedding, and performs semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity.

[0053] The computing device can use an attention mechanism to calculate the semantic similarity between the multimodal projection vectors and the knowledge graph entity embeddings. For each modality's projection vector... Based on this semantic similarity, its embeddings with all entities in the knowledge graph are calculated. Attention score: ; Where sim() is the cosine similarity function, N is the number of entities in the knowledge graph, and the attention score is... Represents the projection vector The semantic association strength with entity j.

[0054] In deep learning and knowledge graph embedding, both the representation of an entire document and the representation of a single word are mapped to points in the same vector space. This vector is designed to encode all the semantic information of its corresponding object.

[0055] In the computation of attention mechanisms, the computation With all The similarity process can be understood as follows: using this macroscopic projection feature as a query, a retrieval is performed in the semantic dictionary of the entire knowledge graph.

[0056] Attention weight This indicates how much of the semantic components in this macroscopic projection feature are derived from microscopic concepts. What has been contributed or explained.

[0057] For example, a spectral image containing rust and cracks might have its visual feature vectors weighted highly with the embeddings of multiple entities such as "metal corrosion," "surface cracks," and "iron oxide." This is a soft alignment that accurately reflects the reality that macroscopic data contains multiple microscopic semantics.

[0058] This seemingly unequal computation, through the softening and weighting of the attention mechanism, can achieve semantic decomposition and association from macro-modal features to micro-entity concepts.

[0059] Preferably, the computing device can calculate the positive attention weight from the multimodal projection vector to the entity embedding and the negative attention weight from the entity embedding to the multimodal projection vector based on the semantic similarity; and adjust the multimodal projection vector based on the comprehensive attention weight to align the multimodal projection vector and the entity embedding.

[0060] The comprehensive attention weight is calculated based on the positive attention weight and the negative attention weight. Specifically, the comprehensive attention weight can be the average of the positive attention weight and the negative attention weight, or it can be calculated by weighting based on learnable weight parameters.

[0061] Preferably, before adjusting the multimodal projection vector based on the comprehensive attention weight, the computing device may reset the comprehensive attention weight between the multimodal projection vector and the entity embedding that does not conform to the inspection and detection logic to 0 based on preset domain constraint rules.

[0062] The domain constraint rules are used to indicate the correct detection logic.

[0063] For example, the domain constraint rules include: IF Entity Type = Instrument Parameters AND Feature Source = Text Report THEN Alignment Allowed (Comprehensive Attention Weights are calculated normally); IF Entity Type = Material Composition AND Feature Source = Sensor Data THEN Alignment Prohibited (Overall Attention Weight Reset to 0).

[0064] The computing device can adjust the multimodal projection vector, moving it towards entities with high attention weights, to align the multimodal projection vector with the entity embedding. The formula for adjusting the multimodal projection vector includes: ; ; ; in, Let i be the projection vector adjusted for the i-th mode. Let be the adjustment rate of the projection vector of the i-th mode. The entropy of the attention distribution, where N is the number of entities. Let be the comprehensive attention weight for the i-th modality.

[0065] The adjustment rate of the multimodal projection vector The distribution entropy is calculated based on the comprehensive attention weight; when attention is concentrated (low entropy), the adjustment range is large; when attention is scattered (high entropy), the adjustment range is small.

[0066] Step 105: The computing device fuses the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks.

[0067] The computing device can fuse the aligned multimodal feature vectors with the knowledge graph entity embeddings to generate a unified optimized fusion vector, which will be used as the input for downstream detection tasks (such as anomaly detection, result determination, etc.).

[0068] Preferably, the computing device can perform weighted fusion of the aligned multimodal projection vector and the entity embedding to obtain a basic fusion vector; based on the basic fusion vector and the aligned multimodal projection vector, perform intramodal attention operation and intermodal attention operation to obtain an enhanced fusion vector; and based on the enhanced fusion vector, the entity embedding, and the downstream inspection and detection task, generate the optimized fusion vector.

[0069] The formula for calculating the basic fusion vector includes: ; in, and The modal feature weights and entity embedding weights are initialized as learnable parameters, where N' is the number of entities participating in the fusion and M is the number of modalities.

[0070] In one possible implementation, the computing device concatenates and averages the aligned multimodal projection vectors to obtain a context vector; based on the multimodal feature vector, the entity embedding, and the context vector, it calculates the modal feature weights and entity embedding weights using a gated neural network.

[0071] Among them, context vector The calculation formulas include: , which represents the overall semantic context of the current sample.

[0072] Among them, modal feature weights The calculation formulas include: Entity embedding weights The calculation formulas include: . , The weight matrix is ​​a learnable matrix. These are learnable bias parameters. () represents the sigmoid function.

[0073] Based on the basic fusion vector, computing devices can delve deeper into the complex interactions within and between modalities.

[0074] First, the computing device can combine the basic fusion vector with the modal features into a sequence: X=[ ; ; ;...; ].

[0075] Then, the intramodal self-attention operation is performed.

[0076] 1. Query, key-value projection: ; in, , and Let be the learnable projection matrix for intramodal attention; , and These are the query vector, key vector, and value vector for intramodal attention, respectively.

[0077] 2. Self-attention calculation: ; in, This is the intra-modal attention weight matrix, reflecting the importance relationships between different positions within the sequence; softmax() is the activation function. The dimension of the key vector; The first output vector is the output vector after the self-attention mechanism interacts with and enhances the input sequence X. It integrates information from each position within each modality and reflects the attentional relationship between different elements within the modality.

[0078] Then, cross-modal attention operations are performed.

[0079] 1. Using each position as the query and the other positions as the key, calculate the cross-projection vector: ; ; ; in, , and Let be the learnable projection matrix for intermodal attention. This is the query vector for intermodal attention. and Let be the key vector and value vector for intermodal attention.

[0080] 2. Cross-attention calculation: ; ; ; in, Let be the attention weights of the i-th position to the other positions. It is the context information obtained from other positions at position i through a cross-attention mechanism. () represents the layer normalization operation. The vector at the i-th position of the second output vector.

[0081] 3. Global aggregation.

[0082] Aggregate the enhanced sequences: ,in, () indicates global pooling. To enhance the fusion vector, it includes rich interaction information from both intramodal and cross-modal modes.

[0083] Specifically, this global pooling can be global average pooling.

[0084] Then, the computing device can directly... As an optimized fusion vector, the enhanced fusion vector with enhanced interaction can also be fused with domain knowledge a second time based on the enhanced fusion vector, and optimized for downstream tasks to obtain a further optimized fusion vector.

[0085] 1. Embedding each knowledge graph entity Calculate the knowledge attention weights: ; in, It calculates the similarity between the enhanced fusion vector and the entity embedding, measuring the degree of semantic association between them; the softmax() function is used to convert the similarity into attention weights. This allows entities with high relevance to be given greater weight, enabling the focus to be placed on knowledge entities that are most relevant to multimodal information.

[0086] 2. Generate knowledge residual terms: ; in, Knowledge residuals are entity knowledge in the knowledge graph that is relevant to the current task. They are incorporated into the original features in the form of "supplementary differences" to make up for the deficiencies of the original features in terms of domain semantics and background knowledge. A learnable knowledge projection matrix used to embed entities from a knowledge graph. Projecting into a new semantic subspace that better matches the current multimodal fusion task can be achieved by training the model based on backpropagation when the model generates errors in downstream tasks.

[0087] 3. Residual Fusion: ; in, It is a hyperparameter that controls the intensity of knowledge injection. Enhancing features for knowledge. Hyperparameters. It can be a preset value or based on... Calculated, such as ,in and These are the learnable weight and bias parameters.

[0088] 4. Standardization of task perception: ; ; Where t is a preset task encoding vector, determined based on the downstream inspection and testing task; The task-adaptive matrix is ​​responsible for highlighting the feature dimensions that are useful for the downstream task and suppressing useless or interfering dimensions. To achieve element-wise multiplication, the task code t is multiplied element by element with the feature vector, thereby modulating the feature dimension by dimension of the task information. () represents the layer normalization operation; Knowledge augmentation vectors adapted for downstream tasks; This is the final optimized fusion vector.

[0089] In this embodiment, the semantic bias problem after cross-modal data fusion is solved by knowledge graph embedding and semantic alignment mechanism, and the accurate association of multimodal data in inspection and detection scenarios is realized. The optimized fusion vector obtained by fusion enhances the interpretability and consistency of the data and improves the accuracy and robustness of subsequent detection models.

[0090] The method provided in the embodiments of this application has been described above. The system provided in the embodiments of this application will be described below.

[0091] Please see Figure 2 , Figure 2 A schematic diagram of the structure of a multimodal data semantic alignment system based on knowledge graph embedding provided in this application embodiment is shown below. Figure 2 As shown, the system 20 includes: Knowledge Graph Module 201: Constructing a knowledge graph in the field of inspection and testing; The feature engineering module 202 is used to extract multimodal feature vectors from heterogeneous inspection and testing data, which includes spectral images, real-time sensor data, and text reports. Projection module 203 is used to project the multimodal feature vector onto the semantic space corresponding to the entity embedding of the knowledge graph to obtain a multimodal projection vector; Alignment module 204 is used to calculate the semantic similarity between the multimodal projection vector and the entity embedding, and to perform semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity. The generation module 205 is used to fuse the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks.

[0092] Preferably, the generation module 205 is specifically used to perform weighted fusion of the aligned multimodal projection vector and the entity embedding to obtain a basic fusion vector; based on the basic fusion vector and the aligned multimodal projection vector, perform intramodal attention operation and intermodal attention operation to obtain an enhanced fusion vector; and based on the enhanced fusion vector, the entity embedding and the downstream inspection and detection task, generate the optimized fusion vector.

[0093] Preferably, the generation module 205 is specifically used to concatenate and average pool the aligned multimodal projection vector to obtain a context vector; calculate the modal feature weights and entity embedding weights through a gated neural network based on the multimodal feature vector, the entity embedding, and the context vector; and perform a weighted summation of the aligned multimodal projection vector and the entity embedding based on the modal feature weights and the entity embedding to obtain the basic fusion vector.

[0094] Preferably, the knowledge graph module 201 is specifically used to extract entities from structured data and unstructured documents in the field of inspection and testing. The categories of these entities include test objects, material composition, test indicators, instruments and equipment, instrument parameters, test conditions, defects and phenomena, as well as test results and judgments. Based on predefined domain relationship types, the relationships between these entities are identified. The knowledge graph is constructed based on these entities and their relationships. Specifically, the knowledge graph module 201 is used to map different categories of entities to different semantic subspaces using a knowledge graph embedding model.

[0095] Preferably, the alignment module 204 is specifically used to calculate the semantic similarity between the multimodal projection vector and the entity embedding; calculate the positive attention weight from the multimodal projection vector to the entity embedding and the negative attention weight from the entity embedding to the multimodal projection vector based on the semantic similarity; and adjust the multimodal projection vector based on the comprehensive attention weight to align the multimodal projection vector and the entity embedding; wherein the comprehensive attention weight is calculated based on the positive attention weight and the negative attention weight, and the adjustment rate of the multimodal projection vector is calculated based on the distribution entropy of the comprehensive attention weight.

[0096] Preferably, the system 20 further includes a constraint module 206, which is used to reset the comprehensive attention weight between the multimodal projection vector and the entity embedding that does not conform to the inspection and detection logic to 0 based on a preset domain constraint rule; wherein the domain constraint rule is used to indicate the correct inspection and detection logic.

[0097] Preferably, the feature engineering module 202 is specifically used to extract the visual feature vector from the spectral image using a convolutional neural network; extract the temporal feature vector from the real-time sensor data using a recurrent neural network with an integrated temporal attention mechanism; extract the text feature vector from the text report using a natural language processing model; and perform L2 normalization processing on the visual feature vector, the temporal feature vector, and the text feature vector.

[0098] The multimodal data semantic alignment system based on knowledge graph embedding provided in this application can be understood by referring to the relevant content in the foregoing method embodiment section, and will not be repeated here.

[0099] like Figure 3 As shown, Figure 3 This is a schematic diagram of a possible logical structure of a computing device provided in an embodiment of this application. The computing device 30 includes a processor 301, a communication interface 302, a memory 303, and a bus 304. The processor 301, the communication interface 302, and the memory 303 are interconnected via the bus 304. In an embodiment of this application, the processor 301 is used to control and manage the operation of the computing device 30. For example, the processor 301 is used to execute... Figure 1 The steps in the embodiments and / or other processes used in the techniques described herein. Communication interface 302 is used to support communication by computing device 30. Memory 303 is used to store program code and data of computing device 30.

[0100] The processor 301 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 304 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0101] In another embodiment of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the above-described... Figure 1 The method described in the embodiments.

[0102] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0103] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A multimodal data semantic alignment method based on knowledge graph embedding, characterized in that, The method includes: Constructing a knowledge graph in the field of testing and inspection; Extract multimodal feature vectors from heterogeneous inspection and testing data, which includes spectral images, real-time sensor data, and text reports; The multimodal feature vector is projected onto the semantic space corresponding to the entity embedding of the knowledge graph to obtain the multimodal projection vector; Calculate the semantic similarity between the multimodal projection vector and the entity embedding, and perform semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity; The aligned multimodal projection vectors are fused with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks.

2. The method according to claim 1, characterized in that, The step of fusing the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks includes: The aligned multimodal projection vectors are weighted and fused with the entity embeddings to obtain a basic fusion vector; Based on the base fusion vector and the aligned multimodal projection vector, intramodal attention operations and intermodal attention operations are performed to obtain the enhanced fusion vector; The optimized fusion vector is generated based on the enhanced fusion vector, the entity embedding, and the downstream inspection and detection task.

3. The method according to claim 2, characterized in that, The step of weightedly fusing the aligned multimodal projection vector with the entity embedding to obtain a basic fusion vector includes: The aligned multimodal projection vectors are concatenated and average pooled to obtain the context vector; Based on the multimodal feature vector, the entity embedding, and the context vector, the modal feature weights and entity embedding weights are calculated using a gated neural network; Based on the modal feature weights and the entity embedding weights, the aligned multimodal projection vectors and the entity embeddings are weighted and summed to obtain the basic fusion vector.

4. The method according to claim 1, characterized in that, The semantic space includes a semantic subspace; the construction of a knowledge graph in the field of inspection and testing includes: Entities are extracted from structured data and unstructured documents in the field of inspection and testing. The categories of the entities include test objects, material composition, test indicators, instruments and equipment, instrument parameters, test conditions, defects and phenomena, as well as test results and judgments. Identify the relationships between the entities based on predefined domain relationship types; The knowledge graph is constructed based on the entities and the relationships; Before projecting the multimodal feature vectors onto the semantic space corresponding to the entity embeddings of the knowledge graph, the method further includes: A knowledge graph embedding model is used to map entities of different categories to different semantic subspaces.

5. The method according to claim 1, characterized in that, The step of calculating the semantic similarity between the multimodal projection vector and the entity embedding, and performing semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity, includes: Calculate the semantic similarity between the multimodal projection vector and the entity embedding; Based on the semantic similarity, calculate the positive attention weight from the multimodal projection vector to the entity embedding, and the negative attention weight from the entity embedding to the multimodal projection vector; Based on the comprehensive attention weights, the multimodal projection vectors are adjusted to align the multimodal projection vectors with the entity embeddings; wherein, the comprehensive attention weights are calculated based on the forward attention weights and the reverse attention weights, and the adjustment rate of the multimodal projection vectors is calculated based on the distribution entropy of the comprehensive attention weights.

6. The method according to claim 5, characterized in that, Before adjusting the multimodal projection vector based on the comprehensive attention weights, the method further includes: Based on preset domain constraint rules, the integrated attention weight between the multimodal projection vector and the entity embedding that does not conform to the inspection and detection logic is reset to 0; wherein, the domain constraint rules are used to indicate the correct inspection and detection logic.

7. The method according to any one of claims 1-6, characterized in that, The multimodal feature vectors include visual feature vectors, temporal feature vectors, and text feature vectors; the extraction of multimodal feature vectors from heterogeneous data for testing and detection includes: The visual feature vector is extracted from the spectral image using a convolutional neural network. A recurrent neural network with an integrated temporal attention mechanism is used to extract temporal feature vectors from the real-time data of the sensor; A natural language processing model is used to extract text feature vectors from the text report; The visual feature vector, the temporal feature vector, and the text feature vector are subjected to L2 normalization.

8. A multimodal data semantic alignment system based on knowledge graph embedding, characterized in that, The system, applied to the method of any one of claims 1-7, comprises: The knowledge graph module constructs a knowledge graph for the field of testing and inspection. The feature engineering module is used to extract multimodal feature vectors from heterogeneous inspection and testing data, which includes spectral images, real-time sensor data, and text reports. The projection module is used to project the multimodal feature vectors onto the semantic space corresponding to the entity embeddings of the knowledge graph, so as to obtain multimodal projection vectors. An alignment module is used to calculate the semantic similarity between the multimodal projection vector and the entity embedding, and to perform semantic alignment between the multimodal projection vector and the entity embedding based on the semantic similarity. A generation module is used to fuse the aligned multimodal projection vector with the entity embedding to generate an optimized fusion vector for downstream inspection and testing tasks.

9. A computing device, characterized in that, include: Memory, used to store programs; A processor for loading the program to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method of any one of claims 1-7.

Images