A method and apparatus for named entity recognition in multimodal knowledge graph construction

Abstract

This invention discloses a method and apparatus for named entity recognition in the construction of a multimodal knowledge graph. The method includes: constructing a multimodal graph based on text and image information, the multimodal graph including text nodes, visual nodes, and edges connecting different nodes; inputting the multimodal graph into multiple sequentially connected fusion layers; for each fusion layer, performing intramodal fusion on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes fused with contextual semantic information; performing cross-modal fusion on the fused nodes to obtain cross-modal fused text nodes and visual nodes; using the cross-modal fused nodes as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained; fusing the text node features output by the last fusion layer with the original text features output by BERT; and performing CRF decoding on the fused features to obtain the named entity recognition result. This application enables full interaction between image and text information and eliminates visual noise interference in the image.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, and in particular to a method and apparatus for identifying named entities in the construction of multimodal knowledge graphs. Background Technology

[0002] With the widespread adoption of 5G technology and the explosive growth of internet data, information is characterized by the coexistence of multiple modalities such as text, images, and videos. How to efficiently process and utilize this massive amount of multimodal unstructured data has become a significant challenge in the field of artificial intelligence. Knowledge graphs, as a structured representation of knowledge, can clearly describe entities in the real world and their relationships, and have become a key infrastructure for information processing and intelligent applications.

[0003] Named Entity Recognition (NER) is a core and fundamental step in the knowledge graph construction process. Its task is to identify and classify entities with specific meanings, such as names of people, places, and organizations, from unstructured text. The accuracy and efficiency of NER directly determine the quality of the knowledge graph and the effectiveness of subsequent applications such as knowledge reasoning and intelligent question answering.

[0004] Currently, traditional named entity recognition primarily relies on a single text modality. However, in scenarios such as social media, information is often composed of both text and images. Simply relying on text information ignores the rich contextual clues contained in images, limiting the accuracy of entity recognition, especially when dealing with ambiguous or brief text content. Therefore, existing methods incorporate image features to assist text entity recognition, i.e., performing named entity recognition by fusing multimodal information such as images and text.

[0005] However, on the one hand, since images often contain background or noisy objects unrelated to the text content, this irrelevant information may introduce noise during the fusion process, thereby interfering with the accuracy of entity recognition and weakening model performance. On the other hand, due to the inherent differences in feature representation between text and image data, a cross-modal semantic gap exists. Therefore, how to effectively align and fuse the semantic information of the two modalities to achieve deep interaction remains a core challenge.

[0006] Therefore, there is an urgent need for a method and device for identifying named entities in the construction of multimodal knowledge graphs to solve problems such as insufficient cross-modal information interaction and susceptibility to noise interference. Summary of the Invention

[0007] This invention provides a method and apparatus for named entity recognition in multimodal knowledge graph construction, which enables full interaction between image and text information and eliminates visual noise interference in images. The technical solution is as follows: On the one hand, a method for named entity recognition in the construction of multimodal knowledge graphs is provided, the method comprising: A multimodal graph is constructed based on text and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. The multimodal graph is input into multiple sequentially connected fusion layers. For each fusion layer, intramodal fusion is performed on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information. Cross-modal fusion is then performed on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes. The cross-modal fused text nodes and visual nodes are used as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. The text node features output from the last fusion layer are fused with the original text features output from BERT, and the fused features are then CRF decoded to obtain the final named entity recognition result.

[0008] On the other hand, a named entity recognition device is provided for multimodal knowledge graph construction, the device comprising: A construction unit is used to construct a multimodal graph based on text information and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. The fusion unit is used to input the multimodal graph into multiple fusion layers that are sequentially connected. For each fusion layer, it performs intramodal fusion on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information; it performs cross-modal fusion on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes; it uses the cross-modal fused text nodes and visual nodes as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. The decoding unit is used to fuse the text node features output by the last fusion layer with the original text features output by BERT, and perform CRF decoding on the fused features to obtain the final named entity recognition result.

[0009] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing computer programs, and the processor for executing the computer programs stored in the memory to implement the steps of the named entity recognition method in the multimodal knowledge graph construction described above.

[0010] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the named entity recognition method in the multimodal knowledge graph construction described above.

[0011] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the named entity recognition method in the construction of the multimodal knowledge graph described above.

[0012] This invention provides a method and apparatus for named entity recognition in the construction of a multimodal knowledge graph. In this embodiment, a multimodal graph is first constructed using text sequences and images as input. Then, the multimodal graph is used as input to multiple fusion layers. By performing intramodal and cross-modal fusion, contextual information within the same modality and semantic information across modalities can be captured simultaneously. This achieves effective alignment of local features between text and images and deep semantic interaction, thereby more accurately achieving information complementarity and effectively solving the cross-modal semantic gap problem. Finally, by fusing the text node features output from the last fusion layer with the original text features output by BERT, the powerful capabilities of the pre-trained language model are preserved, while targeted multimodal contextual information is added. This makes the final feature representation used for decoding more comprehensive and accurate, thereby improving the accuracy of entity recognition. Therefore, this application enables full interaction between image and text information, eliminating visual noise interference in the image. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a named entity recognition method in the construction of a multimodal knowledge graph provided by an embodiment of the present invention; Figure 2 This is an overall architecture diagram of the method of this application provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a named entity recognition device in multimodal knowledge graph construction provided by an embodiment of the present invention; Figure 4 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0016] The specific implementation of the method in this application is described in detail below.

[0017] Please refer to Figure 1 This invention provides a method for identifying named entities in the construction of a multimodal knowledge graph, the method comprising: Step 100: Construct a multimodal graph based on text information and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. Step 102: Input the multimodal graph into multiple sequentially connected fusion layers. For each fusion layer, perform intramodal fusion on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information. Perform cross-modal fusion on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes. Use the cross-modal fused text nodes and visual nodes as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. Step 104: The text node features output by the last fusion layer are fused with the original text features output by BERT, and the fused features are then CRF decoded to obtain the final named entity recognition result.

[0018] In this embodiment, a multimodal graph is first constructed using text sequences and images as input. Then, this multimodal graph is used as input to multiple fusion layers. By performing intramodal and cross-modal fusion, contextual information within the same modality and semantic information across modalities can be captured simultaneously. This achieves effective alignment of local features between text and images and deep semantic interaction, thereby more accurately achieving information complementarity and effectively addressing the cross-modal semantic gap problem. Finally, by fusing the text node features output from the last fusion layer with the original text features output by BERT, the powerful capabilities of the pre-trained language model are preserved while supplementing targeted multimodal contextual information. This makes the final feature representation used for decoding more comprehensive and accurate, thereby improving the accuracy of entity recognition. Therefore, this application enables full interaction between image and text information, eliminating visual noise interference in the image.

[0019] The following description Figure 1 The execution method for each step is shown.

[0020] First, for step 100, a multimodal graph is constructed based on text information and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes.

[0021] In some implementations, the multimodal graph is constructed as follows: Step S1: For the input text information, the text sequence is split into multiple tokens based on the BERT model, and each token is modeled as a text node; the text sequence is syntactically analyzed based on the Stanford parser to identify noun phrases and use them as candidate nodes; for the abstract concepts in the text information, a predefined entity general vocabulary dictionary is constructed to generate additional nodes. Step S2: For the image information, detect the effective visual region corresponding to each candidate node based on the visual localization toolkit. If an effective visual region corresponding to a candidate node is detected, the effective visual region is used as the visual node; if the candidate node cannot be matched with an effective visual region, a zero-initialized vector placeholder is used as the visual node. Step S3: Connect nodes of the same modality and nodes across modalities to obtain the edges of the multimodal graph.

[0022] In steps S1 and S2, by modeling each token as a text node, more fine-grained semantic information can be preserved. Furthermore, in real-world multimodal data, some nouns or entities are abstract concepts, such as "love," "freedom," "risk," and "economy." These concepts do not have concrete, fixed visual representations in images. Visual localization toolkits typically can only detect concrete, physically existing objects (such as "people," "cars," and "dogs"), and cannot detect abstract concepts. Therefore, without processing, these abstract concepts will "disappear" on the visual side, causing the model to rely solely on textual information and losing the meaning of multimodal fusion. For example, for the abstract concept of "risk," there is no directly corresponding object in the image. However, if the image contains a scene of a "volcanic eruption," through a general vocabulary dictionary (e.g., associating "risk" with words like "danger"), the model can connect the textual "risk" with nodes representing dangerous scenes like "volcano" in the image, achieving semantic aided understanding. Therefore, by preserving the semantic information of these abstract concepts in the graph, "visual placeholders" can be provided for subsequent cross-modal fusion, enriching object discovery and letting the model know that these words should also participate in multimodal interactions (although they do not have real images, they can be associated with other nodes through dictionary semantics). In addition, using zero-initialized vector placeholders can avoid feature space collapse.

[0023] It should also be noted that when the image modality fails, the system switches to the plain text Transformer encoding mode. The selected visual object is processed by a ResNet-152 pre-trained model to extract depth features, which are then normalized and aligned with the text nodes to become visual nodes added to the multimodal graph.

[0024] In step S3, the edges of the multimodal graph carry the relationships between different semantic units, including two types: intramodal connections and cross-modal connections. Intramodal connections refer to the connection between two nodes of the same modality, while cross-modal connections refer to the connection between a text word node and its corresponding visual object node. In cross-modal connections, due to the introduction of general lexicons for various entity types, text nodes can also be connected to visual nodes detected by these general lexicons through edges. For example, if the text node is "a golden retriever" and the visual node is "dog," they cannot be connected. However, through general lexicon detection, it can be found that a golden retriever belongs to the canine category, and the text node corresponding to "golden retriever" can be connected to the visual node corresponding to "dog" through the general lexicon. This design can better express the semantic relationships between different modalities, providing a richer and more accurate way to convey information for multimodal graph modeling.

[0025] In some implementations, the edges in the multimodal graph are constructed as follows: For nodes of the same modality, connections can be made using full connectivity or based on syntax or spatial location. For cross-modal nodes, determine whether the visual object corresponding to the visual node matches a noun phrase in the candidate node, or whether the visual object corresponding to the visual node belongs to the same category as a common word in the text node, or whether the visual object corresponding to the visual node has a visual metaphor for an abstract concept in an additional node; if so, connect the visual node to the corresponding text node.

[0026] In addition, before performing intra-modal fusion on text nodes and visual nodes in the multimodal graph, the following steps are also included: Two multilayer perceptrons (MLPs) with ReLU activation functions are used to process text nodes and visual nodes respectively, so as to project features of different modalities into the same space.

[0027] like Figure 2 As shown, for step 102: For each fusion layer, intramodal fusion is performed on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes fused with contextual semantic information, including: For each text node, a self-attention mechanism is used to collect information between the text node and its neighboring nodes to construct a contextual semantic representation of each text node. The text nodes updated by the self-attention mechanism are then subjected to residual linking and normalization to obtain text nodes that incorporate contextual semantic information.

[0028] For each visual node, a self-attention mechanism is used to collect information between the visual node and its neighboring nodes to construct the contextual semantic representation of each visual node. The visual nodes updated by the self-attention mechanism are then subjected to residual linking and normalization to obtain visual nodes that are fused with contextual semantic information.

[0029] The following is the first Taking text nodes and visual nodes in the layer fusion layer as an example, the intra-modal fusion process is illustrated. The fusion process of text nodes is shown in the following equation: In the formula, For the first The output of the layer fusion layer contains fused text nodes with contextual semantic information; It is a multi-head attention function that depends on the query matrix. Key matrix Sum matrix The interaction between these components enables information fusion and transmission. This mechanism helps capture potential dependencies between different locations, thereby enhancing the model's generalization ability across various tasks. In self-attention, All are obtained from the same input representation (such as the token representation of the same sentence) through different linear transformations, therefore the same token is used to initiate queries ( ), is also considered as a candidate that can be matched ( ), while also providing content that can be aggregated ( ); For the first The vector representation of all text nodes in the layer fusion layer is a matrix, which serves as the input for the Query, Key, and Value multi-head attention module.

[0030] Using the formula above, self-attention scores the relevance of each text node (token) to other text nodes (using...). · Calculate attention weights), and then use these weights to... Perform a weighted summation.

[0031] Similarly, the fusion process of visual nodes is shown in the following equation: In the formula, For the first The output of the layer fusion layer contains fused visual nodes with contextual semantic information; For multi-head attention functions; For the first The layer fusion layer is a vector representation matrix of all visual nodes, which serves as the input for the Query, Key, and Value multi-head attention module.

[0032] After updating via the self-attention mechanism, residual linking and layer normalization operations are performed on each node. This involves adding the original input and attention output of the current fusion layer and then performing layer normalization to stabilize the training process and accelerate model convergence.

[0033] In some implementations, cross-modal fusion is performed on text nodes and visual nodes that have been fused with contextual semantic information to obtain cross-modal fused text nodes and visual nodes, including: For each text node, the attention weights between the text node and each connected visual node are calculated based on a cross-modal gating mechanism. Based on each attention weight, the amount of information absorbed by the text node from the corresponding visual node is determined, and the text node is updated.

[0034] In this step, the first step is calculated using the following formula. The first text node and its connected first... Attention weights between visual nodes : After calculating the attention weights, the text nodes updated via the cross-modal gating mechanism are calculated using the following formula: In the formula, For the first The text node is processed by the first Text nodes generated after the layer fusion layer cross-modal gating mechanism is updated; For the first The fusion of the layer outputs contains contextual semantic information. One text node; For the first The fusion of the layer outputs contains contextual semantic information. One visual node; It is an activation function that compresses the input to a range of 0-1; , The first The parameter matrices for text node processing and visual node processing in the layer fusion layer are used to achieve effective integration and utilization of cross-modal information, enhance the diversity and flexibility of fusion, and improve the overall performance and efficiency of the system. This represents the Hadamard product, used for element-wise multiplication to achieve feature selective enhancement. For text nodes The set of adjacent visual nodes.

[0035] For each visual node, the attention weights between the visual node and each connected visual node are calculated based on the cross-modal gating mechanism. Based on each attention weight, the amount of information absorbed by the visual node from the corresponding text node is determined, and the update of the visual node is completed.

[0036] In this step, the first step is calculated using the following formula. The first visual node and the first connected to it Attention weights between text nodes : After calculating the attention weights, the visual nodes updated via the cross-modal gating mechanism are calculated using the following formula: In the formula, For the first The visual node is processed by the first Visual nodes generated after updating the layer fusion layer cross-modal gating mechanism; , The first Parameter matrices for text node processing and visual node processing in the layer fusion layer; For visual nodes A set of adjacent text nodes.

[0037] For the updated text node, based on the cross-modal attention mechanism, with text as the query and image region features as the key and value, the attention weights for text-to-image alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the first attention matrix; based on the first attention weights and the original input text features of the fusion layer, the text-side gating vector is calculated; based on the first attention matrix and the text-side gating vector, the enhanced text node is calculated.

[0038] In this step, the first attention matrix is ​​calculated using the following formula: , , , , In the formula, This is the first attention matrix; , These are the text query matrix and the image query matrix, respectively. , These are the image key matrix and the text key matrix, respectively; Represents batch size; This represents the number of image regions. The maximum length of the text sequence; , These represent the query matrices constructed by applying two independent linear transformations to the text modal input features X. and bond length ; , These represent the query matrices constructed by applying two independent linear transformations to the visual modality input features V. and bond length ; , These are the parameter matrices for learning, each with a dimension of 1. This is used to map the original text features to different semantic subspaces, thereby representing the "query" and "matching" capabilities of the current text features respectively. This is achieved through computation... and The model can more effectively measure the correlation between text features and between text and other modal features in subsequent attention calculations, providing a foundation for cross-modal information alignment and interactive fusion.

[0039] In text-to-image attention, using the text query matrix With image key matrix The relevance score is obtained by taking the dot product of the transpose of the expression, and then scaled using the expression to stabilize the gradient and numerical range. Subsequently, the attention matrix is ​​obtained by normalizing the expression along the key dimension using Softmax. .

[0040] Calculate Then, the text-side gating vector is calculated using the following formula. : In the formula, For the Sigmoid function, Indicates feature splicing; , These are the learnable parameter matrix and the bias term, respectively.

[0041] In this formula, the gating vector is used to adaptively adjust the retention degree and contribution intensity of different modal interaction information in feature fusion, enabling the model to dynamically select more valuable cross-modal information based on the current text-image association.

[0042] Calculate the first attention matrix and text-side gating vector Then, the enhanced text node is calculated using the following formula: In the formula, For the first Text nodes enhanced by layer fusion.

[0043] Finally, Inputting the location feedforward network, we obtain the text nodes after cross-modal fusion of this fusion layer, as shown in the following formula: In the formula, For the first Text nodes after layer fusion and cross-modal fusion.

[0044] For the updated visual node, based on the cross-modal attention mechanism, using the image as the query and the text features as the key and value, the attention weights for image-to-text alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the second attention matrix; based on the second attention weights and the original input visual features of the fusion layer, the visual-side gating vector is calculated; based on the second attention matrix and the visual-side gating vector, the enhanced visual node is calculated.

[0045] Similarly, the second attention matrix is ​​calculated using the following formula: In the formula, This is the second attention matrix; in image-to-text attention, the relevance score is obtained by taking the dot product of the transpose of the image query matrix and the text key matrix, and then scaled to stabilize the gradient and numerical range. Subsequently, the attention matrix is ​​obtained by normalizing it on the key dimension using Softmax. This characterizes the attention distribution of each image region to the text token, thereby achieving alignment and information interaction between the two modalities.

[0046] Calculate Then, the visual-side gating vector is calculated using the following formula. : After calculating the second attention matrix and the visual-side gating vector, the enhanced visual node is calculated using the following formula: In the formula, For the first Visual nodes enhanced by layer fusion.

[0047] The enhanced text nodes and visual nodes are input into the position feedforward network respectively to obtain the cross-modal fused text nodes and visual nodes of this fusion layer.

[0048] Finally, Inputting the location feedforward network, we obtain the visual nodes after cross-modal fusion of this fusion layer, as shown in the following equation: In the formula, For the first Visual nodes after layer fusion and cross-modal fusion.

[0049] Regarding step 104: This step adds a feature fusion layer, which concatenates and linearly projects the enhanced features (512 dimensions) obtained after processing by the multi-layer graph fusion module with the original BERT output features (768 dimensions). This can simultaneously preserve the powerful general semantic information of the pre-trained model and the task-specific cross-modal contextual information learned by the model in this application, forming a richer feature representation.

[0050] Furthermore, CRF decoding: Since visual information is already integrated into the representation of text nodes, the decoder uses Conditional Random Fields (CRF) for sequence labeling. The CRF model can learn the transition patterns between labels (e.g., "B-PER" is likely to be followed by "I-PER"), and performs global optimization on the entire label sequence during decoding to select the optimal label combination and avoid generating illogical label sequences.

[0051] The following section details the specific implementation of CRF decoding: First, construct the following joint probability model: In the formula, In the input sequence Below, label sequence The conditional probability of occurrence; The text sequence obtained by fusing the text node features output by the last fusion layer with the original text features output by BERT; It represents a complete label sequence, that is, the result composed of the predicted labels corresponding to each position in the input sequence; Represents any sequence of candidate labels; It represents the set of all possible label sequences; , The first The, the One predicted label; , The first The, the There are 1 candidate label; exp is an exponential function; , All are emission fractions, generated by BERT feature meridians; , These are all transition matrices, used to learn the transition patterns between labels, such as the probability of B-PER being followed by I-PER; , All are latent functions; For the input sequence The length.

[0052] Conditional Random Fields (CRFs) are probabilistic graphical models primarily used to address the conditional dependencies between input and output label sequences in sequence labeling tasks. Unlike traditional methods, this application employs a joint CRF probabilistic model, considering not only the impact of current input features on the labels but also the transition relationships between labels. Through this joint probabilistic model, CRFs can select the optimal label combination for the entire sequence, thus avoiding conflicts that may arise from local decision-making. Therefore, this joint probabilistic model does not decode each label independently but rather models the entire sequence using CRFs.

[0053] Then, during the training of the joint probability model, parameters are learned through maximum conditional likelihood estimation to maximize the log-likelihood, thereby optimizing the model's performance and ensuring its best fit to the training data. The following maximum conditional likelihood estimation is used here: In the formula, express The log-likelihood.

[0054] By employing the maximum conditional likelihood logarithm described above, we can better understand the meaning of the parameters, thereby optimizing the model and improving training results.

[0055] Finally, in the prediction phase, the joint probability model predicts the output sequence with the highest score using the following formula: In the formula, This represents the label sequence ultimately predicted by the joint probability model. This represents the independent variable that maximizes the objective function. Here, it doesn't mean taking the maximum probability value itself, but rather choosing the label sequence that maximizes the probability. The result is the optimal label sequence, not the maximum probability value.

[0056] like Figure 3 , Figure 4 As shown, this embodiment of the invention provides a named entity recognition device for multimodal knowledge graph construction. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 3 The diagram shown is a hardware architecture diagram of a computing device housing a named entity recognition device in the construction of a multimodal knowledge graph according to an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 4 As shown, a device in a logical sense is formed by the CPU of the computing device in which it is located reading the corresponding computer program from the non-volatile memory into the memory for execution.

[0057] Please refer to Figure 4 This invention provides a named entity recognition device for multimodal knowledge graph construction, the device comprising: The construction unit 400 is used to construct a multimodal graph based on text information and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. The fusion unit 402 is used to connect multiple fusion layers sequentially to the multimodal graph input. For each fusion layer, it performs intramodal fusion on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information; it performs cross-modal fusion on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes; it uses the cross-modal fused text nodes and visual nodes as the input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. Decoding unit 404 is used to fuse the text node features output by the last fusion layer with the original text features output by BERT, and perform CRF decoding on the fused features to obtain the final named entity recognition result.

[0058] In some implementations, the multimodal graph is constructed as follows: For the input text information, the text sequence is split into multiple tokens based on the BERT model, and each token is modeled as a text node; the text sequence is syntactically analyzed based on the Stanford parser to identify noun phrases and use them as candidate nodes; for the abstract concepts in the text information, a predefined entity general vocabulary dictionary is constructed to generate additional nodes. For image information, the effective visual regions corresponding to each candidate node are detected based on the visual localization toolkit. If an effective visual region corresponding to a candidate node is detected, the effective visual region is used as the visual node; if a candidate node cannot be matched with an effective visual region, a zero-initialized vector placeholder is used as the visual node. Connecting nodes of the same modality and nodes across modalities yields the edges of the multimodal graph.

[0059] In some implementations, prior to intra-modal fusion of text nodes and visual nodes in the multimodal graph, the method further includes: Two multilayer perceptrons (MLPs) with ReLU activation functions are used to process text nodes and visual nodes respectively, so as to project features of different modalities into the same space.

[0060] In some implementations, when the fusion unit 402 performs the following operation: for each fusion layer, it performs intra-modal fusion on each text node and each visual node input to the fusion layer to obtain text nodes and visual nodes fused with contextual semantic information: For each text node, a self-attention mechanism is used to collect information between the text node and its neighboring nodes to construct the contextual semantic representation of each text node; residual linking and normalization are performed on the text nodes updated by the self-attention mechanism to obtain text nodes that are integrated with contextual semantic information. For each visual node, a self-attention mechanism is used to collect information between the visual node and its neighboring nodes to construct the contextual semantic representation of each visual node. The visual nodes updated by the self-attention mechanism are then subjected to residual linking and normalization to obtain visual nodes that are fused with contextual semantic information.

[0061] In some implementations, when the fusion unit 402 performs cross-modal fusion of text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes, it performs the following operations: For each text node, the attention weight between the text node and each visual node connected to it is calculated based on the cross-modal gating mechanism, and the amount of information absorbed by the text node from the corresponding visual node is determined based on each attention weight, thus completing the update of the text node; For each visual node, the attention weight between the visual node and each connected visual node is calculated based on the cross-modal gating mechanism, and the amount of information absorbed by the visual node from the corresponding text node is determined based on each attention weight to complete the update of the visual node. For the updated text node, based on the cross-modal attention mechanism, using text as the query and image region features as the key and value, the attention weights for text-to-image alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the first attention matrix; based on the first attention weights and the original input text features of the fusion layer, the text-side gating vector is calculated; based on the first attention matrix and the text-side gating vector, the enhanced text node is calculated; For the updated visual node, based on the cross-modal attention mechanism, using the image as the query and the text features as the key and value, the attention weights for image-to-text alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the second attention matrix; based on the second attention weights and the original input visual features of the fusion layer, the visual-side gating vector is calculated; based on the second attention matrix and the visual-side gating vector, the enhanced visual node is calculated. The enhanced text nodes and visual nodes are input into the position feedforward network respectively to obtain the cross-modal fused text nodes and visual nodes of this fusion layer.

[0062] In some implementations, when the decoding unit 404 performs CRF decoding on the fused features to obtain the final named entity recognition result, it performs the following operations: Construct a joint probability model for CRF decoding; Constructing the maximum conditional likelihood estimate of the joint probability model; The joint probability model is solved based on maximum conditional likelihood estimation, and the output sequence with the highest score is taken as the final named entity recognition result.

[0063] Embodiments of this application also provide a computer device, please refer to... Figure 4 The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement the named entity recognition method in the construction of multimodal knowledge graphs provided in the above-described method embodiments.

[0064] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the named entity recognition method in the construction of multimodal knowledge graphs provided in the above-described method embodiments.

[0065] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform the named entity recognition method in the construction of a multimodal knowledge graph as described in any of the above embodiments.

[0066] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0067] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0068] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for recognizing a named entity in a multi-modal knowledge graph construction, characterized in that, The method includes: A multimodal graph is constructed based on text and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. The multimodal graph is input into multiple sequentially connected fusion layers. For each fusion layer, intramodal fusion is performed on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information. Cross-modal fusion is then performed on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes. The cross-modal fused text nodes and visual nodes are used as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. The text node features output from the last fusion layer are fused with the original text features output from BERT, and the fused features are then CRF decoded to obtain the final named entity recognition result.

2. The method of claim 1, wherein, The multimodal graph is constructed in the following manner: For the input text information, the text sequence is split into multiple tokens based on the BERT model, and each token is modeled as a text node; the text sequence is syntactically analyzed based on the Stanford parser to identify noun phrases and use them as candidate nodes; for the abstract concepts in the text information, a predefined entity general vocabulary dictionary is constructed to generate additional nodes. For image information, the effective visual region corresponding to each candidate node is detected based on the visual localization toolkit. If an effective visual region corresponding to a candidate node is detected, the effective visual region is taken as the visual node. If a candidate node cannot match a valid visual region, a zero-initialized vector placeholder is used as the visual node. Connecting nodes of the same modality and nodes across modalities yields the edges of the multimodal graph.

3. The method according to claim 1, characterized in that, Before performing intra-modal fusion on text and visual nodes in the multimodal graph, the following steps are also included: Two multilayer perceptrons (MLPs) with ReLU activation functions are used to process text nodes and visual nodes respectively, so as to project features of different modalities into the same space.

4. The method according to claim 3, characterized in that, For each fusion layer, intramodal fusion is performed on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes fused with contextual semantic information, including: For each text node, a self-attention mechanism is used to collect information between the text node and its neighboring nodes to construct the contextual semantic representation of each text node; residual linking and normalization are performed on the text nodes updated by the self-attention mechanism to obtain text nodes that are integrated with contextual semantic information. For each visual node, a self-attention mechanism is used to collect information between the visual node and its neighboring nodes to construct the contextual semantic representation of each visual node. The visual nodes updated by the self-attention mechanism are then subjected to residual linking and normalization to obtain visual nodes that are fused with contextual semantic information.

5. The method according to claim 1, characterized in that, The process of performing cross-modal fusion on text nodes and visual nodes that have been integrated with contextual semantic information to obtain cross-modal fused text nodes and visual nodes includes: For each text node, the attention weight between the text node and each visual node connected to it is calculated based on the cross-modal gating mechanism, and the amount of information absorbed by the text node from the corresponding visual node is determined based on each attention weight, thus completing the update of the text node; For each visual node, the attention weight between the visual node and each connected visual node is calculated based on the cross-modal gating mechanism, and the amount of information absorbed by the visual node from the corresponding text node is determined based on each attention weight to complete the update of the visual node. For the updated text node, based on the cross-modal attention mechanism, using text as the query and image region features as the key and value, the attention weights for text-to-image alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the first attention matrix; based on the first attention weights and the original input text features of the fusion layer, the text-side gating vector is calculated; based on the first attention matrix and the text-side gating vector, the enhanced text node is calculated; For the updated visual node, based on the cross-modal attention mechanism, using the image as the query and the text features as the key and value, the attention weights for image-to-text alignment are calculated, and the calculated attention weights are normalized in the key dimension to obtain the second attention matrix; based on the second attention weights and the original input visual features of the fusion layer, the visual-side gating vector is calculated; based on the second attention matrix and the visual-side gating vector, the enhanced visual node is calculated. The enhanced text nodes and visual nodes are input into the position feedforward network respectively to obtain the cross-modal fused text nodes and visual nodes of this fusion layer.

6. The method according to claim 1, characterized in that, The step of performing CRF decoding on the fused features to obtain the final named entity recognition result includes: Construct a joint probability model for CRF decoding; Construct the maximum conditional likelihood estimate of the joint probability model; The joint probability model is solved based on the maximum conditional likelihood estimation, and the output sequence with the highest score is taken as the final named entity recognition result.

7. A named entity recognition device for multimodal knowledge graph construction, characterized in that, The device includes: A construction unit is used to construct a multimodal graph based on text information and image information. The multimodal graph includes text nodes, visual nodes, and edges connecting different nodes. The fusion unit is used to input the multimodal graph into multiple fusion layers that are sequentially connected. For each fusion layer, it performs intramodal fusion on each text node and each visual node input to that fusion layer to obtain text nodes and visual nodes with fused contextual semantic information; it performs cross-modal fusion on the text nodes and visual nodes with fused contextual semantic information to obtain cross-modal fused text nodes and visual nodes; it uses the cross-modal fused text nodes and visual nodes as input to the next fusion layer until the text node features and visual node features output by the last fusion layer are obtained. The decoding unit is used to fuse the text node features output by the last fusion layer with the original text features output by BERT, and perform CRF decoding on the fused features to obtain the final named entity recognition result.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images