Multi-modal named entity recognition system and method based on multi-granularity query guidance

Abstract

The invention provides a multi-modal named entity recognition system and method based on multi-granularity query guidance, and relates to the technical field of multi-modal named entity recognition. The method comprises the following steps: firstly, preprocessing input multi-modal data to respectively obtain text representation and visual representation; constructing a multi-granularity query set; then performing multi-granularity query guide fusion on the text representation and the visual representation by utilizing the multi-granularity query set, and outputting a fused text representation, a fused visual region representation and a query set representation; and finally, based on the fused text representation, the fused visual area representation and the query set representation, performing multi-modal named entity recognition, and outputting a multi-modal named entity recognition result. According to the method and the device, the multi-modal named entity recognition accuracy and efficiency are effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of multimodal named entity recognition, and in particular to a multimodal named entity recognition system and method based on multi-granularity query guidance. Background Technology

[0002] With the rapid development of artificial intelligence, big data, and natural language processing technologies, social media has become an important platform for information dissemination and user interaction, widely used in areas such as public opinion monitoring, automated news reporting, intelligent question answering, product recommendation, and security monitoring. Unlike traditional text-based content, current social media content exhibits a multimodal characteristic, combining text and images.

[0003] However, relying solely on text for Named Entity Recognition (NER) has significant limitations. For example, when a post mentions a person but the accompanying image shows a different object, single-modal recognition methods are prone to misjudgment or omission, making it difficult to meet the practical needs of multimodal content analysis. Therefore, Grounded Multimodal Named Entity Recognition (GMNER) has become a research focus. This task not only requires extracting named entities and their types (such as person names, place names, organization names, brand names, etc.) from the text, but also further locating the corresponding regions of the entities in the image to achieve cross-modal alignment.

[0004] Existing technologies mainly fall into two categories: pipeline-based methods and end-to-end methods. Pipeline methods break down the GMNER task into sub-tasks such as multimodal entity recognition, entity binding, and entity localization. For example, a sequence labeling model is first used to identify text entities, and then a visual localization model is used to determine image regions. However, this method suffers from errors in propagation and low inference efficiency, making it difficult to adapt to complex social media scenarios. End-to-end methods unify text entity recognition and image entity localization into a single model. For example, query guidance is introduced through a machine reading comprehension framework to achieve simultaneous recognition and localization. However, this type of method typically relies on manually constructing query statements, which is time-consuming and labor-intensive, and it is difficult to cover all entity types. It is also prone to misjudgment in ambiguous semantic or multi-entity scenarios. Summary of the Invention

[0005] This invention proposes a multimodal named entity recognition system and method based on multi-granularity query guidance, aiming to solve the problems of low accuracy and efficiency in the existing multimodal named entity recognition technology.

[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A multimodal named entity recognition system based on multi-granularity query guidance includes: The feature extraction unit is used to preprocess the input multimodal data to obtain text representation and visual representation respectively; Multi-granularity queryset building unit, used to build multi-granularity querysets; The query-guided fusion unit is used to fuse the text representation and visual representation using the multi-granularity query set, and output the fused text representation, fused visual region representation and query set representation; The multimodal set recognition unit is used to perform multimodal named entity recognition by utilizing the fused text representation, fused visual region representation and query set representation, and output the multimodal named entity recognition result.

[0007] Preferably, the multimodal data includes text data and image data. Preprocessing the input multimodal data to obtain text representations and visual representations includes: S11. Input the text data into a preset text editor, and use the text editor to perform context modeling on the text data to obtain the text representation; S12. Input the image data into a pre-trained visual encoder, and use the visual encoder to extract the visual features of the image data to obtain the visual representation.

[0008] Preferably, the step of extracting visual features from the image data using the visual encoder to obtain the visual representation includes: S121. Perform region proposal detection on the image data to obtain multiple candidate regions; S122. The candidate region is encoded by the visual encoder to obtain the region-level visual features corresponding to the candidate region; and a special region embedding is added to the candidate region to represent entities that could not be located in the image. S123. Combine the region-level visual features with the special region embedding to obtain the visual representation.

[0009] Preferably, the construction of the multi-granularity query set includes: S21. Input the preset prompt template into the pre-trained type granularity query generator and output the type granularity query embedding; S22. Initialize the type-granularity query embedding as a learned embedding vector, and use the learned embedding vector to construct the entity-granularity query embedding; S23. The type-level query embedding and the entity-level query embedding are fused to obtain the multi-granularity query set.

[0010] Preferably, the step of using the multi-granularity query set to perform multi-granularity query-guided fusion of the text representation and visual representation includes: S31. Input the multi-granularity query set and text representation into a preset Transformer structure to extract text features related to the query; S32. Inject the features of the multi-granularity query set as prefix information into the key layer and value layer of the preset visual Transformer, perform prefix modulation on the visual representation, and output the visual region features guided by the query. S33. The similarity matrix between the multi-granularity query set and the visual region features and text features is used to perform weighted fusion, and the fused text representation, fused visual region representation and query set representation are output.

[0011] Preferably, the similarity matrix between the multi-granularity query set and visual region features and text features... The calculation formula is as follows:

[0012] in, For visual features or text features, For the first The visual region or the first The representation of a text tag, It is a multi-granularity query set.

[0013] Preferably, the multimodal named entity recognition based on the fused text representation, fused visual region representation, and query set representation includes: Using the fused text representation and the query set representation, a boundary probability matrix containing the start and end indices of the predicted candidate entities in the text is generated; Using the fused visual region representation and the query set representation, a matching probability matrix containing the region indexes of predicted candidate entities in the image is generated; The existence detection of type-granular queries is generated using the boundary probability matrix and the matching probability matrix. The existence detection and matching algorithm of the type granularity query is used to perform global optimal matching and output the multimodal named entity recognition result.

[0014] Preferably, the boundary probability matrix includes a probability matrix with a starting index and a probability matrix with an ending index; The probability matrix of the starting index The calculation formula is as follows:

[0015] in, As the first learning parameter, A joint representation for segment boundary localization; The probability matrix of the end index The calculation formula is as follows:

[0016] in, This is the second learning parameter.

[0017] Preferably, the matching probability matrix The calculation formula is as follows;

[0018]

[0019] in, For the third learned parameters, For the fourth learning parameter, For the parameters of the fifth learning, A joint representation of candidate region matching. To integrate visual region representations, This represents the query set.

[0020] This invention also provides a multimodal named entity recognition method based on multi-granularity query guidance, comprising: A multimodal named entity recognition method based on multi-granularity query guidance includes the following steps: S1. Preprocess the input multimodal data to obtain text representation and visual representation respectively; S2. Construct multi-granularity query sets; S3. Utilize the multi-granularity query set to perform multi-granularity query-guided fusion of the text representation and visual representation, and output the fused text representation, fused visual region representation, and query set representation; S4. Based on the fused text representation, fused visual region representation, and query set representation, perform multimodal named entity recognition and output the multimodal named entity recognition result.

[0021] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a multimodal named entity recognition system and method based on multi-granularity query guidance. First, the input multimodal data is preprocessed to obtain text and visual representations. Then, a multi-granularity query set is constructed to achieve fine-grained semantic modeling of entity information in text and images. The text and visual representations are then fused using the multi-granularity query set, outputting a fused text representation, a fused visual region representation, and a query set representation, filtering irrelevant information and enhancing the alignment between text and visual features. Finally, based on the fused multimodal features, accurate identification and localization of multimodal entities are achieved through parallel text fragment boundary localization, candidate region matching, and entity type classification. Attached Figure Description

[0022] Figure 1 This is a structural block diagram of the multimodal named entity recognition system based on multi-granularity query guidance in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a multimodal named entity recognition method based on multi-granularity query guidance in an embodiment of the present invention. Figure 3 This is a flowchart of the multimodal named entity recognition method based on multi-granularity query guidance in an embodiment of the present invention; Figure 4 This is a visualization of some cases in the Twitter-FMNERG benchmark test in an embodiment of the present invention. Detailed Implementation

[0023] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings; To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0024] Example 1 This embodiment provides a multimodal named entity recognition system based on multi-granularity query guidance, such as Figure 1 and Figure 2 As shown, the system includes: The feature extraction unit is used to preprocess the input multimodal data to obtain text representation and visual representation respectively; Multi-granularity queryset building unit, used to build multi-granularity querysets; The query-guided fusion unit is used to fuse the text representation and visual representation using the multi-granularity query set, and output the fused text representation, fused visual region representation and query set representation; The multimodal set recognition unit is used to perform multimodal named entity recognition by utilizing the fused text representation, fused visual region representation and query set representation, and output the multimodal named entity recognition result.

[0025] The goal of the system proposed in this embodiment is to predict a set of multimodal quadruplets, which can be represented as:

[0026] Represents the i-th quadruple. and It is the start and end boundary index of the span of the i-th target entity. Refers to its corresponding entity type. This indicates the grounding area. Note that if the target entity cannot be located in a locatable area in the image, then... Output None; otherwise, It consists of a four-dimensional spatial feature, including the coordinates of the upper left and lower right corners of the grounding region.

[0027] Specifically, the multimodal data includes text data and image data. Preprocessing the input multimodal data to obtain text representations and visual representations includes: S11. Input the text data into a preset text editor, and use the text editor to perform context modeling on the text data to obtain the text representation; S12. Input the image data into a pre-trained visual encoder, and use the visual encoder to extract the visual features of the image data to obtain the visual representation.

[0028] The step of extracting visual features from image data using the visual encoder to obtain the visual representation includes: S121. Perform region proposal detection on the image data to obtain multiple candidate regions; S122. The candidate region is encoded by the visual encoder to obtain the region-level visual features corresponding to the candidate region; and a special region embedding is added to the candidate region to represent entities that could not be located in the image. S123. Combine the region-level visual features with the special region embedding to obtain the visual representation.

[0029] Specifically, given input text data X, the text encoder BERT is used to encode it into a sequence of word embeddings. ,in h is the dimension of the hidden layer. and These are special markers for the start and end positions in sentence embedding. Given the input image data I, we utilize VinVL as a class-independent Region Proposal Network (RPN) to obtain all candidate regions. The proposals of the Top-K regions are retained as our candidate regions, and ViT-B / 32 from the pre-trained CLIP is used as the visual encoder. The initial visual representation of the candidate regions is as follows: To match entities that cannot be located in an image, we use blank images. The input is a visual encoder to construct a special visual token embedding. .at last, and Connection as the final visual representation , where k is the number of candidate regions.

[0030] Specifically, the construction of the multi-granularity query set includes: S21. Input the preset prompt template into the pre-trained type granularity query generator and output the type granularity query embedding; S22. Initialize the type-granularity query embedding as a learned embedding vector, and use the learned embedding vector to construct the entity-granularity query embedding; S23. The type-level query embedding and the entity-level query embedding are fused to obtain the multi-granularity query set.

[0031] Specifically, entity types can provide valuable information for entity fragment extraction and candidate region matching. We designed a prompt template: "Prompt = [TYPE] is an entity type about [MASK]". Here, [TYPE] refers to the entity type name, such as person, location, organization, etc. The prompt template is then fed into a pre-trained BERT model. The type-granular query embedding is computed as the output embedding for the [MASK] location:

[0032] in, It is a type-level query embedding. Indicates the number of entity types.

[0033] Specifically, entity-level queries are randomly initialized with learned input embeddings. During training, these embeddings automatically learn the semantics at the entity granularity level, as well as the corresponding relationships between candidate regions and entity fragments. To ensure that the query embeddings at the type granularity and entity granularity have the same dimension, we replicate the former d times. Subsequently, the multi-granularity query embeddings can be represented as:

[0034] in, Refers to multi-granularity query sets; ; ; For the number of queries, we use token-level addition. To integrate multi-granularity queries, Indicates repetition frequency.

[0035] Specifically, the step of using the multi-granularity query set to perform multi-granularity query-guided fusion of the text representation and visual representation includes: S31. Input the multi-granularity query set and text representation into a preset Transformer structure to extract text features related to the query; S32. Inject the features of the multi-granularity query set as prefix information into the key layer and value layer of the preset visual Transformer, perform prefix modulation on the visual representation, and output the visual region features guided by the query. S33. The similarity matrix between the multi-granularity query set and the visual region features and text features is used to perform weighted fusion, and the fused text representation, fused visual region representation and query set representation are output.

[0036] Specifically, previous multimodal fusion methods often involve the direct fusion of text and visual information due to semantic differences and noise between multimodal data. In contrast, we use queries as an intermediary to guide the integration of text representations and visual region representations separately. This module includes three interaction mechanisms.

[0037] Query-Text Cross-Attention Fusion: Queryset and text sequence These features are fed into a Transformer-based architecture. A cross-attention mechanism is used to fuse these unimodal features.

[0038] Query-Region Prefix Fusion: Building upon prefix tuning and its successful application in multimodal fusion, we propose query-region prefix fusion to reduce semantic bias between multimodal data. Specifically, we use query set features as prefix information and insert them into the candidate region features of each multi-head attention key and value layer in the visual Transformer. First, the query set features are used as prefix information and inserted into the candidate region features of each multi-head attention key and value layer in the visual Transformer. Layer candidate region features Projected onto the query, key, and value vectors:

[0039] For the Layered query set representation We project it onto the key vector and value vector Using the same embedding dimension as prefix information .

[0040]

[0041] in, The linear transformation is represented by the intermediate dimension of 2, indicating that we apply two isolated transformation parameters to the key and value layers. The prefix fusion attention mechanism can be formalized as follows:

[0042] Similarity-Aware Aggregator: To mitigate noise caused by misalignment between candidate regions and entity fragments, we propose a similarity-aware aggregator to learn fine-grained marker alignment between query tags and region / text tags. We represent the query set, text features, and visual features as follows: , and We calculate the tag-level similarity matrix as follows:

[0043] in, Represents visual or textual features. It refers to the first Representation of each visual region or text tag. Finally, fine-grained fusion, integrating the similarity-aware visual or text hidden state into the query hidden state, can be represented as:

[0044] in, It is a trade-off coefficient that balances visual and textual similarity. . and These are linear transformation parameters. It is a bias term.

[0045] Specifically, the multimodal named entity recognition based on the fused text representation, fused visual region representation, and query set representation includes: Using the fused text representation and the query set representation, a boundary probability matrix containing the start and end indices of the predicted candidate entities in the text is generated; Using the fused visual region representation and the query set representation, a matching probability matrix containing the region indexes of predicted candidate entities in the image is generated; The existence detection of type-granular queries is generated using the boundary probability matrix and the matching probability matrix. The existence detection and matching algorithm of the type granularity query is used to perform global optimal matching and output the multimodal named entity recognition result.

[0046] Specifically, we approach this from a global matching perspective, rather than relying on predefined order relationships between entities as in previous methods.

[0047] Fragment boundary localization. Given the text representation output in QFNet. and query set representation We first expand their dimensions to and Then integrate them into a joint representation:

[0048] in, It is a joint representation of fragment boundary localization. and These are the parameters to be learned. Therefore, the probability matrix for each text tag as the starting index can be calculated as follows:

[0049] in, , To learn the parameters. Similarly, we can simply use a new parameter. replace Get the end index The probability matrix.

[0050] Candidate region matching. Given an output visual region representation. and query set representation The candidate region matching task is very similar to boundary localization, also using a query set to predict the corresponding visual index of the candidate regions. This process can be formalized as follows:

[0051] in, This represents the probability matrix of a match. , and These are all parameters for learning.

[0052] Entity Classification. Since the query set is endowed with type-level information during construction, the entity classification task can be viewed as existence detection of type-granular queries. Furthermore, considering that boundary location and region matching information are useful for query existence detection, we concatenate them with the query. The existence detection of type queries can be formalized as:

[0053]

[0054] in, This indicates the probability of the existence of a type-granular query. and These are the parameters learned. Finally, the multimodal entity quadruple predicted by the i-th query can be represented as... The predicted set of quadruplets is .

[0055] Bipartial matching loss. We constructed a loss function based on optimal bipartial matching. However, due to the number of queries... Greater than the total number of ground truths We defined a label for those non-assignable predictions. Then the label Add to ground truth set Copy until its quantity reaches Our training objective is to find the optimal allocation that minimizes the global cost. .search Arrangement of elements It can be formalized as follows:

[0056] in, Represents the ground truth quadruple and The cost of matching pairs between predicted quadruplets under the index. Indicates when An indicator function that sets a value of 1 when true and 0 otherwise. Optimal allocation. The Hungarian algorithm can be used to solve this problem efficiently. For joint training of the model, the final loss function is defined as follows:

[0057] This embodiment proposes a multimodal named entity recognition system based on multi-granularity query guidance. The system first preprocesses the input multimodal data to obtain text and visual representations. Then, by constructing a multi-granularity query set, fine-grained semantic modeling of entity information in text and images is achieved. The system then uses this multi-granularity query set to perform multi-granularity query-guided fusion of the text and visual representations, outputting a fused text representation, a fused visual region representation, and a query set representation, filtering irrelevant information and enhancing the alignment effect between text and visual features. Finally, based on the fused multimodal features, accurate identification and localization of multimodal entities are achieved through parallel text fragment boundary localization, candidate region matching, and entity type classification. This invention effectively combines multi-granularity query guidance and multimodal fusion, successfully constructing a universal framework applicable to different social media content and image-text combinations: users only need to input text and image data to achieve automatic identification and localization of multimodal named entities, significantly improving entity recognition accuracy and cross-modal alignment capabilities, providing an efficient and reliable technical means for social media analysis, intelligent question answering, public opinion monitoring, and multimodal information extraction.

[0058] Example 2 This embodiment evaluates the performance of the multimodal named entity recognition system based on multi-granularity query guidance described in the above embodiments.

[0059] This embodiment uses the Twitter-GMNER and Twitter-FMNERG datasets to experimentally validate the model's performance. Twitter-GMNER contains only four entity types: Person (PER), Organization (OROG), Location (LOC), and Other (OTHER). Twitter-FMNERG expands the GMNER dataset to include 8 coarse-grained and 51 fine-grained entity types. Both are based on two publicly available MNER Twitter datasets: Twitter-2015 and Twitter-2017.

[0060] All experiments were performed on four NVIDIA RTX 3090 GPUs using PyTorch 1.9.1. For fair comparison, we used a pre-trained BERT model as the text encoder, a ViT-B / 32 model from a pre-trained CLIP as the visual encoder, and VinVL as the class-independent RPN. To improve model efficiency, we froze the visual encoder and RPN and specified the number of layers in the QFNet to 3. We initialized the learnable query part with a normal distribution N(0, 0.02). For training, we set the batch size to 16 and the learning rate to 2 × 10e. 5. The training epochs were set to 50. Our model used an Adam optimizer with a linear warm-up ratio of 0.05. To allow the learnable multi-granularity query to learn the initial query semantics, we first froze the parameters of the pre-trained model and trained it separately for only 5 epochs on the learnable query. Statistical details and specific examples of the results obtained by this method are shown in Table 1 and Table 2, respectively. Figure 3 As shown.

[0061] Table 1. Statistical data of the Twitter-GMNER dataset.

[0062] These experimental results demonstrate that this method can effectively establish intra-entity relationships between entity spans and entity regions, and distinguish between ambiguous entities, such as Adidas (the company) and Adidas Ace (the brand's product). Furthermore, by appropriately modeling the relationships between entities, current predictions become insensitive to previous errors, thus improving the accuracy of judgments in social media analysis systems.

[0063] Example 3 This invention also provides a multimodal named entity recognition method based on multi-granularity query guidance, such as... Figure 4 As shown, the method includes the following steps: S1. Preprocess the input multimodal data to obtain text representation and visual representation respectively; S2. Construct multi-granularity query sets; S3. Utilize the multi-granularity query set to perform multi-granularity query-guided fusion of the text representation and visual representation, and output the fused text representation, fused visual region representation, and query set representation; S4. Based on the fused text representation, fused visual region representation, and query set representation, perform multimodal named entity recognition and output the multimodal named entity recognition result.

[0064] This embodiment proposes a novel end-to-end multimodal named entity recognition method. It achieves joint extraction of text and visual entities by modeling reasonable relationships within and between multimodal entities. Compared to previous methods, this model offers more robust performance while maintaining efficiency. Compared to existing technologies, this invention proposes a framework based on multimodal ensemble prediction, innovatively modeling GMNER as an end-to-end multimodal ensemble prediction task. Unlike methods that rely on manually constructed query statements, this embodiment utilizes a set of learnable query sets to adaptively learn fine-grained features of text fragments and visual regions within entities through joint training. Unlike sequence generation methods, this embodiment employs optimal bipartite graph matching decoding to ensure that the current prediction result is globally optimal, avoiding reliance on autoregressive sequence decoding and thus preventing the exposure of biases.

[0065] This embodiment also designs a query-guided fusion mechanism (QFNet). Compared with the traditional cross-modal attention mechanism, this method uses queries as an intermediary to avoid direct fusion of image and text representations, alleviates the noise impact caused by modal heterogeneity and image-text misalignment, and ensures that the model achieves higher accuracy and robustness in complex multimodal tasks.

[0066] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A multimodal named entity recognition system based on multi-granularity query guidance, characterized in that, include: The feature extraction unit is used to preprocess the input multimodal data to obtain text representation and visual representation respectively; Multi-granularity queryset building unit, used to build multi-granularity querysets; The query-guided fusion unit is used to fuse the text representation and visual representation using the multi-granularity query set, and output the fused text representation, fused visual region representation and query set representation; The multimodal set recognition unit is used to perform multimodal named entity recognition by utilizing the fused text representation, fused visual region representation and query set representation, and output the multimodal named entity recognition result.

2. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 1, characterized in that, The multimodal data includes text data and image data, and the feature extraction unit includes a text editor and a visual encoder; The input multimodal data is preprocessed to obtain text and visual representations, including: S11. Use the text editor to perform context modeling on the text data to obtain the text representation; S12. The visual features of the image data are extracted using the visual encoder to obtain the visual representation.

3. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 2, characterized in that, The step of extracting visual features from image data using the visual encoder to obtain the visual representation includes: S121. Perform region proposal detection on the image data to obtain multiple candidate regions; S122. The candidate region is encoded by the visual encoder to obtain the region-level visual features corresponding to the candidate region; and a special region embedding is added to the candidate region to represent entities that could not be located in the image. S123. Combine the region-level visual features with the special region embedding to obtain the visual representation.

4. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 1, characterized in that, The multi-granularity query set construction unit includes a type-granularity query generator, a query embedder, and a query fusion unit; The construction of multi-granularity query sets includes: S21. Use the type granularity query generator to parse the preset prompt template and output the type granularity query embedding; S22. Initialize the type-granularity query embedding into a learned embedding vector using the query embedding device, and construct the learned embedding vector into an entity-granularity query embedding. S23. Use a query fusion tool to fuse the type-level query embedding with the entity-level query embedding to obtain the multi-granularity query set.

5. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 1, characterized in that, The query guidance fusion unit is equipped with a cross-attention mechanism, which is used to fuse the multi-granularity query set, the text representation, and the visual representation, respectively, including: S31. Input the multi-granularity query set and text representation into a preset Transformer structure to extract text features related to the query; S32. Inject the features of the multi-granularity query set as prefix information into the key layer and value layer of the preset visual Transformer, perform prefix modulation on the visual representation, and output the visual region features guided by the query. S33. The similarity matrix between the multi-granularity query set and the visual region features and text features is used to perform weighted fusion, and the fused text representation, fused visual region representation and query set representation are output.

6. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 5, characterized in that, The similarity matrix between the multi-granularity query set and visual region features and text features The calculation formula is as follows: in, For visual features or text features, For the first The visual region or the first The representation of a text tag, It is a multi-granularity query set.

7. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 1, characterized in that, The multimodal named entity recognition based on the fused text representation, fused visual region representation, and query set representation includes: Using the fused text representation and the query set representation, a boundary probability matrix containing the start and end indices of the predicted candidate entities in the text is generated; Using the fused visual region representation and the query set representation, a matching probability matrix containing the region indexes of predicted candidate entities in the image is generated; The existence detection of type-granular queries is generated using the boundary probability matrix and the matching probability matrix. The existence detection and matching algorithm of the type granularity query is used to perform global optimal matching and output the multimodal named entity recognition result.

8. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 7, characterized in that, The boundary probability matrix includes a probability matrix for the starting index and a probability matrix for the ending index. The probability matrix of the starting index The calculation formula is as follows: in, As the first learning parameter, A joint representation for segment boundary localization; The probability matrix of the end index The calculation formula is as follows: in, This is the second learning parameter.

9. The multimodal named entity recognition system based on multi-granularity query guidance according to claim 7, characterized in that, The matching probability matrix The calculation formula is as follows; in, As the third learning parameter, This is the fourth learning parameter. For the parameters of the fifth learning, A joint representation of candidate region matching. To integrate visual region representations, This represents the query set.

10. A multimodal named entity recognition method based on multi-granularity query guidance, characterized in that, Includes the following steps: S1. Preprocess the input multimodal data to obtain text representation and visual representation respectively; S2. Construct multi-granularity query sets; S3. Utilize the multi-granularity query set to perform multi-granularity query-guided fusion of the text representation and visual representation, and output the fused text representation, fused visual region representation, and query set representation; S4. Based on the fused text representation, fused visual region representation, and query set representation, perform multimodal named entity recognition and output the multimodal named entity recognition result.

Images