A method for multi-modal named entity recognition of dynamic fusion of original graph and generated graph

Abstract

This invention discloses a multimodal named entity recognition method that dynamically fuses original and generated images, belonging to the interdisciplinary fields of natural language processing, computer vision, and artificial intelligence. The method includes: acquiring a text sequence, original image, and entity label sequence of multimodal samples; performing text preprocessing on the text sequence to generate corresponding generated images; constructing inputs for the text, original image, and generated images, respectively inputting them into a graph-text shared space coding model to generate text embeddings and image embeddings, constructing matching features to obtain corresponding matching scores; constructing a fusion prior, inputting the original image, generated image, and fusion prior into a dynamic fusion network to generate a pixel-level fusion mask; fusing the original image and generated image to obtain enhanced visual input, which, along with the text input, serves as input to a downstream multimodal named entity recognition model, outputting a named entity recognition label sequence to improve the accuracy of named entity recognition.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, computer vision and artificial intelligence, and specifically to a multimodal named entity recognition method, system, device and storage medium that dynamically fuses original and generated images. Background Technology

[0002] With the rapid development of internet platforms such as social media, online news, and short-text communities, a large amount of multimodal data, composed of both text and image information, has emerged. Named entity recognition (MNER), a key foundational task in information extraction, aims to identify names of people, places, organizations, and other entities from unstructured data, and has significant application value in scenarios such as event monitoring, intelligent question answering, knowledge graph construction, and business analysis. Compared to traditional pure text MNER tasks, MNER not only requires modeling the contextual semantics of text sequences but also needs to comprehensively utilize the image content corresponding to the text to improve the ability to identify ambiguous entities, implicit entities, and context-dependent entities.

[0003] Existing multimodal named entity recognition methods typically use raw text and raw images as joint inputs, extracting semantic features through text encoders and visual encoders respectively, and then completing entity classification using methods such as feature concatenation, attention interaction, or cross-modal fusion. However, in real-world applications, raw images often suffer from problems such as high content noise, unclear subject prominence, redundant background, and indirect text semantic mapping. Furthermore, social media texts often contain noise information such as links, repost tags, user identifiers, and hashtags, leading to significant semantic discrepancies between the image and text modalities. This affects the alignment of multimodal features and thus limits the performance of named entity recognition.

[0004] To enhance visual semantic supplementation capabilities, some solutions attempt to introduce generated images from text or other external visual information to alleviate the inadequacy of the original image's representation. However, while generated images can strengthen text-related semantics to some extent, they may still suffer from content offsets, detail distortions, or inconsistencies with the real scene. Directly fusing the original and generated images with fixed weights, simple replacements, or coarse-grained stitching makes it difficult to adaptively select based on the actual matching degree between the two types of images and the text in different samples. This easily introduces low-reliability visual information into the subsequent recognition process, leading to errors in entity boundary judgment or entity category confusion.

[0005] Therefore, there is an urgent need to propose a new multimodal named entity recognition method that can dynamically filter and fuse the original image and the generated image based on the degree of image-text matching at the input end, and use the enhanced visual input and the original text input together for subsequent multimodal named entity recognition, so as to improve the semantic consistency of image and text and the accuracy of entity recognition. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art by providing a multimodal named entity recognition method that dynamically fuses the original image and the generated image. This method solves the problems of existing multimodal named entity recognition methods in short text scenarios on social media, such as high noise in the original image, weak semantic alignment between the image and text, uncontrollable credibility of the generated image, and insufficient cross-modal local information interaction.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal named entity recognition method that dynamically fuses original and generated images, comprising:

[0008] Obtain the text sequence of the multimodal named entity recognition sample, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence;

[0009] The text sequence is preprocessed, and a text-to-image diffusion generation model is used to generate an image that corresponds to the semantics of the text sequence.

[0010] Construct text input, original image input, and generated image input, and input them into the image-text shared space coding model respectively to generate text embedding, original image embedding, and generated image embedding;

[0011] Based on the text embedding, original image embedding and generated image embedding, original image-text matching features and generated image-text matching features are constructed respectively. Global image-text matching features and local region-word matching features are also constructed respectively, resulting in global matching score, local matching score, region-level matching score and matching score space map.

[0012] A fusion prior is constructed based on the local matching scores of the original image and the local matching scores of the generated image. The original image, the generated image, the matching score spatial map and the fusion prior are input into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms.

[0013] The original image and the generated image are dynamically fused using the pixel-level fusion mask, and the residual enhancement term is superimposed to obtain an enhanced visual input;

[0014] The enhanced visual input and text input are used as inputs to the downstream multimodal named entity recognition model, which outputs a named entity recognition label sequence.

[0015] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images as described in this invention, generating the generated image corresponding to the semantics of the text sequence includes:

[0016] Perform a cleaning operation on the text sequence, and then truncate the length of the cleaned text sequence;

[0017] Construct prompt words based on the cleaned text sequence;

[0018] In the text-to-image diffusion generation model, latent variables are sampled from a Gaussian noise distribution, and cue words are encoded as conditional vectors.

[0019] The latent variables are updated through multi-step inverse diffusion iteration using a denoising model to obtain the final latent variables;

[0020] The decoder is used to restore the final latent variables to the generated image that corresponds to the semantics of the text.

[0021] This preferred solution cleans, truncates, and constructs prompt words to reduce the interference of noise such as links, forwarding tags, and topic symbols on the text-to-image process, so that the generated image maintains a high degree of consistency with the semantics of the text and provides supplementary visual information for subsequent text-to-image matching.

[0022] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images as described in this invention, the construction of text input, original image input, and generated image input includes:

[0023] The text sequence is segmented to obtain a word sequence, and the word sequence is truncated or padded to the preset maximum sequence length.

[0024] When the original word is segmented into multiple sub-words, the first sub-word inherits the original entity label, and the remaining sub-words are marked with extended labels. The entity labels are aligned with the word sequence, and a text validity mask is constructed to distinguish valid input from the padding position.

[0025] The original image and the generated image are preprocessed separately to obtain tensor forms of uniform size;

[0026] The original image object, the generated image object, the original text string, and the image identifier are retained to form a two-layer data organization form containing tensor input and original input, resulting in text input, original image input, and generated image input.

[0027] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images as described in this invention, the generated text embedding, original image embedding, and generated image embedding include:

[0028] The image-text shared space coding model includes a visual encoder and a text encoder;

[0029] The text input, original image input, and generated image input are encoded using a visual encoder and a text encoder respectively, resulting in the original image embedding and the generated image embedding, and the text embedding is output.

[0030] Normalization is performed on the text embedding, the original image embedding, and the generated image embedding respectively to obtain the normalized text embedding, the original image embedding, and the generated image embedding;

[0031] Low-rank adaptive fine-tuning is used to adapt the visual encoder to the task domain. The low-rank adaptive fine-tuning is applied to at least one projection of the visual attention layer to obtain the low-rank adaptive result.

[0032] A low-rank matrix increment is introduced into the linear transformation, and the low-rank adaptive result is adjusted by a scaling factor.

[0033] This preferred scheme uses a text-image shared space coding model to jointly encode text, original image and generated image, and performs normalization processing on the embedding vector to make different modal features in a unified representation space; at the same time, by introducing a low-rank adaptive increment in the visual attention layer, the visual encoder's adaptability to the current task domain is improved with fewer trainable parameters.

[0034] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images as described in this invention, the construction of original image-text matching features and generated image-text matching features includes:

[0035] Extract multi-dimensional semantic interaction information from the normalized original image embedding and text embedding, and obtain the original image-text matching features based on the combination of multi-dimensional semantic interaction information.

[0036] Extract multi-dimensional semantic interaction information from the normalized generated image embedding and text embedding, and obtain the generated image-text matching features based on the combination of multi-dimensional semantic interaction information;

[0037] The original image-text matching features and the generated image-text matching features are respectively input into a shared scoring head, and after non-linear mapping, the original image matching score and the generated image matching score are obtained.

[0038] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images as described in this invention, the construction of fusion priors includes:

[0039] Based on the difference between the local matching scores of the original image and the local matching scores of the generated image, the sample-level fusion prior is calculated using the activation function;

[0040] The sample-level fusion prior is extended to a fusion prior space map with the same size as the image, and the original image matching score space map and the generated image matching score space map are obtained from the region-level matching scores.

[0041] The original image, the generated image, the difference map between the original image and the generated image, the element-wise product map between the original image and the generated image, the matching fractional space map of the original image, the matching fractional space map of the generated image, and the fusion prior space map are concatenated into channels and used as input to the dynamic fusion network, which outputs the original fusion mask and the residual enhancement term, respectively.

[0042] The sample-level fusion prior is converted into a prior bias, which is then superimposed on the original fusion mask. Finally, through an activation function, the pixel-level fusion mask is obtained.

[0043] This preferred scheme constructs a sample-level fusion prior using the local matching scores of the original image and the generated image, and combines the spatial map generated by the region-level matching scores as the input of the dynamic fusion network, enabling the fusion network to simultaneously utilize the overall confidence of the samples and local spatial matching information to output a fusion mask and residual enhancement terms.

[0044] As a preferred embodiment of the multimodal named entity recognition method for dynamic fusion of original and generated images according to the present invention, the dynamic fusion of the original and generated images includes:

[0045] Using the pixel-level fusion mask as weight, the original image and the generated image are summed pixel by pixel to obtain the basic fusion image;

[0046] The residual enhancement term is scaled by a preset scaling factor and superimposed on the base fused image to obtain the final enhanced visual input for multimodal named entity recognition.

[0047] This invention provides a multimodal named entity recognition system that dynamically fuses the original image and the generated image.

[0048] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal named entity recognition system that dynamically fuses original and generated images, comprising:

[0049] The data acquisition module is used to acquire the text sequence of the multimodal named entity recognition sample, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence;

[0050] The preprocessing module is used to preprocess the text sequence and generate an image corresponding to the semantics of the text sequence using a text-to-image diffusion generation model.

[0051] The input building module is used to construct text input, original image input, and generated image input, and input them into the image-text shared space coding model to generate text embedding, original image embedding, and generated image embedding respectively;

[0052] The feature construction module is used to construct global image-text matching features and local region-word matching features based on the text embedding, original image embedding and generated image embedding, image region features and text word features, respectively, to obtain global matching score, local matching score, region-level matching score and matching score space map;

[0053] The fusion prior construction module is used to construct a fusion prior based on the local matching scores of the original image and the local matching scores of the generated image. The original image, the generated image, the matching score spatial map and the fusion prior are input into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms.

[0054] The dynamic fusion module is used to dynamically fuse the original image and the generated image using the pixel-level fusion mask, and superimpose the residual enhancement term to obtain enhanced visual input;

[0055] The named entity recognition label sequence output module is used to take the enhanced visual input and text input as input to the downstream multimodal named entity recognition model and output a named entity recognition label sequence.

[0056] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of a multimodal named entity recognition method for dynamic fusion of original and generated graphs.

[0057] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of a multimodal named entity recognition method for dynamic fusion of original and generated graphs.

[0058] The present invention has at least the following beneficial effects:

[0059] (1) Generate images by using a text-to-image diffusion generation model to supplement the entity-related semantics implicit in the text or not obvious in the original image into visual information. When the subject of the original image is not prominent, the background interference is strong, or the semantic alignment between the image and text is weak, the generated image can provide additional text-related visual cues.

[0060] (2) The semantic embeddings of text, original image and generated image are extracted by the image-text shared space coding model, and the matching degree between the original image and the generated image and the text is evaluated by the shared scoring head, so that the credibility of the two types of images is under the same discrimination scale, and the low credibility visual information interference caused by fixed fusion or simple splicing is reduced.

[0061] (3) By using a local region-word matching mechanism, the local semantic correspondence between image regions and text words is calculated to obtain region-level matching scores and local matching scores. This mechanism can distinguish between regions in the image that are semantically related to entities and regions that are unrelated to the background, thereby improving the accuracy of visual information selection.

[0062] (4) The original image matching score space map and the generated image matching score space map are generated by the regional matching score, and the sample-level fusion prior input dynamic fusion network is combined to make the fusion process consider both the overall image and text credibility and the local matching differences in different spatial locations, thereby realizing the adaptive fusion of the original image and the generated image.

[0063] (5) An enhanced visual input is generated by pixel-level fusion mask and residual enhancement term, and used together with text input for downstream multimodal named entity recognition. This enhanced visual input can provide more relevant visual support for entity boundary recognition and entity category discrimination. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the overall process of a multimodal named entity recognition method that dynamically fuses original and generated images, as provided in one embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0067] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a multimodal named entity recognition method that dynamically fuses original and generated images, including:

[0068] S100: Obtain the text sequence of the multimodal named entity recognition sample, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence;

[0069] In this embodiment of the invention, given a set of multimodal samples containing text and images, the goal is to perform sequence labeling on each word in the text sequence to identify named entity categories such as person names, place names, organization names, and miscellaneous entities.

[0070] Let the training dataset be represented as:

[0071]

[0072] in, The total number of samples, For the first A text sequence of samples, For the original image, This is the sequence of actual labels corresponding to each position in the text sequence. This represents the text length.

[0073] To enhance visual semantic supplementation capabilities, a generated image corresponding to the semantics of the text is generated, denoted as . This expands the single original image input into a ternary input representation as follows:

[0074]

[0075] in, For the extended ternary input, This is a text-based image generation.

[0076] The learning objective for constructing the mapping from the ternary input to the sequence of named entity labels is expressed as:

[0077]

[0078] in, The named entity label sequence predicted by the model. This is the mapping function that the model needs to learn.

[0079] It should be noted that in this embodiment, the single original image input is expanded into a ternary input consisting of text, the original image, and the generated image, and a mapping relationship is established from the ternary input to the named entity label sequence. This setting enables the generated image to participate in the subsequent image-text matching scoring and dynamic fusion process, thereby providing supplementary visual information for downstream named entity recognition.

[0080] S200: Perform text preprocessing on the text sequence, and use the text-to-image diffusion generation model to generate a generated image that corresponds to the semantics of the text sequence;

[0081] S300: Construct text input, original image input, and generated image input, and input them into the image-text shared space coding model respectively to generate text embedding, original image embedding, and generated image embedding;

[0082] S400: Based on text embedding, original image embedding, generated image embedding, image region features and text word features, global image-text matching features and local region-word matching features are constructed respectively to obtain global matching score, local matching score, region-level matching score and matching score space map;

[0083] S500: Construct a fusion prior based on the local matching scores of the original image and the local matching scores of the generated image. Input the original image, the generated image, the matching score spatial map and the fusion prior into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms.

[0084] S600: Dynamically fuse the original image and the generated image using a pixel-level fusion mask, and superimpose residual enhancement terms to obtain enhanced visual input;

[0085] S700: It takes enhanced visual input and text input as input to the downstream multimodal named entity recognition model and outputs a named entity recognition label sequence.

[0086] In this embodiment of the invention, the downstream multimodal named entity recognition model can be a recognition network that includes a text encoder, a visual encoder, and a sequence label decoder.

[0087] It should be noted that in this embodiment, the text, original image, and generated image are first uniformly represented by a graph-text shared space coding model. Then, a fusion prior is constructed based on the graph-text matching score, and the matching score space graph is input into the dynamic fusion network. Thus, the fusion process can retain reliable visual information in the original image while introducing supplementary semantics related to the text in the generated image, resulting in enhanced visual input for downstream named entity recognition.

[0088] In this embodiment of the invention, step S200 includes the following sub-steps A1-A5;

[0089] In A1: Clean the text sequence and truncate the length of the cleaned text sequence;

[0090] In A2: Construct prompt words based on the cleaned text sequence;

[0091] In A3: In the text-to-image diffusion generation model, latent variables are sampled from a Gaussian noise distribution, and cue words are encoded as conditional vectors;

[0092] In A4: The latent variables are updated through multi-step inverse diffusion iteration using a denoising model to obtain the final latent variables;

[0093] In A5: The decoder is used to restore the final latent variables to the generated image that corresponds to the semantics of the text.

[0094] In this embodiment of the invention, since social media texts often contain URL links, repost tags, user mention symbols, topic symbols, and redundant whitespace characters, directly inputting them into a text-to-image model can easily generate image content irrelevant to the task's semantics. Therefore, the text cleaning operation includes removing link information, repost tags, user mention symbols, topic symbols, and redundant whitespace characters, and truncating the length of the cleaned text.

[0095] The construction of prompt words includes directly using the cleaned text as prompt words, or adding descriptive statements to the cleaned text to limit the semantic consistency and scene style of the generated images.

[0096] For example, let the original text be... The cleaned text is ; Construct text-based image prompts based on cleaned text; The prompt constructor is denoted as The prompt word is then represented as ,in, Indicates prompt word pattern, These are prompt words;

[0097] The prompt word pattern is designed in the following three ways:

[0098] Original mode: ;

[0099] Simplified mode:

[0100] ,

[0101] Social media model:

[0102] It should be noted that by adding scene description statements before the cleaned text, we can provide more explicit generation constraints for the text-to-image model while preserving the core semantics of the original text, thereby making the generated image closer to the target task scene.

[0103] In this embodiment of the invention, prompt words are obtained. Then, the LatentDiffusion model, a text-to-image diffusion generation model, is used to generate the generated image. .

[0104] Specifically, latent variables are sampled from the Gaussian noise distribution:

[0105]

[0106] in, The initial latent variables are obtained by sampling from the Gaussian noise distribution. It has a mean of 0 and a covariance matrix that is the identity matrix. The Gaussian distribution.

[0107] The prompt words are encoded as conditional vectors:

[0108]

[0109] in, This indicates a text conditional encoder. This is the text conditional vector obtained after encoding the prompt words. In the inverse diffusion process... The denoising model is updated progressively at each time step as follows:

[0110]

[0111] in, The parameter is Denoising network, Indicates the guiding standard. The first one obtained after denoising and updating Each time step has a latent variable.

[0112] go through After stepwise inverse diffusion, the final latent variable is obtained. The image is then restored by the decoder as follows:

[0113]

[0114] in, For parameters decoder, As the final latent variable, To generate an image.

[0115] The image generation process can be summarized as follows:

[0116]

[0117] in, Use a random seed. For the number of reasoning steps, To guide the scale, and These represent the height and width of the generated image, respectively. This is a function for generating images.

[0118] The generation parameters can be set as follows:

[0119]

[0120] It should be noted that model parameters can be loaded with half-precision or brain-floating precision, and memory optimization methods such as attention slicing can be used to reduce resource consumption during the image generation process.

[0121] At the sample level, all samples in the training, validation, and test sets are deduplicated by image identifier, ensuring that each image identifier corresponds to only one image generation process. For samples that have already been generated and do not need to be overwritten, the saved generated images can be reused directly to improve overall processing efficiency.

[0122] In this embodiment of the invention, step S300 includes the following sub-steps B1-B4;

[0123] In B1: The text sequence is segmented to obtain a word sequence, and the word sequence is truncated or padded to the preset maximum sequence length;

[0124] In B2: When the original word is segmented into multiple sub-words, the first sub-word inherits the original entity label, and the remaining sub-words are marked with extended labels. The entity labels are aligned with the word sequence, and a text validity mask is constructed to distinguish valid input from the padding position.

[0125] In B3: The original image and the generated image are preprocessed separately to obtain tensor forms of uniform size;

[0126] In B4: the original image object, the generated image object, the original text string, and the image identifier are preserved, forming a two-layer data organization form that includes tensor input and original input, resulting in text input, original image input, and generated image input.

[0127] In this embodiment of the invention, for any multimodal named entity recognition sample, text input, original image input and generated image input are constructed respectively.

[0128] After the text sequence is processed by the word segmenter, the resulting word sequence is represented as:

[0129]

[0130] in, To determine the maximum sequence length after truncation or padding, and considering that text is typically short in multimodal named entity recognition scenarios, the preset maximum sequence length is 128. For the first The word sequence obtained after a text sequence has been processed by a word segmenter The first to the second in the word sequence Each word element.

[0131] If an original word is segmented into multiple sub-words, the first sub-word inherits the original entity tag, and the remaining sub-words are marked with extended tags to ensure that the tags are aligned with the word sequence.

[0132] The text validity mask is constructed as follows:

[0133]

[0134] in:

[0135]

[0136] in, For the first The text validity mask corresponding to each sample. The first to the second in the word sequence The value of each word position.

[0137] On the visual side, the original image and the generated image are preprocessed into tensors of uniform size as follows:

[0138]

[0139] in, This indicates image preprocessing operations, including resizing, tensorization, and normalization. The tensor of the preprocessed original image. This is the generated image tensor after preprocessing.

[0140] To accommodate subsequent image-text matching scoring and recognition modeling, the original image object, generated image object, original text string, and image identifier can be retained simultaneously, thus forming a two-layer data organization form that includes tensor input and original input.

[0141] It should be noted that by truncating or padding the word sequence, aligning the sub-word labels, and performing unified preprocessing on the original and generated images, it is possible to ensure that the text labels correspond one-to-one with the word positions, while ensuring that the two image inputs have consistent size and tensor form, which facilitates subsequent encoding, matching, and recognition processing.

[0142] In this embodiment of the invention, after completing steps B1-B4, step S300 further includes steps B5-B9.

[0143] In B5: the image-text shared space coding model includes a visual encoder and a text encoder;

[0144] In B6: The text input, the original image input, and the generated image input are encoded using a visual encoder and a text encoder, respectively, to obtain the global embedding of the original image and the global embedding of the generated image, and the global embedding of the text is output.

[0145] In B7: Normalization is performed on the text global embedding, the original image embedding, and the generated image embedding respectively to obtain the normalized text global embedding, the original image global embedding, and the generated image global embedding;

[0146] In B8: Low-rank adaptive fine-tuning is used to adapt the visual encoder to the task domain. The low-rank adaptive fine-tuning is applied to at least one projection of the visual attention layer to obtain the low-rank adaptive result.

[0147] In B9: a low-rank matrix increment is introduced into the linear transformation, and the low-rank adaptive result is adjusted by a scaling factor.

[0148] In this embodiment of the invention, a shared spatial coding model for images and text is used to jointly represent images and text.

[0149] Specifically, let the visual encoder be... The text encoder is Then we have:

[0150]

[0151] in, Embedded to the original image, To generate image embeddings, For text embedding,

[0152] All three are global embeddings, which are the final outputs of the encoder and usually represent the global information contained in the entire image or text.

[0153] For the original image, To generate an image, It is a text sequence.

[0154] L2 regularization is performed on the global embedding of text, the global embedding of the original image, and the global embedding of the generated image.

[0155] For any embedding vector, first calculate its L2 norm. Then, divide the embedding vector by the corresponding L2 norm to obtain a normalized vector of unit length, i.e.:

[0156]

[0157] in, For embedding vectors, It is the L2 norm. This is a normalized vector.

[0158] Therefore, the normalized embedding representations of the original image global embedding, the generated image global embedding, and the text global embedding are as follows:

[0159]

[0160] in, Global embedding of the normalized original image. For global embedding of the normalized generated image, This is a global embedding of the normalized text.

[0161] In this embodiment of the invention, in order to improve the adaptability of the image-text sharing space coding model in the current task domain and control the training cost, low-rank adaptive fine-tuning is introduced only for the visual coding part, while keeping the main parameters of the text coding part frozen.

[0162] Specifically, let the projected linear layer of the visual attention layer in the visual encoder be:

[0163]

[0164] The form after introducing low-rank adaptive fine-tuning is:

[0165]

[0166] in:

[0167]

[0168] in, The output features of the original linear transformation This is the original linear layer weight matrix. The input features are linear transformations. The output characteristics are those of the linear transform after low-rank adaptive fine-tuning. It is a low-rank adaptive increment matrix. It is a low-rank adaptive scaling factor. Let be the rank of the low-rank matrix. These are the two smaller matrices obtained from the low-rank decomposition. Given a real-valued space containing the input feature dimensions, This is a real number space for outputting feature dimensions.

[0169] Low-rank adaptive fine-tuning is applied to query projection, key projection, value projection, and output projection in the visual attention layer.

[0170] Let the corresponding projection matrices be respectively , , and The update forms are as follows:

[0171]

[0172] The low rank value can be set to The scaling factor can be set to .

[0173] in, To query the projection, For key projection, For value projection, For output projection, For query vector, For key vectors, For value vectors, This is the output vector.

[0174] It should be noted that, through the above methods, the visual encoder's ability to represent the task relevance of the original and generated images can be enhanced while maintaining the overall stability of the shared image space.

[0175] In this embodiment of the invention, step S400 includes the following sub-steps C1-C7;

[0176] In C1: Extract the multi-dimensional semantic interaction information of the normalized original image embedding and text embedding, and obtain the original image-text global matching feature based on the combination of the multi-dimensional semantic interaction information.

[0177] In C2: Extract the multi-dimensional semantic interaction information of the normalized generated image embedding and text embedding, and obtain the global matching feature of the generated image-text based on the combination of the multi-dimensional semantic interaction information.

[0178] In C3: The global features of original image-text matching and global features of generated image-text matching are input into the shared scoring head, and after non-linear mapping, the global matching scores of the original image and the global matching scores of the generated image are obtained.

[0179] In C4: After obtaining the global matching score, extract the region feature sequence of the original image, the region feature sequence of the generated image, and the word feature sequence of the text. Construct local region-word matching input based on the region feature sequence and the word feature sequence.

[0180] In C5: Image region features and text word features are mapped to the same latent space, and region-word cross-attention calculation is performed to obtain region-word fusion features.

[0181] In C6: Extract multi-dimensional semantic interaction information between image region features and region-word fusion features, construct region-level matching features, and obtain region-level matching scores through non-linear mapping.

[0182] In C7: weighted fusion of regional matching features is performed to obtain local matching scores, and the regional matching scores are rearranged and upsampled to the image size to obtain a matching score space map.

[0183] In this embodiment of the invention, after obtaining the normalized image global embedding and text global embedding, the original image-text global matching features and the generated image-text global matching features are constructed respectively. The multi-dimensional semantic interaction information includes the joint representation of the normalized original image global embedding and text global embedding or the normalized generated image global embedding and text global embedding, difference measurement, element-by-element interaction, and directional similarity.

[0184] Specifically, for any image embedding and text embedding The matching features between the two are defined as follows:

[0185]

[0186] in, This represents vector concatenation. Represents element-wise product. Image embedding, For text embedding, For cosine similarity,

[0187] The global matching features between the original image and the text are represented as follows:

[0188]

[0189] The global matching features between the generated image and text are represented as follows:

[0190]

[0191] in, For the first Original image embedding in each sample, For the first Generate image embeddings from each sample. For the first Text embedding in each sample.

[0192] Two types of matching features are input into a shared scoring head to obtain a global matching score on a uniform scale. Assuming the shared scoring head consists of a feature projection layer, a hidden layer, and an output layer, then:

[0193] ,

[0194] in, To match the logical output (logit). For global matching scores, The projected features To share the weights and biases of the feature projection layer in the scoring head, It is the ReLU activation function. , To share the weights and biases of the hidden layers in the score header, , To share the weights and biases of the output layer in the score header, For the hidden layer characteristics after ReLU activation, This is the global matching feature vector for the original image-text or the generated image-text.

[0195] The global matching scores of the original image and the generated image are obtained respectively:

[0196]

[0197] in, For activation function, For the first The original image-text global matching score for each sample. For the first Global image-text matching score for each sample. The matching logit output for the original image branch. To generate the matching logit for the output of the image branch.

[0198] It should be noted that by sharing the scoring head, the global matching evaluation of the original image branch and the generated image branch can be guaranteed to be under a unified discrimination standard. The global matching score mainly serves as an auxiliary supervision signal for image-text matching during training, used to constrain the image-text shared spatial coding model to learn the semantic consistency of image and text at the sample level. In this embodiment of the invention, after obtaining the global matching score from C1 to C3, in order to determine which visual content in the original image and the generated image is more relevant to the current text semantics or text words at the region level, a local region-word matching method is introduced. Specifically, while outputting the global image embedding and the global text embedding, the image-text shared spatial coding model also outputs the hidden layer state of the last layer at the token level of the visual encoder and the text encoder. After removing the global classification label, we get:

[0199]

[0200] in, Represents the original image. Indicates the generated image; Indicates the first The region feature sequence of the corresponding image in each sample Represents the feature sequence of words in the text; Indicates the first In the nth sample The first class of images Image region features, , The number of image regions. Indicates the first In the nth sample Text word features corresponding to each word element , This represents the number of lexical units. The above region feature sequences and global embeddings... , and Both originate from the same image-text shared space coding model. The difference lies in that global embedding is the final output of the image-text shared space coding model, while region feature sequences and word feature sequences are the outputs of the last hidden layer of the image-text shared space coding model.

[0201] In this embodiment of the invention, image region features are used as queries, and text word features are used as keys and values ​​to perform region-word cross-attention calculation:

[0202]

[0203] in, , , These represent linear mapping matrices for the query, key, and value, respectively. Represents the dimension of the latent space. This represents the attention weights between image regions and text words. This represents the region-word fusion feature sequence corresponding to each image region.

[0204] Furthermore, image region features With corresponding region-word fusion features Normalization is performed, and the matching feature function described earlier is used. First, region-level matching features are constructed for each image region. Then, a non-linear mapping is performed on these region-level matching features to obtain the region-level matching score between each image region and the text semantics, denoted as […]. The region-level matching score is primarily used to check whether each patch of the image matches the text.

[0205] In this embodiment of the invention, multiple region-level matching features are weighted and fused to obtain local matching features, and the local matching features are then nonlinearly mapped to obtain local matching scores. The local matching score is used to summarize the local matching results of all image patches into a single score for the entire image, which is then used as a prior for subsequent sample-level fusion.

[0206] Local matching scores are used to construct priors for subsequent sample-level fusion, while region-level matching scores are used to generate a matching score space graph. The matching score space graph is then used to backlay the matching scores of each image patch into the image space.

[0207] Simultaneously, the region-level matching scores are rearranged into a two-dimensional region grid and upsampled to the image size to obtain a matching score spatial map:

[0208]

[0209] in, This indicates that the original image matches the fractional space graph. This represents the generated graph matching score spatial graph. The advantage of this approach is that the region-level matching graph can characterize the local semantic consistency between different image regions and text words, thus providing finer-grained spatial guidance for the subsequent dynamic fusion network to generate pixel-level fusion masks.

[0210] In this embodiment of the invention, step S500 includes the following sub-steps D1-D4;

[0211] In D1: Based on the difference between the local matching scores of the original image and the local matching scores of the generated image, the sample-level fusion prior is calculated using the activation function;

[0212] In D2: The sample-level fusion prior is extended to a fusion prior space map with the same size as the image, and the original image matching score space map and the generated image matching score space map generated from the region-level matching scores in step S400 are obtained;

[0213] In D3: The original image, the generated image, the difference map between the original image and the generated image, the element-wise product map between the original image and the generated image, the matching fractional space map of the original image, the matching fractional space map of the generated image, and the fusion prior space map are concatenated into channels and used as input to the dynamic fusion network, which outputs the original fusion mask and the residual enhancement term respectively.

[0214] In D4: The sample-level fusion prior is converted into a prior bias, which is superimposed on the original fusion mask. Through the activation function, the final pixel-level fusion mask is obtained.

[0215] In this embodiment of the invention, the sample-level fusion prior is defined as:

[0217] ,

[0218] in, The original image-text local matching score for the i-th sample. The generated image-text local matching score for the i-th sample is... This serves as a priori for sample-level fusion.

[0219] when hour, A value closer to 1 indicates that the original image is relatively more credible;

[0220] when hour, A value closer to 0 indicates that the generated image is relatively more believable.

[0221] The sample-level fusion prior is extended to a fusion prior space graph, represented as:

[0222]

[0223] in, To integrate the prior space map, This is a 1×H×W all-1 tensor, where H and W are the height and width of the corresponding image. Original image matching score space graph. Matching the generated graph with the fractional space graph It is obtained by rearranging and upsampling the regional matching scores in step S400.

[0224] To generate a pixel-level fused mask, the original image, the generated image and their difference information, interaction information, and scoring prior are jointly input into a convolutional fusion network. The input is defined as:

[0225]

[0226] in, For multimodal fusion feature maps, The tensor of the preprocessed original image. The generated image tensor after preprocessing. This is a difference map between the original image and the generated image. This is an element-wise product graph of the original image and the generated image. Match the fractional space graph to the original image. To generate a fractional space graph.

[0227] Since the original image, generated image, difference map, and product map are all three-channel, while the two-way scoring map and prior map are all single-channel, the total number of input channels is 15.

[0228] The fusion network employs a convolutional coding-decoding structure, outputting the original mask and residual enhancement terms, denoted as follows:

[0229]

[0230]

[0231] in, For mask prediction branch, For residual prediction branch, For the first The original pixel-level fused mask output by the fusion network for each sample.

[0232] To further incorporate sample-level reliable priors into the pixel-level fusion process, the prior bias is defined as:

[0233]

[0234] Thus, the final fusion mask is obtained:

[0235]

[0236] in, To incorporate prior bias terms from sample-level priors, This is the final pixel-level fusion mask obtained after introducing a priori bias.

[0237] It should be noted that this invention constructs a sample-level fusion prior based on image-text matching scores, extends it into a spatial graph, and concatenates it with multi-scale image features before inputting it into a fusion network. A pixel-level mask and residual terms are generated through an encoding / decoding structure. Simultaneously, a prior bias is introduced to correct the mask output, making the fusion result more closely resemble a reliable image, improving image fusion accuracy and semantic consistency, and providing high-quality visual features for subsequent recognition.

[0238] In this embodiment of the invention, step S600 includes the following sub-steps E1-E2;

[0239] In E1: The original image and the generated image are weighted and summed pixel by pixel using the pixel-level fusion mask as the weight to obtain the basic fusion image;

[0240] In E2: The residual enhancement term is scaled by a preset scaling factor and superimposed on the base fused image to obtain the final enhanced visual input for multimodal named entity recognition.

[0241] In this embodiment of the invention, the basic fused image is constructed based on the fusion mask as follows:

[0242]

[0243] Combining the residual enhancement term, the final enhanced visual input is obtained:

[0244]

[0245] in, This is the residual scaling factor. , For the first The base fused image after weighting the samples using a fusion mask. For the final pixel-level fusion mask, The tensor of the preprocessed original image. The generated image tensor after preprocessing. This is an element-wise multiplication operation. To ultimately enhance visual input, This is a residual enhancement term. This enables pixel-level dynamic image fusion guided by sample-level reliable priors.

[0246] After obtaining the enhanced visual input, it is fed together with the text input into the downstream multimodal named entity recognition model to obtain the entity label prediction results at each position of the text.

[0247] It should be noted that pixel-level weighted fusion of the original image and the generated image is achieved by using a fusion mask, and visual details are optimized by combining residual enhancement. Under the guidance of trusted priors, the feature quality is improved, providing more accurate visual support for subsequent named entity recognition.

[0248] The above is an illustrative scheme of a multimodal named entity recognition method that dynamically fuses original and generated graphs according to this embodiment. It should be noted that the technical solution of this multimodal named entity recognition system that dynamically fuses original and generated graphs belongs to the same concept as the technical solution of the aforementioned multimodal named entity recognition method that dynamically fuses original and generated graphs. Details not described in detail in the technical solution of the multimodal named entity recognition system that dynamically fuses original and generated graphs in this embodiment can be found in the description of the technical solution of the aforementioned multimodal named entity recognition method that dynamically fuses original and generated graphs.

[0249] This embodiment presents a multimodal named entity recognition system that dynamically fuses original and generated images, including:

[0250] The data acquisition module is used to acquire the text sequence of the multimodal named entity recognition sample, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence;

[0251] The preprocessing module is used to preprocess the text sequence and generate an image corresponding to the semantics of the text sequence using a text-to-image diffusion generation model.

[0252] The input building module is used to construct text input, original image input, and generated image input, and input them into the image-text shared space coding model to generate text embedding, original image embedding, and generated image embedding respectively;

[0253] The feature construction module is used to construct global image-text matching features and local region-word matching features based on the text embedding, original image embedding and generated image embedding, image region features and text word features, respectively, to obtain global matching score, local matching score, region-level matching score and matching score space map;

[0254] The fusion prior construction module is used to construct a fusion prior based on the local matching scores of the original image and the local matching scores of the generated image. The original image, the generated image, the matching score spatial map and the fusion prior are input into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms.

[0255] The dynamic fusion module is used to dynamically fuse the original image and the generated image using the pixel-level fusion mask, and superimpose the residual enhancement term to obtain enhanced visual input;

[0256] The named entity recognition label sequence output module is used to take the enhanced visual input and text input as input to the downstream multimodal named entity recognition model and output a named entity recognition label sequence.

[0257] This embodiment also provides a computer device applicable to a multimodal named entity recognition method that dynamically fuses original and generated graphs, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the multimodal named entity recognition method that dynamically fuses original and generated graphs as proposed in the above embodiment.

[0258] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a multimodal named entity recognition method for dynamic fusion of original and generated graphs as proposed in the above embodiments.

[0259] The storage medium proposed in this embodiment and the multimodal named entity recognition method for dynamic fusion of original image and generated image proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0260] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0261] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal named entity recognition method that dynamically fuses original and generated images, characterized in that, Bag include: Obtain the text sequence of the multimodal named entity recognition sample, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence; The text sequence is preprocessed, and the preprocessed text sequence is used to generate an image that corresponds to the semantics of the text sequence using a text-to-image diffusion generation model. Construct text input, original image input, and generated image input, and input them into the image-text shared space coding model respectively to generate text embedding, original image embedding, and generated image embedding; Based on text embedding, original image embedding, generated image embedding, image region features and text word features, global image-text matching features and local region-word matching features are constructed respectively, resulting in global matching score, local matching score, region-level matching score and matching score space map; A fusion prior is constructed based on the local matching scores of the original image and the local matching scores of the generated image. The original image, the generated image, the matching score spatial map and the fusion prior are input into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms. The original image and the generated image are dynamically fused using a pixel-level fusion mask, and the residual enhancement term is superimposed to obtain an enhanced visual input; The enhanced visual input and text input are used as inputs to the downstream multimodal named entity recognition model, and the output is a sequence of named entity recognition labels.

2. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 1, characterized in that, Generating images that correspond to the semantics of the text sequence includes: Perform a cleaning operation on the text sequence, and then truncate the length of the cleaned text sequence; Construct prompt words based on the cleaned text sequence; In the text-to-image diffusion generation model, latent variables are sampled from a Gaussian noise distribution, and cue words are encoded as conditional vectors. The latent variables are updated through multi-step inverse diffusion iteration using a denoising model to obtain the final latent variables; The decoder is used to restore the final latent variables to the generated image that corresponds to the semantics of the text. Constructing text input, original image input, and generated image input includes: The text sequence is segmented to obtain a word sequence, and the word sequence is truncated or padded to the preset maximum sequence length. When the original word is segmented into multiple sub-words, the first sub-word inherits the original entity label, and the remaining sub-words are marked with extended labels. The entity labels are aligned with the word sequence, and a text validity mask is constructed to distinguish valid input from the padding position. The original image and the generated image are preprocessed separately to obtain tensor forms of uniform size; The original image object, the generated image object, the original text string, and the image identifier are retained to form a two-layer data organization form containing tensor input and original input, resulting in text input, original image input, and generated image input.

3. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 2, characterized in that, Generating text embeddings, original image embeddings, and generated image embeddings include: The image-text shared space coding model includes a visual encoder and a text encoder; The text input, original image input, and generated image input are encoded using a visual encoder and a text encoder respectively, resulting in the original image embedding and the generated image embedding, and the text embedding is output. Normalization is performed on the text embedding, the original image embedding, and the generated image embedding respectively to obtain the normalized text embedding, the original image embedding, and the generated image embedding; Low-rank adaptive fine-tuning is used to adapt the visual encoder to the task domain. The low-rank adaptive fine-tuning is applied to at least one projection of the visual attention layer to obtain the low-rank adaptive result. A low-rank matrix increment is introduced into the linear transformation, and the low-rank adaptive result is adjusted by a scaling factor; The original image-text matching features and the generated image-text matching features are constructed separately, including: Extract multi-dimensional semantic interaction information from the normalized original image embedding and text embedding, and obtain the original image-text matching features based on the combination of multi-dimensional semantic interaction information. Extract multi-dimensional semantic interaction information from the normalized generated image embedding and text embedding, and obtain the generated image-text matching features based on the combination of multi-dimensional semantic interaction information; The original image-text matching features and the generated image-text matching features are respectively input into a shared scoring head. After non-linear mapping, the original image matching score and the generated image matching score are obtained. Extract the original image region feature sequence, generate the image region feature sequence and the text word feature sequence, perform region-word cross attention calculation, and obtain the region-word fusion feature; Region-level matching features are constructed based on image region features and region-word fusion features, and region-level matching scores are obtained through nonlinear mapping; Multiple region-level matching features are weighted and fused to obtain the local matching scores of the original image and the local matching scores of the generated image. Based on the region-level matching scores, the original image matching score space map and the generated image matching score space map are generated.

4. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 3, characterized in that, Building fusion priors includes: Based on the difference between the local matching scores of the original image and the local matching scores of the generated image, the sample-level fusion prior is calculated using the activation function; The sample-level fusion prior is extended to a fusion prior space map with the same size as the image, and the original image matching score space map and the generated image matching score space map are obtained from the region-level matching scores. The original image, the generated image, the difference map between the original image and the generated image, the element-wise product map between the original image and the generated image, the matching fractional space map of the original image, the matching fractional space map of the generated image, and the fusion prior space map are concatenated into channels and used as input to the dynamic fusion network, which outputs the original fusion mask and the residual enhancement term, respectively. The sample-level fusion prior is converted into a prior bias, which is superimposed on the original fusion mask. The final pixel-level fusion mask is obtained by passing an activation function. Dynamic fusion of the original and generated images includes: Using the pixel-level fusion mask as weight, the original image and the generated image are summed pixel by pixel to obtain the basic fusion image; The residual enhancement term is scaled by a preset scaling factor and superimposed on the base fused image to obtain the final enhanced visual input for multimodal named entity recognition.

5. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 4, characterized in that, For any multimodal named entity recognition sample, construct text input, original image input, and generated image input respectively. After the text sequence is processed by the word segmenter, the resulting word sequence is represented as: in, To determine the maximum sequence length after truncation or padding, the text is set to short text, with a preset maximum sequence length of 128. For the first The word sequence obtained after a text sequence has been processed by a word segmenter The first to the second in the word sequence Each word element, If an original word is segmented into multiple sub-words, the first sub-word inherits the original entity tag, and the remaining sub-words are marked with extended tags to ensure that the tags are aligned with the word sequence. The text validity mask is constructed as follows: in: in, For the first The text validity mask corresponding to each sample. The first to the second in the word sequence The value of each word element position, This represents the validity value at the j-th position in the word sequence of the i-th sample. On the visual side, the original image and the generated image are preprocessed into tensors of uniform size as follows: in, This indicates image preprocessing operations, including resizing, tensorization, and normalization. The tensor of the preprocessed original image. This is the generated image tensor after preprocessing.

6. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 5, characterized in that, Let the projected linear layer of the visual attention layer in the visual encoder be: The form after introducing low-rank adaptive fine-tuning is: in: in, The output features of the original linear transformation This is the original linear layer weight matrix. The input features are linear transformations. The output characteristics are those of the linear transform after low-rank adaptive fine-tuning. It is a low-rank adaptive increment matrix. It is a low-rank adaptive scaling factor. Let be the rank of the low-rank matrix. These are the two smaller matrices obtained from the low-rank decomposition. Given a real-valued space containing the input feature dimensions, This is a real number space for outputting feature dimensions.

7. The multimodal named entity recognition method for dynamic fusion of original and generated images as described in claim 6, characterized in that, Define the sample-level fusion prior as: in, The original image-text local matching score for the i-th sample. The generated image-text local matching score for the i-th sample is... As a priori for sample-level fusion, The Sigmoid activation function maps the difference between the local matching scores of the original image and the generated image to the interval (0,1), thus obtaining weight values ​​that can be directly used as priors for sample-level fusion. when hour, A value closer to 1 indicates that the original image is relatively more credible; when hour, A value closer to 0 indicates that the generated image is relatively more believable. The sample-level fusion prior is extended to a fusion prior space graph, represented as: in, To integrate the prior space map, It is a full-1 tensor with dimensions 1×H×W, where H and W are the height and width of the corresponding image.

8. A multimodal named entity recognition system that dynamically fuses original and generated graphs, employing the multimodal named entity recognition method for dynamic fusion of original and generated graphs as described in any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to acquire the text sequence of multimodal named entity recognition samples, the original image corresponding to the text sequence, and the entity label sequence corresponding to each word in the text sequence; The preprocessing module is used to preprocess the text sequence and generate an image that corresponds to the semantics of the text sequence using a text-to-image diffusion generation model. The input building module is used to construct text input, original image input, and generated image input, and input them into the image-text shared space coding model to generate text embedding, original image embedding, and generated image embedding respectively; The feature construction module is used to construct global image-text matching features and local region-word matching features based on text embedding, original image embedding, generated image embedding, image region features and text word features, respectively, to obtain global matching score, local matching score, region-level matching score and matching score space map; The fusion prior construction module is used to construct a fusion prior based on the local matching scores of the original image and the local matching scores of the generated image. The original image, the generated image, the matching score spatial map and the fusion prior are input into the dynamic fusion network to generate a pixel-level fusion mask and residual enhancement terms. The dynamic fusion module is used to dynamically fuse the original image and the generated image using a pixel-level fusion mask, and to superimpose residual enhancement terms to obtain enhanced visual input; The Named Entity Recognition Tag Sequence Output Module is used to take the enhanced visual input and text input as input to the downstream multimodal named entity recognition model and output a named entity recognition tag sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal named entity recognition method for dynamic fusion of original and generated graphs as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal named entity recognition method that dynamically fuses the original image and the generated image as described in any one of claims 1 to 7.

Images