Multi-modal named entity recognition model construction method based on visual prefix and comparative learning

Abstract

The invention belongs to the related technical field of knowledge maps, and particularly relates to a multi-modal named entity recognition model construction method based on visual prefix and comparative learning, which is used for constructing a named entity recognition network and comprises a visual prompt extraction module and a comparison learning module, the multi-level feature extraction module is used for extracting object-level multi-level features including low-level texture features and high-level semantic information in an image and using the multi-level features as visual prompts; the dynamic gating mechanism unit is used for calculating a weight for each level of features in the visual prompt to perform weighted summation to serve as a visual prefix; the semantic feature extraction module is used for performing semantic feature extraction of fused visual features on the text data based on the visual prefixes to obtain fused features; in the training process, the image feature and the fusion feature of each difficult sample in the sample set are projected to a contrast learning space, and the average contrast loss of all the difficult samples and the named entity recognition loss based on the fusion feature are added to back propagation to adjust model parameters. According to the invention, the accuracy of multi-modal knowledge extraction can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to knowledge graphs, and more specifically, relates to a method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning. Background Art

[0002] In manufacturing systems where the manufacturing supply chain and service chain converge, business data processing often features massive, multimodal data. Multimodal data comes from numerous sources, is highly heterogeneous, and has varying structures and representations. Downstream business graph construction and domain knowledge modeling cannot directly utilize this data. Therefore, knowledge extraction is necessary to extract knowledge from this massive multimodal data, form a unified representation, and construct a knowledge graph to support downstream tasks. Knowledge extraction is a massive task, and named entity recognition (NER) is a crucial component of knowledge extraction. Improving the accuracy of named entity recognition is crucial to the entire knowledge extraction process.

[0003] Traditional named entity recognition methods are primarily designed for pure text data, relying on natural language processing techniques to extract knowledge by identifying named entities in text and classifying them into predefined categories. However, these methods struggle to effectively handle the complexity of multimodal data. Consequently, multimodal named entity recognition (MNER) methods have emerged. MNER incorporates image resources as auxiliary information during the named entity extraction process, leveraging the complementarity between images and text to improve the accuracy of named entity recognition.

[0004] While some existing multimodal knowledge extraction methods can fuse data from different modalities, they still have some shortcomings in practical applications. First, the distribution of image and text features differs significantly. Images are represented as pixel values, characterized by high dimensionality and complexity, while text is represented as token sequences, characterized by discreteness and semantics. This distribution difference means that directly fusing image and text features can lead to information inconsistency and even generate noise. For example, in an image, color, background objects, and other features may be irrelevant to the semantic information in the text, thus affecting the accuracy of the extraction results. Therefore, how to obtain relevant object-level visual information to enhance the NER effect and reduce the interference of irrelevant images is an urgent problem. Furthermore, implicit alignment of multimodal data relies on the model's ability to select parts of the image that are relevant to the current text entity. If there are too many visual objects, the model may distract attention in the cross-modal attention mechanism, reducing its focus on key entities. For example, if the text describes a person, but the image contains multiple unrelated objects (such as a chair or table in the background), the model may mistakenly associate the features of these objects with the person entity. This inconsistency in the number of entities makes it difficult for the model to accurately align entities in the text and image, further increasing the complexity of the extraction task. These factors together result in the fact that the accuracy and robustness of the extraction results of existing methods still need to be improved. Summary of the Invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning, which aims to improve the accuracy of multimodal knowledge extraction.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning is provided, comprising: Constructing a named entity recognition network, including: a visual cue extraction module, used to extract a first multi-level feature containing low-level texture features and high-level semantic information at the object level in the text image pair, and unify the sizes of different level features in the first multi-level feature by scaling, splicing the features of each level along the channel dimension to obtain a second multi-level feature used as a visual cue; a dynamic gating mechanism unit, used to calculate a weight for each level feature in the visual cue to reflect the importance of the level feature in the current task, and perform a weighted summation of the features of each level in the visual cue based on the weight, and the summation result is inserted into the Transformer of the semantic feature extraction module as a visual prefix; the semantic feature extraction module is used to extract semantic features that fuse visual features on the text data in the text image pair through the Transformer to obtain a fused feature; The named entity recognition network is trained. During the training process, the image features and the fusion features of each difficult sample in a predetermined training sample set are projected into a contrastive learning space, and negative samples and positive samples of each difficult sample are constructed, where the positive sample is the difficult sample and there are multiple negative samples, which are samples composed of the image features in the difficult sample and the fusion features in other difficult samples; the similarity between the image features and the fusion features in all negative samples and positive samples is calculated, and the contrast loss of each difficult sample is calculated based on the similarity corresponding to the positive sample in each difficult sample and the similarity corresponding to all negative samples; the average contrast loss of all difficult samples is added to the named entity recognition loss based on the fusion feature, and the model parameters are adjusted by backpropagation to complete the construction.

[0007] Furthermore, the visual cue extraction module specifically extracts image features through the ResNet-50 model, and performs average pooling operations on the feature maps of different shapes output by each sublayer of the ResNet-50 model to unify the width and height of the feature maps of different shapes, where the first dimension in the feature map shape represents the number of channels and the remaining dimensions are the width and height of the feature map; all feature maps are spliced ​​along the channel dimension to obtain a second multi-level feature.

[0008] Furthermore, the text semantic feature extraction module specifically extracts hierarchical features of text data through the BERT model, and uses the first token in the output of the last hidden layer of the BERT model, namely the [CLS] token, as the semantic feature of each text data.

[0009] Furthermore, the dynamic gating mechanism unit determines the weights of features at each level through a neural network including an activation function and a normalization function.

[0010] Further, the visual cue of each difficult sample is used as the image feature; and / or, a two-layer MLP is used to perform the projection operation.

[0011] Furthermore, the Yolo11 model is used to perform target detection on each training sample in the training sample set, and the training samples whose target number reaches the preset requirement are regarded as difficult samples.

[0012] According to another aspect of the present invention, a multimodal named entity recognition method is provided, in which a text image pair to be recognized is input into a named entity recognition model constructed by the multimodal named entity recognition model construction method as described above to implement multimodal named entity recognition.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the above-mentioned method.

[0015] According to another aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the method described above when executed by a processor.

[0016] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art: This invention provides a method for constructing a multimodal named entity recognition model based on visual prefixes and contrastive learning. Given a text segment and its associated visual description image, a language model is first used to extract semantic features of the text data. Simultaneously, the method uses a visual network to extract hierarchical features of the image and unifies the sizes of features at different levels. Subsequently, a dynamic gating mechanism is used to input the image features as visual prefixes into each Transformer layer of the language model. A multi-head attention mechanism dynamically adjusts the contribution of features from different modalities to achieve preliminary fusion of multimodal features. Considering that inconsistent numbers of visual objects and entities can cause deviations in text-image alignment, which in turn affects the accuracy of entity recognition, this invention proposes a debiased contrastive learning method during the training of the multimodal named entity recognition model. Multi-entity samples are selected from random samples as difficult samples for contrastive learning. This method enhances the model's learning effect on multi-entity samples, thereby alleviating the problem of bias caused by the number of entities and improving the accuracy and robustness of knowledge extraction. This method is effectively used to extract knowledge and construct knowledge graphs from massive multimodal data in manufacturing supply chains and service chains. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning provided by an embodiment of the present invention; Figure 2 A schematic diagram of a multimodal feature fusion architecture provided by an embodiment of the present invention; Figure 3 A schematic diagram of a comparative learning mechanism provided by an embodiment of the present invention; Figure 4 A schematic diagram showing a comparison between an embodiment of the present invention and different baseline methods on a public dataset; Figure 5 An example diagram of named entity recognition in a supply chain scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0019] Example 1 A method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning, comprising: Constructing a named entity recognition network, including: a visual cue extraction module, which is used to extract a first multi-level feature containing low-level texture features and high-level semantic information at the object level in the text image pair, and unify the sizes of different level features in the first multi-level feature by scaling, and splicing the features of each level along the channel dimension to obtain a second multi-level feature, which is used as a visual cue; a dynamic gating mechanism unit, which is used to calculate a weight for each level feature in the visual cue to reflect the importance of the level feature in the current task, and perform a weighted summation of the features of each level in the visual cue based on the weight, and the summation result is inserted into the Transformer of the semantic feature extraction module as a visual prefix; the above-mentioned semantic feature extraction module is used to extract semantic features that fuse visual features on the text data in the text image pair through the above-mentioned Transformer to obtain a fused feature; A named entity recognition network is trained. During the training process, the image features and the fusion features of each difficult sample in a predetermined training sample set are projected into a contrastive learning space, and negative and positive samples of each difficult sample are constructed. The positive sample is the difficult sample, and there are multiple negative samples, which are samples composed of the image features in the difficult sample and the fusion features in other difficult samples. The similarity between the image features and the fusion features in all negative and positive samples is calculated, and the contrast loss of each difficult sample is calculated based on the similarity corresponding to the positive sample in each difficult sample and the similarity corresponding to all negative samples. The average contrast loss of all difficult samples is added to the named entity recognition loss based on the fusion feature, and the model parameters are adjusted by backpropagation to complete the construction.

[0020] The method of this embodiment aims to solve the problem of modal interaction in existing multimodal knowledge extraction technology, especially the problem that the model has difficulty paying sufficient attention to key entities and lacks sufficient object-level visual information when the entity density in the image is high and dispersed. A method for constructing a multimodal named entity recognition model based on multimodal data fusion and contrastive learning is proposed. This method improves the accuracy and robustness of knowledge extraction by constructing a unified knowledge expression form, integrating multimodal data features such as text and images, and introducing a contrastive learning mechanism. It is used to extract knowledge from massive multimodal data in the manufacturing supply chain and service chain and construct a knowledge graph.

[0021] Specifically, given a piece of text and its associated visual description image, the method first uses a language model to extract the semantic features of the text data. At the same time, the method uses a visual network to extract the hierarchical features of the image and unifies the size of the features at different levels. Subsequently, the method of this embodiment uses a dynamic gating mechanism to input the image features as visual prefixes into each layer of the Transformer of the language model, and uses a multi-head attention mechanism to dynamically adjust the contribution of different modal features to complete the preliminary fusion of multimodal features. In addition, when the number of visual objects and entities is inconsistent, the text image alignment will be biased, which will affect the accuracy of entity recognition. To solve the above problem, in the training process of the multimodal named entity recognition model, a debiased contrastive learning method is proposed. Multi-entity samples are screened out from random samples as difficult samples for contrastive learning, so as to enhance the model's learning effect on multi-entity samples, thereby alleviating the problem of bias caused by the number of entities.

[0022] This embodiment validates the effectiveness of its method by comparing the performance of its constructed model (abbreviated as VPCL) with a multi-class baseline model for multimodal named entity recognition on two datasets. Experimental results demonstrate that this embodiment outperforms other models on datasets such as Twitter-2015 and Twitter-2017. This demonstrates that the multimodal named entity recognition method based on visual prefix and contrastive learning has high accuracy in addressing multimodal named entity recognition.

[0023] Figure 1 The overall structure of the multimodal named entity recognition model construction method based on hierarchical visual prefix and multi-entity difficult sample contrast learning is shown. Figure 1As shown in , for given multimodal unstructured data, the model first extracts features from text and images separately, where multi-scale feature extraction is performed on images, and then the feature scaling is unified. Subsequently, the extracted text and image features are preliminarily fused using a cross-attention mechanism. Furthermore, in the process of training the model, images containing too many entities and their corresponding text pairs are defined as difficult samples, and image features are compared with text features that have been preliminarily fused with image features. During the training phase, the sum of named entity recognition and contrastive learning losses is used as the total loss to train the model. Finally, named entity recognition is performed on the fused features through conditional random fields.

[0024] As a preferred implementation method, the visual cue extraction module specifically extracts image features through the ResNet-50 model, and performs average pooling operations on the feature maps of different shapes output by each sublayer of the ResNet-50 model to unify the width and height of the feature maps of different shapes, wherein the first dimension in the feature map shape represents the number of channels, and the remaining dimensions are the width and height of the feature map; all feature maps are spliced ​​along the channel dimension to obtain a second multi-level feature.

[0025] Specifically, a ResNet-50 model (the preferred model) is used to extract object-level feature pyramids (multi-scale features containing both low-level texture features and high-level semantic information) from the image. The features at different levels are then scaled to a uniform size. Specifically, the image is fed into a ResNet-50 model. The model's four sublayers are traversed, and the feature maps output by each sublayer are average pooled. This yields four feature maps with shapes of (256, 2, 2), (512, 2, 2), (1024, 2, 2), and (2048, 2, 2). These feature maps serve as visual cues for the hierarchy, with the first dimension representing the number of channels, and the remaining dimensions representing the width and height of the input image's feature map. Subsequently, all feature maps are concatenated along dimension 1 (the channel dimension) to form a final shape of (3840, 2, 2) and resized to a final shape of (4, 3840). These visual cues are then used for subsequent multimodal feature fusion.

[0026] As a preferred implementation, the text semantic feature extraction module specifically extracts the hierarchical features of the text data through the BERT model, and uses the first token in the output of the last hidden layer of the BERT model, namely the [CLS] token, as the semantic feature of each text data.

[0027] Specifically, if Figure 2As shown in the figure, given a text segment and its associated visual description image, the BERT model is first used as the base model. The BERT model tokenizes the text and converts the tokenized text into a sequence of IDs. Based on the ID sequence input_ids, the semantic features of the text data are extracted. This is the output of the last hidden layer of the BERT model, with a shape of (sequence_length, hidden_size). Each position in sequence_length corresponds to a token in the input ID sequence, and its vector represents the meaning of the token in the entire context. In BERT, the [CLS] token is typically located at the beginning of this output sequence (sequence_length, hidden_size). The final hidden state of the [CLS] token is considered to represent the entire input ID sequence. Therefore, the first token of sequence_output (i.e., index 0) is used as the text feature representation for each example, completing the semantic extraction of the text data.

[0028] As a preferred implementation, the dynamic gating mechanism unit determines the weights of features at each level through a neural network including an activation function and a normalization function.

[0029] A dynamic gating mechanism determines the importance of each feature level in the visual cue by calculating a weight for each level. This weighted sum is then used to calculate the visual prefix, which is then inserted into the key and value fields of each transformer layer in the BERT model. A multi-head attention mechanism dynamically adjusts the contribution of features from different modalities, completing the initial fusion of multimodal features. Specifically, the system processes multi-level visual features extracted from the image, such as a shape of (4, 3840). These features have different resolutions or levels of abstraction (e.g., low-level texture features and high-level semantic features). To integrate this information, the system performs a weighted average of features from all levels. Subsequently, a dynamic gating mechanism is used to determine the importance of each level of features. This mechanism preferably uses a small neural network to calculate a weight for each level of features. This process ensures that the model can dynamically adjust the contribution of features from each level based on the specific content of the input (4, 3840). After preferably applying an activation function (such as Leaky ReLU) and normalization (such as Softmax), the resulting weight vector reflects the importance of visual features at different levels in the current task. Using this calculated weight vector, the system performs a weighted summation of visual features at different levels to generate a visual prefix. This visual prefix integrates multi-scale and multi-level visual information, providing a more comprehensive representation of the input image's content.

[0030] These generated visual prefixes are then used to enhance each Transformer layer of the BERT model. Specifically, the visual prefixes are inserted into the Transformer's Key and Value sections as additional information. This allows BERT to not only rely on the contextual information of the text itself, but also utilize additional cues provided by images to better understand the text's content. Through these steps, the system achieves a preliminary fusion of multimodal features.

[0031] For convenience, the visual cues of each difficult sample can be preferably used as the image features in the above contrastive learning. Of course, other visual models can also be used to extract the features of difficult samples as the image features here; in addition, a two-layer MLP can be used to perform the above projection operation.

[0032] Furthermore, the Yolo11 model is used to perform target detection on each training sample in the training sample set, and the training samples whose target number reaches the preset requirement are regarded as difficult samples.

[0033] Specifically, when there are too many visual objects in an image, multimodal models typically rely on attention mechanisms to locate key visual objects. This disperses attention weights across a large number of irrelevant objects, making it difficult for the model to focus on the core entities described in the text, and resulting in inaccurate alignment between text and image. To address this issue, this example employs a debiased contrastive learning method, performing contrastive learning on difficult multi-entity samples.

[0034] As an example, Figure 3 As shown in , when performing contrastive learning on multi-entity samples, first, image features and text features are received as input, and then a two-layer MLP is used to project the image features and text features into the contrastive learning space, with dimensions of 128. The projected features are then normalized. Subsequently, the batch size of the image features is obtained. The batch size is a hyperparameter that can be set by yourself. It is set to 8 during implementation. The number of entities in the image is detected by the Yolo model, and the 4 to 6 samples with the largest number of entities in each batch are selected as difficult samples. For each difficult sample index, the similarity between the image and the text is measured by the size of the dot product of the image and text features, and the positive sample similarity and negative sample similarity are calculated, and the contrast loss of each sample is accumulated. Positive sample similarity The calculation formula is: in, It is i The vector after image feature projection, It is i The vector after text feature projection, is the temperature parameter.

[0035] Negative sample pair similarity The calculation formula is: in, is the batch size.

[0036] Contrastive loss for a single hard example The calculation formula is: Finally, the accumulated contrast loss is divided by the number of difficult samples to get the average contrast loss. The calculation formula of the average contrast loss is: in, is the number of difficult samples, is the index set of difficult samples.

[0037] Backpropagation training is performed using the average contrastive loss. This method optimizes the learning process of the shared latent semantic space between text and image, achieving alignment and effectively mitigating the bias caused by the inconsistency in the number of objects in text and image. Finally, named entity recognition results are obtained using conditional random fields.

[0038] In order to better verify the effectiveness of the method of the present invention, the following application examples are given: Figure 4 This paper demonstrates the performance of VPCL against other baseline methods for named entity recognition, with the best results shown in bold. The methods are categorized into two types: text-only named entity recognition methods and multimodal named entity recognition methods based on text and images. While the proposed VPCL slightly trails MNER-QG in precision and HVPNeT in recall on the Twitter-2015 and Twitter-2017 datasets, it achieves the best balance between the two, ultimately outperforming other models in F1 scores. Specifically, on the Twitter-2015 dataset, the proposed method achieves an F1 value of 75.54%, and on the Twitter-2017 dataset, it reaches 87.37%. These performances significantly outperform text-only named entity recognition methods such as BERT-CRF and CNN-BiLSTM-CRF, and also surpass previous multimodal named entity recognition methods. These results demonstrate the effectiveness and robustness of the multimodal named entity recognition method VPCL based on visual prefix and contrastive learning in dealing with multimodal named entity recognition problems.

[0039] like Figure 5 As shown, a text segment from a supply chain scenario and its corresponding image are input. VPCL's ResNet-50 module extracts hierarchical visual features of the unloading scene. For example, low-level features (channel 256) focus on the texture of train tracks and the metallic reflectivity of steel coils, while high-level features (channel 512) focus on the cylindrical contours of stacked steel coils and the lifting robotic arm. These features at different levels are then integrated and resized to resemble the same shape. Dynamic weighting is then used to assign weights to the resized features at different levels to generate a visual prefix associated with the text. This visual prefix is ​​inserted into the Key and Value fields of each Transformer layer in the BERT model. This establishes an association between the "steel coil" in the text and the cylindrical contour in the image, strengthening confidence in the material entity. The Dahai Industrial Park logo in the image and the text "Dahai Industrial Park" are then cross-modally verified to avoid misidentification as other locations.

[0040] In VPCL, the Yolo11 model treats images containing multiple steel coils and numerous other entities as difficult examples and uses contrastive learning to ensure the model focuses on key entities and avoids distractions from irrelevant background objects. For example, an image might show multiple steel coils being unloaded. The model must correctly associate the text "multiple steel coils" with the steel coils in the image while excluding irrelevant objects, such as nearby unloading tools or workers.

[0041] Finally, CRF decoding is performed based on BERT and hierarchical feature fusion to output the final named entity recognition results, namely, company entity: "Pangang Company" (ORG), cargo entity: "steel coil" (MATERIAL), transportation vehicle: "freight train" (VEHICLE), location: "Dahai Industrial Park" (LOCATION).

[0042] Example 2 A multimodal named entity recognition method is provided, wherein a text image pair to be recognized is input into a named entity recognition model constructed by the multimodal named entity recognition model construction method as described above, to realize multimodal named entity recognition.

[0043] The relevant technical solutions are the same as above and will not be repeated here.

[0044] Example 3 The present application also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0045] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory.

[0046] The relevant technical solutions are the same as above and will not be repeated here.

[0047] Example 4 The present application also relates to a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0048] Specifically, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0049] The relevant technical solutions are the same as above and will not be repeated here.

[0050] Example 5 The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method of the above-described embodiment of the present invention.

[0051] The relevant technical solutions are the same as above and will not be repeated here.

[0052] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal named entity recognition model based on visual prefix and contrastive learning, characterized in that: include: Constructing a named entity recognition network, including: a visual cue extraction module, used to extract a first multi-level feature containing low-level texture features and high-level semantic information at the object level in the text image pair, and unify the sizes of different level features in the first multi-level feature by scaling, splicing the features of each level along the channel dimension to obtain a second multi-level feature used as a visual cue; a dynamic gating mechanism unit, used to calculate a weight for each level feature in the visual cue to reflect the importance of the level feature in the current task, and perform a weighted summation of the features of each level in the visual cue based on the weight, and the summation result is inserted into the Transformer of the semantic feature extraction module as a visual prefix; the semantic feature extraction module is used to extract semantic features that fuse visual features on the text data in the text image pair through the Transformer to obtain a fused feature; The named entity recognition network is trained. During the training process, the image features and the fusion features of each difficult sample in a predetermined training sample set are projected into a contrastive learning space, and negative samples and positive samples of each difficult sample are constructed, where the positive sample is the difficult sample and there are multiple negative samples, which are samples composed of the image features in the difficult sample and the fusion features in other difficult samples; the similarity between the image features and the fusion features in all negative samples and positive samples is calculated, and the contrast loss of each difficult sample is calculated based on the similarity corresponding to the positive sample in each difficult sample and the similarity corresponding to all negative samples; the average contrast loss of all difficult samples is added to the named entity recognition loss based on the fusion feature, and the model parameters are adjusted by backpropagation to complete the construction.

2. The method for constructing a multimodal named entity recognition model according to claim 1, wherein: The visual cue extraction module specifically extracts image features through the ResNet-50 model, and performs average pooling operations on the feature maps of different shapes output by each sublayer of the ResNet-50 model to unify the width and height of the feature maps of different shapes, where the first dimension in the feature map shape represents the number of channels and the remaining dimensions are the width and height of the feature map; all feature maps are spliced ​​along the channel dimension to obtain a second multi-level feature.

3. The method for constructing a multimodal named entity recognition model according to claim 1, wherein: The text semantic feature extraction module specifically extracts the hierarchical features of the text data through the BERT model, and uses the first token in the output of the last hidden layer of the BERT model, namely the [CLS] token, as the semantic feature of each text data.

4. The method for constructing a multimodal named entity recognition model according to claim 1, wherein: The dynamic gating mechanism unit determines the weights of features at each level through a neural network including an activation function and a normalization function.

5. The method for constructing a multimodal named entity recognition model according to claim 1, wherein: The visual clues of each difficult sample are used as the image features; and / or, a two-layer MLP is used to perform the projection operation.

6. The method for constructing a multimodal named entity recognition model according to claim 1, wherein: The Yolo11 model is used to perform target detection on each training sample in the training sample set, and the training samples whose target number reaches the preset requirement are regarded as difficult samples.

7. A multimodal named entity recognition method, characterized in that: The text image pair to be recognized is input into the named entity recognition model constructed by the multimodal named entity recognition model construction method according to any one of claims 1 to 6 to realize multimodal named entity recognition.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 and / or the steps of the method according to claim 7 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to perform the steps of the method according to any one of claims 1 to 6 and / or the steps of the method according to claim 7.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 and / or the steps of the method according to claim 7 are implemented.

Images