Multimodal named entity recognition method and device based on depth interaction of image-text features

Abstract

This invention relates to the field of named entity recognition technology, and discloses a multimodal named entity recognition method and apparatus based on deep interaction of text and image features. The method includes: acquiring original text and an original image; extracting multimodal features from the original text and the original image respectively to obtain a text feature sequence and an image feature sequence fused with multi-scale features; performing cross-modal deep interaction fusion on the text feature sequence and the image feature sequence to obtain an interacted multimodal feature sequence; performing context-aware adaptive gating fusion on the interacted multimodal feature sequence to obtain a fused feature sequence; and performing conditional random field sequence decoding on the fused feature sequence to obtain an entity label sequence. This invention achieves deep bidirectional interaction and dynamic adaptive fusion of multimodal information, effectively solving the text ambiguity problem and improving the accuracy of named entity recognition.

Description

Technical Field

[0001] This invention relates to the field of named entity recognition technology, and more specifically to a multimodal named entity recognition method and apparatus based on deep interaction of image and text features. Background Technology

[0002] In the field of natural language processing, named entity recognition (NER) is a fundamental task that plays a crucial role in downstream applications such as information retrieval, question answering systems, and knowledge graph construction. Its goal is to identify and classify entities with specific meanings from text, such as names of people, places, and organizations.

[0003] However, relevant NER methods are mainly geared towards plain text data, and achieve automatic entity recognition through methods such as part-of-speech tagging, syntactic analysis, and statistical learning. But when there is insufficient text context information or semantic ambiguity, entity recognition ambiguity becomes prominent, and the relevant visual information contained in other modal data cannot be effectively utilized, resulting in low entity recognition accuracy. Summary of the Invention

[0004] This invention provides a multimodal named entity recognition method and apparatus based on deep interaction of image and text features to solve the problem of prominent ambiguity in entity recognition and the inability to effectively utilize the relevant visual information contained in other modal data.

[0005] In a first aspect, the present invention provides a multimodal named entity recognition method based on deep interaction of image and text features, the method comprising:

[0006] The original text and original image are obtained, and multimodal feature extraction is performed on the original text and original image respectively to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion.

[0007] Cross-modal deep interactive fusion is performed on the text feature sequence and the image feature sequence after multi-scale feature fusion to obtain the interactive multimodal feature sequence;

[0008] The multimodal feature sequences after interaction are subjected to context-aware adaptive gating fusion to obtain a fused feature sequence;

[0009] The fused feature sequence is decoded using a conditional random field to obtain the entity label sequence.

[0010] This invention provides a multimodal named entity recognition method based on deep interaction of text and image features. By extracting multimodal features from the original text and image, and then fusing the text feature sequence with the image feature sequence after multi-scale feature fusion, a cross-modal deep interaction is achieved. This enables bidirectional deep interaction between text and image features, allowing for a more accurate understanding of the joint semantics of text and images and effectively resolving text ambiguity. Furthermore, by performing context-aware adaptive gating fusion on the interacting multimodal feature sequence, more complex modal dependencies can be learned, enhancing the recognition ability for different types of entities. Additionally, conditional random field sequence decoding is applied to the fused feature sequence, utilizing label transition probabilities to ensure that the output entity label sequence conforms to grammatical rules, guaranteeing the global optimality of the prediction results. This method achieves deep bidirectional interaction and dynamic adaptive fusion of multimodal information, effectively resolving text ambiguity and improving the accuracy of named entity recognition.

[0011] Secondly, the present invention provides a multimodal named entity recognition device based on deep interaction of image and text features, the device comprising:

[0012] The extraction module is used to acquire the original text and the original image, and to extract multimodal features from the original text and the original image respectively, so as to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion;

[0013] The interactive fusion module is used to perform cross-modal deep interactive fusion of text feature sequences and image feature sequences fused with multi-scale features to obtain interactive multimodal feature sequences.

[0014] The adaptive gating fusion module is used to perform context-aware adaptive gating fusion on the multimodal feature sequences after interaction to obtain a fused feature sequence.

[0015] The decoding module is used to perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence.

[0016] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal named entity recognition method based on deep interaction of image and text features as described in the first aspect or any corresponding embodiment.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the multimodal named entity recognition method based on deep interaction of image and text features as described in the first aspect or any corresponding embodiment.

[0018] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the multimodal named entity recognition method based on deep interaction of image and text features as described in the first aspect or any corresponding embodiment. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the first process of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the multi-scale image feature extraction process according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the second process of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention;

[0024] Figure 5 This is a flowchart illustrating the cross-modal Transformer interaction mechanism according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the third process of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention;

[0026] Figure 7 This is a flowchart illustrating the context-aware adaptive gating fusion mechanism according to an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of the overall architecture of a multimodal entity recognition model according to an embodiment of the present invention;

[0028] Figure 9 This is a structural block diagram of a multimodal named entity recognition device based on deep interaction of image and text features according to an embodiment of the present invention;

[0029] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0032] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0033] As an optional application scenario of this invention, such as Figure 1 As shown, this multimodal named entity recognition system based on deep interaction of image and text features may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0034] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0035] With the rapid development of the internet and social media platforms, the amount of user-generated content (UGC) has exploded. This type of content typically exists in a multimodal form, where text information is combined with visual elements such as images and videos to form a complete semantic expression unit. For example, in a social media post, the image is often closely related to its accompanying text, and the two complement each other.

[0036] In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) is a fundamental task that plays a crucial role in downstream applications such as information retrieval, question answering systems, and knowledge graph construction. Its goal is to identify and classify entities with specific meanings from text, such as names of people, places, and organizations. However, existing NER methods primarily target plain text data, achieving automatic entity recognition through methods like part-of-speech tagging, syntactic analysis, and statistical learning. When textual contextual information is insufficient or semantic ambiguity exists, entity recognition ambiguity becomes prominent, failing to effectively utilize relevant visual information contained in other modalities. For example, the specific meaning of "apple" in an Apple product launch tweet is ambiguous. It could refer to edible fruit, classifying it as a miscellaneous (MISC) entity, or it could refer to a technology company, classifying it as an organization (ORG) entity. In such cases, relying solely on textual information is insufficient to accurately distinguish the entity type, while image information contains rich contextual semantics, providing important supplementary clues for entity recognition.

[0037] Current research in Multimodal Named Entity Recognition (MNER) attempts to incorporate visual information into entity recognition tasks, assisting text-based modalities in improving the accuracy of entity localization and classification. MNER aims to identify and extract entities of different data types from multimodal data, using them as additional input to extend traditional text-based named entity recognition and provide a rich and accurate data source for multimodal entity linking.

[0038] However, most existing multimodal named entity recognition schemes employ relatively simple fusion strategies, such as directly concatenating extracted text features with image features, or using a one-way attention mechanism to allow image information to assist in text understanding. These methods fail to fully exploit the bidirectional interaction between modalities, resulting in insufficient modal fusion. Consequently, the accuracy and robustness of entity recognition in complex scenarios still need improvement.

[0039] Therefore, how to design an effective multimodal fusion mechanism, make full use of the complementarity of text and image information, and achieve fine-grained context-aware adaptive fusion has become a key technical problem that urgently needs to be solved in the field of multimodal entity recognition.

[0040] In summary, to address the limitations of insufficient entity recognition accuracy caused by inadequate modal information fusion and the inability to dynamically adapt to context when processing multimodal data, the embodiments of this invention aim to solve the following technical problems:

[0041] (1) Problem of insufficient deep interaction: How to achieve deep two-way semantic interaction between text features and image features, rather than just one-way assistance or simple splicing, so that the two modal information can enhance and calibrate each other, and fully explore the cross-modal semantic association between text and images.

[0042] (2) Static fusion weight problem: How to design a dynamic fusion mechanism that can adaptively adjust the contribution weights of text modality and visual modality information according to the specific content of the current text context, rather than using a fixed or simple weighting method.

[0043] (3) Insufficient expressive power of gating networks: How to construct gating networks with stronger nonlinear expressive power, which can learn complex modal dependencies and achieve accurate weight calculation through deep feature transformation.

[0044] (4) Lack of context awareness: How to not only consider the characteristics of the current token itself when calculating modality fusion weights, but also be able to perceive its local context information (the semantics of the words before and after), so as to make more accurate judgments on ambiguous words.

[0045] (5) Feature representation and sequence dependency problem: How to preserve the spatial structure of the image and the sequence dependency of the text to achieve fine-grained multimodal feature representation; and how to further consider the constraint relationship of the entity label sequence itself on the basis of effectively fusing multimodal information to ensure that the final output entity label sequence is legal and globally optimal.

[0046] To address the aforementioned issues, this invention provides a multimodal named entity recognition method based on deep interaction of text and image features. Through a context-aware adaptive gating fusion framework, it achieves deep interaction and dynamic fusion of multimodal information, thereby improving the accuracy of named entity recognition. This method provides a complete end-to-end solution, covering the entire process from multimodal feature extraction, deep interaction, adaptive fusion to sequence decoding, and has high integration and practical value.

[0047] According to an embodiment of the present invention, a multimodal named entity recognition method based on deep interaction of image and text features is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0048] This embodiment provides a multimodal named entity recognition method based on deep interaction of image and text features, which can be used in the aforementioned terminal devices. Figure 2 This is a flowchart of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0049] Step S201: Obtain the original text and the original image, and perform multimodal feature extraction on the original text and the original image respectively to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion.

[0050] Specifically, the original text and original image are the text and images in the target file, which can be an entity file or related information in a URL.

[0051] Further, the input raw text, which is in sequence form, is received. Text feature extraction is performed on the raw text, specifically including: encoding the input text using a pre-trained language model (such as RoBERTa) to generate word-level feature representations containing rich contextual semantics.

[0052] (1)

[0053] in, A text sequence representing the original input text. For sequence length, The hidden layer dimension of RoBERTa is 768.

[0054] To further enhance the local feature representation, the output sequence of the RoBERTa model is fed into a bidirectional long short-term memory (BiLSTM) network for encoding, resulting in:

[0055] (2)

[0056] (3)

[0057] (4)

[0058] in, and These are the hidden states of the forward and backward LSTMs, respectively. For the input word vectors, This represents the bidirectional hidden state after splicing.

[0059] Finally, a linear projection layer is used to map the output of the BiLSTM to a text feature space of a preset dimension, resulting in the final text feature sequence. :

[0060] (5)

[0061] Wherein, the projection dimension is set to , and For learnable parameters, This is the output text sequence of BiLSTM.

[0062] Furthermore, text encoders can also employ pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and CNN (Convolutional Neural Network).

[0063] Furthermore, such as Figure 3 As shown, the original image paired with the original text is received. This embodiment performs multi-scale image feature extraction on the original image. Unlike using only a single-scale 7×7 feature map, this embodiment simultaneously extracts and fuses features from three scales: high-level, mid-level, and low-level. The steps include: using a pre-trained convolutional neural network (such as ResNet-50) as the backbone network to extract the image's depth features. ResNet-50 is pre-trained on ImageNet and contains 50 convolutional layers; extracting feature maps at multiple scales at different stages. That is, stage 4 outputs high-level semantic features. It contains abstract semantics (such as "this is a person"), and is suitable for recognizing people's names and organization names; That is, the mid-layer features output by stage3 It contains medium-granularity visual patterns, balancing semantics and details; Stage 2 outputs low-level detailed features. It preserves details such as texture and edges, making it suitable for recognizing entities that require detailed information.

[0064] Flatten two-dimensional feature maps of different scales into a one-dimensional sequence:

[0065] (6)

[0066] (7)

[0067] (8)

[0068] Adaptive average pooling is used to unify the feature sequences at different scales to the same number of regions (e.g., 49), resulting in:

[0069] (9)

[0070] (10)

[0071] By using linear projection (Proj), features with different numbers of channels are mapped to the same dimension, resulting in the mapped features:

[0072] (11)

[0073] (12)

[0074] (13)

[0075] By using learnable weights for weighted fusion, the feature representation includes both high-level semantics and preserves low-level details, resulting in more complete image features. The feature representation is as follows:

[0076] (14)

[0077] in, The learnable parameters are determined by softmax normalization.

[0078] Layer normalization is performed on the fused image features, mapping the image region feature sequence to the same dimension as the text features, resulting in the image feature sequence after multi-scale feature fusion. :

[0079] (15)

[0080] Furthermore, the image encoder can also employ other visual backbone networks, such as Vision Transformer, Swin Transformer, or EfficientNet, as long as they can provide sequential local image features.

[0081] Step S202: Perform cross-modal deep interactive fusion of the text feature sequence and the image feature sequence after multi-scale feature fusion to obtain the interactive multimodal feature sequence.

[0082] Step S203: Perform context-aware adaptive gating fusion on the multimodal feature sequence after interaction to obtain the fused feature sequence.

[0083] Specifically, after cross-modal interaction, the enhanced multimodal information needs to be effectively fused for use in the final entity recognition task.

[0084] Step S204: Perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence.

[0085] The multimodal named entity recognition method based on deep interaction of text and image features provided in this embodiment extracts multimodal features from the original text and image, and performs cross-modal deep interaction fusion of the text feature sequence and the image feature sequence after multi-scale feature fusion. This achieves bidirectional deep interaction between text and image features, enabling a more accurate understanding of the joint semantics of text and images and effectively resolving text ambiguity. Furthermore, by performing context-aware adaptive gating fusion on the interacting multimodal feature sequence, more complex modal dependencies can be learned, enhancing the recognition ability for different types of entities. In addition, conditional random field sequence decoding is performed on the fused feature sequence, and the label transition probability is used to ensure that the output entity label sequence conforms to the grammatical rules, guaranteeing the global optimality of the prediction results. The above method realizes deep bidirectional interaction and dynamic adaptive fusion of multimodal information, effectively solving the text ambiguity problem and improving the accuracy of named entity recognition.

[0086] This embodiment provides a multimodal named entity recognition method based on deep interaction of image and text features, which can be used in the aforementioned terminal devices. Figure 4 This is a flowchart of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0087] Step S401: Obtain the original text and the original image, and perform multimodal feature extraction on the original text and the original image respectively to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion. For details, please refer to... Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0088] Step S402: Perform cross-modal deep interactive fusion of the text feature sequence and the image feature sequence after multi-scale feature fusion to obtain the interactive multimodal feature sequence.

[0089] Specifically, step S402 includes:

[0090] Step S4021 involves superimposing position encoding and modality type embedding onto the text feature sequence and the image feature sequence after multi-scale feature fusion, respectively, to obtain the encoded enhanced text feature sequence and the encoded enhanced image feature sequence.

[0091] Specifically, such as Figure 5 As shown, in order to distinguish the position and information source modality in the sequence, learnable positional encoding and modal embedding are superimposed on the text feature sequence and the image feature sequence after multi-scale feature fusion, respectively.

[0092] Furthermore, sinusoidal positional encoding is added to the text feature sequence, with an even-numbered dimension. for:

[0093] (16)

[0094] in, Represents the sine position code. Indicates the position index of a word in the sequence. The dimension index of the position encoding vector.

[0095] Odd Dimensions for:

[0096] (17)

[0097] Furthermore, a two-dimensional positional encoding (considering the row and column positions of the original feature map) is added to the image feature sequence after multi-scale feature fusion, with even-numbered dimensions. for:

[0098] (18)

[0099] in, This represents the row and column coordinates of the image features in the original feature map.

[0100] Odd Dimensions for:

[0101] (19)

[0102] Furthermore, add modality 0 embeddings to all text feature vectors:

[0103] (20)

[0104] Add modality 1 embeddings to all image feature vectors:

[0105] (twenty one)

[0106] in, and These represent the positional codes for text and images, respectively. Represents the modality type embedding matrix. and These represent the encoded enhanced text feature sequence and the encoded enhanced image feature sequence, respectively.

[0107] Step S4022: The encoded enhanced text feature sequence and the encoded enhanced image feature sequence are concatenated in the sequence dimension to obtain a mixed modality sequence.

[0108] Specifically, such as Figure 5 As shown, the encoded and enhanced text feature sequence is concatenated with the image feature sequence fused with multi-scale features along the sequence dimension to form a longer mixed modality sequence. :

[0109] (twenty two)

[0110] Step S4023: Perform Transformer encoding on the mixed modal sequence to obtain the interactive multimodal feature sequence.

[0111] Specifically, Transformer encoding of mixed-modal sequences can be performed using a Transformer encoder (a sequence model based on an attention mechanism), or other variants, such as Transformers with specific sparse attention patterns, to improve computational efficiency.

[0112] Furthermore, such as Figure 5 As shown, inputting the mixed sequence into a multi-layer (e.g., 6-layer) multi-head Transformer encoder yields:

[0113] (twenty three)

[0114] in, To prevent padding positions from participating in attention calculations, the attention mask matrix is ​​used to ensure that padding positions are not involved in attention calculations.

[0115] Under the self-attention mechanism of Transformer, each element in the sequence (whether it is a text token or an image region) will calculate an attention score with all other elements and perform information aggregation.

[0116] The multi-head attention mechanism in Transformer is defined as follows:

[0117] (twenty four)

[0118] (25)

[0119] (26)

[0120] in, This represents the output of concatenating all heads along the feature dimension. This represents the output projection matrix, which maps the concatenated features back to the original dimensions. , , Indicates the first The projection matrix of the head, Represents the query matrix. Represents the key matrix. Represents a value matrix. The dimension of the key vector. This represents the scaling factor, preventing the gradient from vanishing due to an excessively large dot product.

[0121] Furthermore, such as Figure 5 As shown, multi-head attention allows each feature in the mixed sequence to pay attention to other features in the sequence (e.g., text words pay attention to corresponding image patches, and image patches pay attention to corresponding text words), achieving cross-modal feature association; Add & Layer Norm: residual connection (Add) + layer normalization (Layer Norm) stabilizes training and preserves the original features; Feed-Forward Network (FFN): performs non-linear transformation on the features output by attention to further extract deep features; the above process is repeated in multiple layers, so that text features and image features can fully interact and align bidirectionally within the model, complementing each other and enhancing semantic representation.

[0122] Step S403: Perform context-aware adaptive gating fusion on the interacted multimodal feature sequence to obtain the fused feature sequence. For details, please refer to [link to relevant documentation]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0123] Step S404: Perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0124] The multimodal named entity recognition method based on deep interaction of image and text features provided in this embodiment enables the model to more accurately understand the joint semantics of image and text through bidirectional deep interaction achieved by cross-modal Transformer, effectively solving the problem of text ambiguity.

[0125] This embodiment provides a multimodal named entity recognition method based on deep interaction of image and text features, which can be used in the aforementioned terminal devices. Figure 6 This is a flowchart of a multimodal named entity recognition method based on deep interaction of image and text features according to an embodiment of the present invention, such as... Figure 6 As shown, the process includes the following steps:

[0126] Step S601: Obtain the original text and the original image, and perform multimodal feature extraction on the original text and the original image respectively to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion. For details, please refer to... Figure 4 Step S401 of the illustrated embodiment will not be described again here.

[0127] Step S602 involves performing cross-modal deep interactive fusion on the text feature sequence and the image feature sequence fused with multi-scale features to obtain the interactive multimodal feature sequence. For details, please refer to [link to details]. Figure 4 Step S402 of the illustrated embodiment will not be described again here.

[0128] Step S603: Perform context-aware adaptive gating fusion on the multimodal feature sequence after interaction to obtain the fused feature sequence.

[0129] Specifically, step S603 includes:

[0130] Step S6031: Sequence separation is performed on the multimodal feature sequence after interaction to obtain text feature vector and image feature vector.

[0131] Specifically, after the above 6 layers of Transformer encoding, the output sequence is separated into text parts. and image part ,forward The first 49 are text feature vectors, and the last 49 are image feature vectors.

[0132] Step S6032: Context-aware calculations are performed on the text feature vector and the image feature vector respectively to obtain the context statistical feature vector and the image context vector.

[0133] Specifically, such as Figure 7 As shown, unlike considering only the relationship between the current text token and the image, a local context window is introduced. By comprehensively considering the semantics of the preceding and following words, gating decisions can be made more accurate by providing semantic clues. For example, for the word "Apple" in "Apple released a new iPhone", it is easier to determine whether to rely on an image by combining the context of "released" and "iPhone". For the word "Apple" in "I bought an apple", it is also easier to determine the context of "bought" and "apple".

[0134] In some optional implementations, step S6032 above includes:

[0135] Step a1: Obtain the local context window and extract local context features based on the text feature vector using the local context window.

[0136] Specifically, for the first result obtained after the interaction Feature vector of a text token It not only considers its own characteristics, but also introduces a local context window and sets it to (Two tokens before and after), extract local context features, and apply padding or truncation to the boundary positions. The local context features... It can be represented as:

[0137] (27)

[0138] Step a2: Perform mean pooling and max pooling on the local context features to obtain the context statistical feature vector.

[0139] Specifically, the correlation between the token and its context and the features of all image regions is calculated through an attention mechanism, and the context statistical feature vector is calculated using mean pooling and max pooling. and :

[0140] (28)

[0141] (29)

[0142] Step a3: Determine the query vector based on the text feature vector and the context statistical feature vector, and determine the key vector based on the image feature vector.

[0143] Specifically, query vectors that incorporate local contextual information It can be represented as:

[0144] (30)

[0145] Furthermore, the key vector derived from image region features It can be represented as:

[0146] (31)

[0147] in, .

[0148] Step a4: Calculate the attention weights based on the query vector and key vector, and normalize the attention weights to obtain the normalized attention weights.

[0149] Specifically, calculate attention weights :

[0150] (32)

[0151] Furthermore, the attention weights are normalized to obtain the normalized attention weights. :

[0152] (33)

[0153] Step a5: Calculate the image context vector based on the image feature vector and the normalized attention weights.

[0154] Specifically, the weighted summation yields the enhanced image context vector. The calculation formula is as follows:

[0155] (34)

[0156] Step S6033: Based on text feature vectors, context statistical feature vectors, and image context vectors, a deep gating network is used to determine the gating weights.

[0157] Specifically, such as Figure 7 As shown, compared to the simple structure of "feature concatenation + linear transformation + Hadamard product", the deep structure of "three-layer MLP (Multilayer Perceptron) + residual connection" can learn more complex modal dependencies. Therefore, a deep gating network is used to determine the gating weights. The deep gating network adopts a three-layer MLP structure, including a feature preprocessing layer, a nonlinear mapping layer, and a gating weight generation layer. The first layer introduces contextual statistical features (mean, maximum value) to enrich the information; the second layer, Tanh nonlinear mapping, provides stronger feature transformation capabilities; and the residual connection prevents gradient vanishing, making the training of the deep network more stable.

[0158] Furthermore, the gating mechanism can be simplified to a two-layer MLP to calculate the fusion weights, or other feature fusion techniques such as bilinear pooling can be introduced.

[0159] In some optional implementations, step S6033 includes:

[0160] Step b1 involves concatenating the text feature vector, context statistical feature vector, and image context vector, and then performing a nonlinear transformation on the concatenated comprehensive feature vector to obtain the nonlinearly transformed feature vector.

[0161] Specifically, the feature preprocessing layer concatenates text features, image context features, and contextual statistical features (mean pooling and max pooling) to obtain a concatenated comprehensive feature vector. :

[0162] (35)

[0163] The above-mentioned concatenated comprehensive feature vector Perform a nonlinear transformation to obtain the eigenvectors after the nonlinear transformation. :

[0164] (36)

[0165] Step b2: Perform nonlinear mapping on the eigenvectors after nonlinear transformation to obtain the nonlinearly mapped eigenvectors.

[0166] Specifically, the nonlinear mapping layer performs deep feature learning using the Tanh activation function (hyperbolic tangent activation function) to extract more abstract feature representations. The expression for the Tanh activation function is:

[0167] (37)

[0168] in, , , This represents the eigenvectors after nonlinear mapping.

[0169] Step b3: Based on the eigenvectors after nonlinear transformation, determine the eigenvectors after residual connection using residual connection.

[0170] Specifically, the gated weight generation layer introduces residual connections, the expression for which is:

[0171] (38)

[0172] in, , This is a learnable scalar parameter (initialized to 0.1). This represents the eigenvector after the residual.

[0173] Step b4: Fuse the eigenvectors after nonlinear mapping and the eigenvectors after residuals to obtain the fused eigenvectors.

[0174] Specifically, the features after fusing the residuals, i.e., the fused feature vector. The expression is:

[0175] (39)

[0176] Step b5: Perform a linear transformation on the fused feature vector to obtain the gating weights.

[0177] Specifically, to prevent gradient vanishing, the final gate weights are generated. :

[0178] (40)

[0179] in, , .

[0180] Step S6034: Adaptive temperature adjustment is performed on the text feature vector and image feature vector using gating weights to obtain text modal weights and image modal weights.

[0181] Specifically, to address the issue of unadjustable weight distribution characteristics caused by using a fixed softmax (normalized exponential function), a learnable temperature coefficient is introduced to dynamically adjust the weight distribution. The softmax is calculated adaptively to the temperature coefficient, and finally, the weights are gated for calculation. When the value is small (e.g., 0.5), the softmax distribution is sharp, with weights concentrated in a specific mode, making it suitable for samples with high certainty (e.g., cases with clearly defined images). When the temperature... When the value is large (e.g., 4.0), the softmax distribution is smooth and the weights of the two modes are relatively balanced, which is suitable for blurry samples (e.g., when there is insufficient image information). The model improves its adaptability to different samples by automatically learning the optimal temperature for each sample, which is equivalent to introducing a "confidence" mechanism.

[0182] Furthermore, such as Figure 7 As shown, a learnable temperature coefficient is introduced to dynamically adjust the concentration of the weight distribution based on the degree of certainty of the sample. For samples with high certainty, the temperature coefficient is smaller and the weight distribution is more concentrated, while for fuzzy samples, the temperature coefficient is larger and the weight distribution is smoother.

[0183] In some optional implementations, step S6034 above includes:

[0184] Step c1: Based on text feature vectors and image feature vectors, an adaptive temperature coefficient is determined using a temperature prediction network.

[0185] Specifically, text feature vectors and image feature vectors are used as input to the temperature prediction network:

[0186] (41)

[0187] Single-layer network predicts temperature coefficient (i.e., adaptive temperature coefficient) The calculation formula is:

[0188] (42)

[0189] in , For learnable parameters, The preset maximum temperature value (set to 5.0). .

[0190] Step c2: Determine the text modality weights and image modality weights based on the adaptive temperature coefficient and gating weights.

[0191] Specifically, scaling using temperature coefficient Expand to calculate text modal weights, text modal weights The calculation formula is:

[0192] (43)

[0193] Expand to calculate image modal weights, image modal weights The calculation formula is:

[0194] (44)

[0195] in, .

[0196] Step S6035: The text feature vector and the image feature vector are weighted and fused using text modality weight and image modality weight respectively to obtain the fused feature sequence.

[0197] Specifically, linear projection is performed on the text and image features respectively to obtain the linearly projected text feature vector and image feature vector, i.e. and The expression for linear projection is:

[0198] (45)

[0199] (46)

[0200] in, These are learnable parameters.

[0201] Furthermore, text modality weights and image modality weights are used to weight and combine the text features and image context features to obtain the fused features. :

[0202] (47)

[0203] Furthermore, by stabilizing the training process through layer normalization, the expression for layer normalization is:

[0204] (48)

[0205] in, The fusion features are represented after the normalization of the layer.

[0206] Furthermore, the above process is repeated for all text tokens to obtain the final fused feature sequence. :

[0207] (49)

[0208] Step S604: Perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence.

[0209] Specifically, in order to resolve the dependencies between entity labels (for example, the start label "B-LOC" of a "location" entity should be followed by its inner label "I-LOC"), a Conditional Random Field (CRF) is used as the decoding layer.

[0210] Furthermore, in the entity label sequence decoding stage, in simple scenarios, the CRF layer can be replaced with a softmax classifier, but the label sequence validity constraint must be sacrificed.

[0211] Furthermore, the fused feature sequence is input into a fully connected classification layer to obtain the emission score of each token across all entity labels. :

[0212] (50)

[0213] in Here is the weight matrix of the classifier. This is the bias vector of the classifier. This is the sequence length (i.e., the number of text tokens). It represents the number of tag categories.

[0214] Furthermore, the CRF layer combines the aforementioned emission scores with a learnable label transition score matrix and uses the Viterbi algorithm to decode a globally optimal and valid entity label path as the final recognition result.

[0215] Furthermore, the CRF layer defines the output label sequence given the input. conditional probability :

[0216] (51)

[0217] in Input data (text or image). For candidate outputs (category labels, words, etc.) The set of all possible label sequences.

[0218] Furthermore, a scoring function is used to measure the degree of matching between the input and output. The expression for the scoring function is:

[0219] (52)

[0220] in, The input sequence is (text / image features). To output the label sequence (length) ), For the first The input feature vector at each position, For tags Embedded vector, It is a transition matrix. Indicates from the label Move to label The score.

[0221] Furthermore, during training, the negative log-likelihood loss is minimized. :

[0222] (53)

[0223] Furthermore, the Viterbi algorithm is used during inference. Find the optimal path.

[0224] Furthermore, unlike methods that do not impose constraints on gating weights, entropy regularization is introduced to prevent modal bias and minimize... Equivalent to maximizing This encourages a more even distribution of weights and avoids always... or This improves the model's generalization ability, while entropy regularization allows the model to maintain sensitivity to both modes, preventing the model from overfitting to the mode preference of the training data.

[0225] Furthermore, a multi-task loss function is adopted, and the total loss function includes the CRF loss. And entropy regularization term Total loss function It can be represented as:

[0226] (54)

[0227] Among them, the gated regularization loss The calculation formula is:

[0228] (55)

[0229] The formula for calculating entropy is:

[0230] (56)

[0231] The above calculations prevent the gating mechanism from always favoring a certain mode.

[0232] Furthermore, a multimodal entity recognition model is trained through the above steps, and then the trained multimodal entity recognition model is used to perform entity recognition and output entity label sequences.

[0233] The multimodal named entity recognition method based on deep interaction of image and text features provided in this embodiment has stronger nonlinear expression capabilities based on a three-layer multilayer perceptron and a deep gated network with residual connections. Compared with simple single-layer linear transformation, it can learn more complex modal dependencies. Secondly, the context-aware mechanism enables the model to dynamically determine whether to focus more on text or image when recognizing different words, which enhances the model's adaptability and robustness in cases of incomplete modal information or noise. Multi-scale image feature fusion provides richer visual representations and enhances the model's ability to recognize different types of entities. In addition, the adaptive temperature coefficient can dynamically adjust the weight distribution according to the determinism of the samples, generating concentrated weights for clear samples and smoothing weights for blurry samples. Entropy regularization constraints prevent the model from over-relying on a certain modality and maintain good generalization ability. Finally, combined with the conditional random field decoding layer, the label transition probability ensures that the output entity label sequence conforms to the grammatical rules (e.g., "B-PER" cannot be followed by "I-LOC"), guaranteeing the global optimality of the prediction results.

[0234] The following specific example illustrates the steps and effectiveness of a multimodal named entity recognition method based on deep interaction of image and text features.

[0235] Example 1:

[0236] like Figure 8 As shown, a context-aware adaptive gating fusion framework realizes deep interaction and dynamic fusion of multimodal information to improve the accuracy of named entity recognition. The specific steps of the multimodal named entity recognition method based on deep interaction of image and text features include:

[0237] Input text and image. The text is processed by a text encoder to output a text feature sequence. The text encoder uses RoBERTa and BiLSTM. The image is processed by a multi-scale image encoder to output a multi-scale feature fusion image feature sequence.

[0238] By performing cross-modal deep interactive fusion of text feature sequences and image feature sequences fused with multi-scale features, text features and image features can fully interact and align bidirectionally within the model, complementing and enhancing semantic representation, and outputting the interactive multimodal feature sequence.

[0239] The adaptive fusion module utilizes MLP-based nonlinear gating to perform context-aware adaptive gating fusion on the multimodal feature sequences after interaction, resulting in a fused feature sequence.

[0240] The fused feature sequence is decoded using a conditional random field to obtain the entity label sequence.

[0241] Example 2:

[0242] To verify the technical effectiveness of the multimodal named entity recognition method based on deep interaction of image and text features, two widely used datasets in the field of multimodal entity recognition, Twitter2015 and Twitter2017, were selected as experimental data. Both datasets include four types of entities: Person (PER), Location (LOC), Organization (ORG), and Miscellaneous (MISC). Each sentence has a corresponding image.

[0243] Twitter2015 collected image and text content generated by Twitter users between 2014 and 2015. The annotation was completed independently by two people using the BIO method, totaling 8257 sentences, including 4000 in the training set, 1000 in the validation set, and 3257 in the test set, with a total of 12800 entities.

[0244] Twitter2017 collected multimodal tweets on the themes of "sports" and "social" from 2016 to 2017. The data was labeled by three individuals using the BIO method (a sequence labeling method). There were a total of 4,819 sentences, including 3,373 in the training set, 723 in the validation set, and 723 in the test set, with a total of 8,724 entities.

[0245] To reduce the error caused by experimental randomness, five different random seeds were set for the proposed method (42, 100, 500, 2025, 3407), and the average F1 score (a comprehensive index used to measure the performance of classification models) of the five experimental results was used as the evaluation index, and the standard deviation was given.

[0246] While the large model possesses powerful general-purpose capabilities, the dedicated small model proposed in this invention achieves a superior performance ratio in fine-grained multimodal entity alignment tasks within specific domains through deep feature interaction. Based on preliminary experiments and theoretical analysis, the experimental results are shown in Table 1 below. Through experimental comparison, this invention demonstrates certain effectiveness in multimodal named entity recognition tasks.

[0247] Table 1:

[0248]

[0249] On the Twitter2015 and Twitter2017 datasets, it significantly improves the precision and recall of entity recognition compared to the baseline method.

[0250] This embodiment also provides a multimodal named entity recognition device based on deep interaction of image and text features. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0251] This embodiment provides a multimodal named entity recognition device based on deep interaction of image and text features, such as... Figure 9 As shown, it includes:

[0252] The extraction module 901 is used to acquire the original text and the original image, and to extract multimodal features from the original text and the original image respectively, so as to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion;

[0253] The interactive fusion module 902 is used to perform cross-modal deep interactive fusion of text feature sequences and image feature sequences fused with multi-scale features to obtain interactive multimodal feature sequences.

[0254] The adaptive gating fusion module 903 is used to perform context-aware adaptive gating fusion on the multimodal feature sequence after interaction to obtain a fused feature sequence;

[0255] The decoding module 904 is used to perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence.

[0256] The multimodal named entity recognition device based on deep interaction of image and text features provided in this embodiment of the invention can execute the multimodal named entity recognition method based on deep interaction of image and text features provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0257] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0258] The following is a detailed reference. Figure 10 This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from memory 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0259] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0260] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1009, or installed from a memory 1008, or installed from a ROM 1002. When the computer program is executed by the processor 1001, it performs the functions defined in the multimodal named entity recognition method based on deep interaction of graph and text features according to embodiments of the present invention.

[0261] Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0262] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the multimodal named entity recognition method based on deep interaction of graph and text features shown in the above embodiments is implemented.

[0263] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0264] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multimodal named entity recognition method based on deep interaction of image and text features, characterized in that, The method includes: Obtain the original text and the original image, and perform multimodal feature extraction on the original text and the original image respectively to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion; The text feature sequence and the image feature sequence fused with the multi-scale features are subjected to cross-modal deep interactive fusion to obtain the interactive multimodal feature sequence; The multimodal feature sequence after the interaction is subjected to context-aware adaptive gating fusion to obtain a fused feature sequence; The fused feature sequence is decoded using a conditional random field to obtain an entity label sequence; The step of performing context-aware adaptive gating fusion on the multimodal feature sequence after the interaction to obtain a fused feature sequence includes: The multimodal feature sequence after the interaction is separated to obtain text feature vectors and image feature vectors; Context-aware calculations are performed on the text feature vector and the image feature vector respectively to obtain the context statistical feature vector and the image context vector; Based on the text feature vector, the context statistical feature vector, and the image context vector, a deep gating network is used to determine the gating weights; wherein, the deep gating network adopts a three-layer MLP structure, including a feature preprocessing layer, a nonlinear mapping layer, and a gating weight generation layer; The text feature vector and the image feature vector are adaptively temperature-adjusted using the gating weights to obtain the text modal weights and the image modal weights, respectively. The text feature vector and the image feature vector are weighted and fused using the text modality weight and the image modality weight, respectively, to obtain the fused feature sequence; The step of determining gating weights using a deep gating network based on the text feature vector, the context statistical feature vector, and the image context vector includes: The text feature vector, the context statistical feature vector, and the image context vector are concatenated, and the concatenated comprehensive feature vector is subjected to a nonlinear transformation to obtain a nonlinearly transformed feature vector; wherein, the feature preprocessing layer concatenates the text features, image context features, and context statistical features to obtain the concatenated comprehensive feature vector. : in, For text feature vectors, For image context vectors, and Contextual statistical feature vectors; The concatenated composite feature vector Perform a nonlinear transformation to obtain the eigenvectors after the nonlinear transformation. : in, and These are learnable parameters; The feature vector after the nonlinear transformation is nonlinearly mapped to obtain a nonlinearly mapped feature vector. The nonlinear mapping layer uses the Tanh activation function for deep feature learning to extract a more abstract feature representation. The expression for the Tanh activation function is: in, This represents the eigenvector after nonlinear mapping. and These are learnable parameters; Based on the feature vector after the nonlinear transformation, the residual feature vector is determined using residual connections; wherein, the gated weight generation layer introduces residual connections, and the expression for the residual connections is: in, For learnable scalar parameters, This represents the eigenvector after the residual. These are learnable parameters; The eigenvectors after nonlinear mapping and the eigenvectors after residual are fused to obtain a fused eigenvector; wherein, the fused eigenvector... The expression is: The fused feature vector is linearly transformed to obtain the gate weights; wherein the gate weights The expression is: in, and These are learnable parameters.

2. The method according to claim 1, characterized in that, The step of performing cross-modal deep interactive fusion of the text feature sequence and the image feature sequence fused with the multi-scale features to obtain the interactive multimodal feature sequence includes: The text feature sequence and the image feature sequence after multi-scale feature fusion are respectively superimposed with position encoding and modality type embedding to obtain the encoded enhanced text feature sequence and the encoded enhanced image feature sequence; The encoded and enhanced text feature sequence and the encoded and enhanced image feature sequence are concatenated along the sequence dimension to obtain a mixed modality sequence; The mixed-modal sequence is encoded using Transformer to obtain the interactive multimodal feature sequence.

3. The method according to claim 1, characterized in that, The step of performing context-aware calculations on the text feature vector and the image feature vector respectively to obtain a context statistical feature vector and an image context vector includes: Obtain a local context window, and extract local context features based on the text feature vector using the local context window; The local context features are subjected to mean pooling and max pooling respectively to obtain the context statistical feature vector; The query vector is determined based on the text feature vector and the context statistical feature vector, and the key vector is determined based on the image feature vector; The attention weights are calculated based on the query vector and the key vector, and then the attention weights are normalized to obtain the normalized attention weights. The image context vector is calculated based on the image feature vector and the normalized attention weights.

4. The method according to claim 1, characterized in that, The step of adaptively adjusting the text feature vector and the image feature vector using the gating weights to obtain text modality weights and image modality weights includes: Based on the text feature vector and the image feature vector, an adaptive temperature coefficient is determined using a temperature prediction network. The text modality weights and the image modality weights are determined based on the adaptive temperature coefficient and the gating weights.

5. A multimodal named entity recognition device based on deep interaction of image and text features, characterized in that, The apparatus for implementing the multimodal named entity recognition method based on deep interaction of image and text features as described in any one of claims 1 to 4 includes: The extraction module is used to acquire the original text and the original image, and to extract multimodal features from the original text and the original image respectively, so as to obtain the text feature sequence and the image feature sequence after multi-scale feature fusion; The interactive fusion module is used to perform cross-modal deep interactive fusion of the text feature sequence and the image feature sequence after multi-scale feature fusion to obtain the interactive multimodal feature sequence. An adaptive gating fusion module is used to perform context-aware adaptive gating fusion on the multimodal feature sequence after the interaction to obtain a fused feature sequence; The decoding module is used to perform conditional random field sequence decoding on the fused feature sequence to obtain the entity label sequence.

6. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal named entity recognition method based on deep interaction of image and text features as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the multimodal named entity recognition method based on deep interaction of image and text features as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute the multimodal named entity recognition method based on deep interaction of image and text features as described in any one of claims 1 to 4.

Images