A visual prefix-based generative multi-modal information extraction method

Abstract

The application discloses a kind of generative multimodal information extraction methods based on visual prefix in the technical field of deep learning natural language processing, specifically: input text information and image information, realize text and image relationship extraction, visual representation as insertable visual prefix, to guide error-insensitive prediction decision text representation, realize hierarchical multi-scale visual features as fused visual prefix, and multimodal information extractor effectively extracts features.The application fuses visual information and text information, interacts and updates visual features and text features through multi-level visual prefix attention mechanism, combines a unified multimodal information extractor, unifies multimodal information extraction task as a generation problem using instruction tuning, and can realize automatic regression generation information extraction result.

Description

TECHNICAL FIELD

[0001] The application is in the technical field of deep learning natural language processing, and specifically relates to a generative multimodal information extraction method based on visual prefixes. BACKGROUND

[0002] Multimodal information extraction is a research field involving natural language text, images, audio and other modal data, including MNER (Multimodal Named Entity Recognition) and MRE (Multimodal Relation Extraction). With the development of the Internet, a large amount of text, image and audio data has been widely generated, which contains rich information.

[0003] The essence of MNER and MRE tasks is how to learn great visual features and how to incorporate them into text representations to improve NER (Named Entity Recognition) and RE (Relation Extraction). Most of them rely heavily on pre-training on a large amount of extra annotated image-text correlation corpus, and only focus on the whole image, ignoring the bias of relevant object-level visual fusion. In practical applications, irrelevant objects will directly affect the discourse reasoning; at the same time, it is not a trivial matter to obtain absolutely relevant object-level visual information to enhance the text. Therefore, an effective method should learn better visual representations and reduce the error sensitivity of irrelevant object images for social media NER and RE tasks.

[0004] Current methods usually focus on the above specific tasks, which mostly use task-specific model structures and make dataset-specific adjustments for the task at hand. Such a paradigm leads to some limitations: first, it leads to a lack of generalizability because they tend to be too suitable for the patterns of specific tasks or even datasets. Second, it is time-consuming to design, train and maintain separate models for each task, hindering the process of large-scale deployment of practical multimodal systems. Finally, due to their independent training nature, these methods cannot effectively utilize the shared knowledge between different multimodal information extraction tasks, thus weakening the performance of each task. SUMMARY

[0005] In order to solve the above technical problems, the application provides a generative multimodal information extraction method based on visual prefix, which fuses visual information and text information, interacts and updates visual features and text features through a multi-level visual prefix attention mechanism, combines a unified multimodal information extractor, and unifies the multimodal information extraction task into a generative problem using instruction optimization, so that the automatic regression generation information extraction result can be realized.

[0006] In order to achieve the above purpose, the application is realized by the following technical scheme:

[0007] The application is a generative multimodal information extraction method based on visual prefix, comprising the following steps:

[0008] Step 1, starting from a visualizable backbone of a deep learning model, for an image of a given data set, the deep learning model is used for processing, and the preprocessing operation includes: normalizing the pixel value to [0, 1], randomly cropping the edge by 10%-20%, randomly horizontally flipping or color jittering, the generated I is the standardized input tensor, which is consistent with the dimension of the input image, and the Transformer extracts layered multi-scale visual features from the input image;

[0009] Step 2, using a dynamic gate module to predict the gating probability of each layer of multi-scale visual features The degree vector of the i-th multi-scale visual feature when executing the l-th layer is represented, and the value of the degree vector reflects the importance of each multi-scale visual feature to each layer in the Transformer;

[0010] Step 3, based on the dynamic gate module, the gating probability is multiplied by the multi-scale visual features of each layer and they are connected to derive the final aggregated layered visual features V g to match the l-th layer of the image encoder in the Transformer, and obtain the visual prefix feature

[0011] Step 4, the visual prefix feature obtained in step 3 is input as a visual prefix, and the visual prefix feature is input into each layer of the Transformer, and the input visual prefix feature is gradually enhanced and converted through the self-attention mechanism and the feedforward neural network, so as to extract fine image feature representation layer by layer, and obtain the final visual feature h v ;

[0012] Step 5, given the sequence of input text W={W1, W2,..., W m}, m is the length of the text sequence, the final text feature h is calculated by the text encoder e , the cross-modal information fusion is realized using a dynamic gate control strategy g', the text-aware visual representation M is obtained, the final text feature h e is combined with the text-aware visual representation M to generate the final cross-modal C.

[0013] Step 6, the text decoder generates the output structure in an autoregressive manner, encodes and decodes the cross-modal C obtained in step 5, and repeats step 6, in the i-th step, the state of the text decoder is represented as the final cross-modal C and the previous state as the condition.

[0014] Further improvements of the present application are that in the step 1, ResNet is used as the Transformer processing the image, the Transformer includes an input layer, an encoder and a decoder,

[0015] The input layer adjusts the input image to a standard size HxWxC to generate an input feature tensor I through preprocessing:

[0016] I=Preprocess(I raw )

[0017] Where I raw represents the original RGB input image with a size of HxWxC.

[0018] The encoder includes an initial convolutional layer, a multi-resolution extraction module and a feature fusion layer, wherein

[0019] The initial convolutional layer uses grouped convolution with a kernel size of kxk and a channel number of c1, and improves the nonlinear representation ability through an activation function GELU, specifically:

[0020] V init =GELU(Conv(I,k=7,c1=64,s=2))

[0021] Where V init is the output tensor and s is the stride.

[0022] The multi-resolution extraction module includes a multi-scale pyramid module and a channel attention module, each branch in the multi-scale pyramid module has a different convolution kernel size k1, k2, k3, and outputs features of different resolutions, respectively, the outputs of different branches are spliced through the channel dimension of the channel attention module to generate multi-scale features V ms ,

[0023] V ms =Concat(Conv(V init, k1=3, c1=32), Conv(V init , k2=5, c2=32), Conv(V init , k3=7, c3=32)

[0024] Multi-scale feature V ms For extracting multi-scale features, capturing semantic information at different resolutions;

[0025] Feature fusion layer: the depth separable convolution of the feature fusion layer reduces the computational overhead while maintaining the spatial resolution, and outputs a tensor feature V∈R H′×W′×C′ :

[0026] V=Attention(DepthwiseConv(V ms , k=3, c2=128)

[0027] In the depth separable convolution, the convolution kernel k is 3, and the output channel c2 is 128; in the self-attention mechanism, matrix multiplication is used to extract global context information and enhance the global consistency of the features;

[0028] Decoder: including a global context fusion module, using a self-attention mechanism to enhance the output tensor feature V representation, output hierarchical features Each layer corresponds to a specific scale, and the hierarchical features Embedding time step position encoding, forming hierarchical multi-scale visual feature representation V1, V2,..., V n :

[0029] {V1, V2, V3, V n}=HierarchicalSplit(V)

[0030] The output tensor feature V is processed in layers to generate multi-scale outputs, and the resolution of each multi-scale visual feature decreases.

[0031] Finally, global context enhancement is performed to obtain context-aware output features V context :

[0032] V context =Attention(GAP(V))

[0033] Wherein, GAP is global average pooling, and Attention is self-attention.

[0034] Further improvement of the application lies in that the step 2 specifically comprises the following steps:

[0035] Step 2.1, generating logits of gate signals

[0036]

[0037] where f(·) is the activation function ReLU, w i is the weighted coefficient calculated for each output tensor feature V i i, P(V i ) is the feature after pooling of the i-th output tensor feature;

[0038] Step 2.2, each output tensor feature V i will be given a weight w i according to its spatial position in the image;

[0039] Step 2.3, based on the logits calculated in step 2.1, the Softmax function is used to normalize to obtain the gating probability of each layer

[0040]

[0041] This step is still to generate a probability distribution between [0, 1] from logits, which is used to control the contribution of each output tensor feature in the current Transformer.

[0042] Further improvement of the application is that the step 3 specifically comprises the following steps:

[0043] Step 3.1, the dynamic gate g derives the final aggregated visual prefix feature to match the l-th layer in the Transformer, denoted as:

[0044]

[0045] where, is the gating probability of each layer of visual prefix feature , indicating the weighted importance of the feature, is the visual feature from different scales in the l-th layer;

[0046] Step 3.2, formally, the visual prefix feature corresponding to the l-th layer of the Transformer is obtained through the following connection operation:

[0047]

[0048] where, is the fusion result of multi-scale visual features, is the visual prefix feature, and the weighted and gated visual prefix feature is obtained through the dynamic gate module, and the visual prefix feature The visual prefix-based attention mechanism will be used to enhance the hierarchical representation of the text modality.

[0049] A further improvement of the present application is that in step 4, the input visual prefix feature The following processing is performed:

[0050] Step 4.1, the visual prefix feature sequence generated from step 3 The input sequence is constructed, where n represents the number of layers of visual features, n≤l, each represents the i-th layer of visual prefix feature, and each layer of the Transformer includes the calculation process of query (Q), key (K), and value (v):

[0051] Q (l) =X input W l Q , K (l) =X input W l K , V (l) =X input W l V

[0052] where W l Q , W l K , W l V is a learned linear transformation matrix, representing the weight parameters of the Transformer layer. X input is the visual feature sequence input to the Transformer, Q (l) is the query matrix, K (l) is the key matrix, and V (l) is the value matrix.

[0053] Step 4.2, the self-attention mechanism calculation of each layer of the Transformer aggregates information through the weighted relationship between the query (Q), key (K), and value (v):

[0054]

[0055] where d is the feature dimension; is used to scale the size of the inner product to avoid large calculation values.

[0056] Step 4.3, after the self-attention mechanism calculation, the feature Z (l) , after residual connection and layer normalization, is used as the input of the feedforward neural network for further nonlinear transformation:

[0057] FFN(Z(l) )=ReLU(Z (l) W1+b1)W2+b2

[0058] Wherein, W1, W2 are weight matrices of the feedforward network, respectively used for the first linear transformation and the second linear transformation, and b1 and b2 are bias terms of the feedforward network.

[0059] Step 4.4, the visual prefix feature extracted from the original image After aggregation and layer-by-layer enhancement and aggregation by the Transformer, the final visual feature h v is obtained.

[0060]

[0061] Further improvement of the application is that the step 5 specifically comprises the following steps:

[0062] Step 5.1, calculating the final text feature by a text encoder (Transformer)

[0063] h e =Text-Enconder{W1,W2,···,W m}

[0064] Wherein, d t is the dimension of the text, and Text-Enconder(·) represents that the text sequence is processed by a text encoder or a similar model to generate the text feature h e , which captures the semantic information of the text sequence.

[0065] Step 5.2, using a dynamic gate control strategy g' generated by a LeakyReLU activation function:

[0066]

[0067] Wherein, and are the key and value vector projections obtained by linear transformation of the text representation and the visual representation, respectively.

[0068] Step 5.3, using the dynamic gate control strategy g' to realize cross-modal information fusion, in the cross-modal fusion process, the text feature h e is used as the query (Q), and the final visual feature h v obtained in step 4 is used as the key (K) and the value (v), by calculating the similarity between the query (Q) and the key (K), the attention weight is generated, and the generated attention weight is applied to the final visual feature h v , so as to obtain the text-aware visual representation M:

[0069]

[0070] where Q is the query from the text, i.e. e the query vector after a proper linear transformation; K and V are the final visual features h v the vectors of keys and values after linear transformation, softmax(·) is used to calculate the attention weights, ensuring the weighted sum is 1;

[0071] Step 5.4, the final text feature h e is combined with the visual representation M based on the visual prefix of text perception through a weighted sum to generate the final cross-modal representation C:

[0072] C = h e + g' · M.

[0073] A further improvement of the present application is that in step 6, the text decoder is composed of multiple layers of transformers, and a third sublayer of the transformer is additionally inserted, which performs multi-head attention on the cross-modal C output by the dynamic gating strategy g', the previous state is:

[0074]

[0075] Based on the state The image and the final visual feature h v are decoded by linear projection and softmax function to generate the text sequence, which are features, and are dynamically integrated with the final text feature h e The text decoder automatically generates information extraction results in an autoregressive manner.

[0076] The beneficial effects of the present application are:

[0077] The present application shows its wide applicability for multi-modal information extraction.

[0078] The present application proposes a dynamic gating aggregation strategy d, which realizes hierarchical multi-scale visual features as fused visual prefixes, so as to better reason between the same and different channels, thereby obtaining text representations and image representations containing richer semantics.

[0079] The present application is a generative multi-modal information extraction method, in which the visual and text features are dynamically integrated, and the text decoder can automatically generate information extraction results in an autoregressive manner. BRIEF DESCRIPTION OF DRAWINGS

[0080] Fig. 1 is a schematic diagram of the multi-modal information extraction framework of the present application.

[0081] Fig. 2 is a flowchart of the multimodal information extraction of the present application. DETAILED DESCRIPTION

[0082] Embodiments of the present application will be described below with reference to drawings. Many practical details are described below in the following description in order to provide a thorough understanding of the present application. However, it will be apparent to those skilled in the art that these practical details are not intended to limit the present application. That is, these practical details are not necessary in some embodiments of the present application.

[0083] As shown in Figs. 1-2 , the present application is a generative multimodal information extraction method based on visual prefix, specifically comprising the following steps:

[0084] Step 1, starting from a visual backbone of a deep learning model, using images of Twitter-15, Twitter-17 and MNRE data sets, using a deep learning model for processing, the Transformer extracts hierarchical multi-scale visual features from the input image, wherein the preprocessing operation includes: normalizing the pixel value to [0, 1], randomly cropping the edge 10%-20%, randomly flipping horizontally or color jittering, the generated I is the normalized input tensor, keeping the same dimension as the input image.

[0085] In this step, the present application uses ResNet as the Transformer for processing images, and the images associated with the sentence maintain multiple visual objects related to the entities in the sentence, further providing more semantic knowledge to assist information extraction, the Transformer includes an input layer, an encoder and a decoder,

[0086] Input layer: adjust the input image to a standard size HxWxC, generate an input feature tensor I through preprocessing, i.e. normalization, random cropping:

[0087] I = Preprocess(I raw )

[0088] wherein I raw represents the original RGB input image with a size of HxWxC, and here an RGB image with a size of 224x224x3 is selected.

[0089] Encoder: the encoder includes an initial convolutional layer, a multi-resolution extraction module and a feature fusion layer, wherein,

[0090] Initial convolutional layer: use group convolution, kernel size kxk, channel number c1, improve non-linear representation ability through activation function GELU, specifically:

[0091] V init= GELU(Conv(I, k = 7, c1 = 64, s = 2))

[0092] wherein V init is the output tensor, s is the stride; the convolution kernel size k is selected as 7, the output channel c1 = 64, the stride s = 2, which is used for down-sampling, and the size of the output tensor V init is 112x112x64.

[0093] The multi-resolution extraction module includes a multi-scale pyramid module and a channel attention module, each branch in the multi-scale pyramid module has a different convolution kernel size k1, k2, k3, and outputs features of different resolutions respectively, and the outputs of different branches are spliced through the channel dimension of the channel attention module to generate multi-scale features V ms

[0094] The channel attention module improves the representation ability of important region features after output:

[0095] V ms = Concat(Conv(V init , k1 = 3, c1 = 32), Conv(V init , k2 = 5, c2 = 32), Conv(V init , k3 = 7, c3 = 32))

[0096] The size of V ms is 112x112x96, and the multi-scale features V ms are used to extract multi-scale features and capture semantic information at different resolutions.

[0097] The feature fusion layer: the depthwise separable convolution of the feature fusion layer reduces the calculation overhead while maintaining the spatial resolution, and the output tensor feature V ∈ R H′×W′×C′ :

[0098] V = Attention(DepthwiseConv(V ms , k = 3, c2 = 128))

[0099] In the depthwise separable convolution, the convolution kernel k is 3, and the output channel c2 is 128; in the self-attention mechanism, matrix multiplication is used to extract global context information and enhance the global consistency of the features; the output tensor feature V has a size of 112x112x128.

[0100] The decoder includes a global context fusion module, which uses a self-attention mechanism to enhance the representation of the output tensor feature V and outputs hierarchical features Each layer corresponds to a specific scale, and the hierarchical features ​The middle embedding time step position coding forms a hierarchical multi-scale visual feature representation V1, V2, …, V4:

[0101] {V1, V2, V3, V4} = HierarchicalSplit(V)

[0102] The output tensor feature V is processed in layers to generate multi-scale outputs, and the resolution of each multi-scale visual feature decreases; V1: 112x112x64, V2: 56x56x128, V3: 28x28x256, and V4: 14x14x512.

[0103] Finally, global context enhancement is performed to obtain the context-aware output feature V context :

[0104] V context = Attention(GAP(V))

[0105] Wherein, GAP is global average pooling, which compresses the spatial dimension to generate a global feature vector R 128 . Attention is self-attention, which uses global features to enhance multi-scale visual features to obtain context-aware output features V context .

[0106] Step 2, use the dynamic gate module to predict the gating probability of each layer of multi-scale visual features The degree vector of the i-th multi-scale visual feature when executing the l-th layer is represented, and the value of the degree vector reflects the importance of each multi-scale visual feature to each layer in the Transformer. Specifically, the following steps are included:

[0107] Step 2.1, generate logits of gate signals

[0108]

[0109] Wherein, f(·) is the activation function ReLU, w i is the weight coefficient calculated for each output tensor feature V i , and P(V i ) is the feature after pooling of the i-th output tensor feature. Traditional pooling methods only average all regions of the feature map, which may lose some important spatial information. By introducing weighted pooling, different weights can be assigned according to the importance of each region, thereby retaining more detailed information.

[0110] Step 2.2, each output tensor feature V i will be given a weight w i according to its spatial position in the image;

[0111] Step 2.3, logits calculated based on step 2.1 The Softmax function is used to normalize, resulting in the gating probability of each layer

[0112]

[0113] This step is still to generate a probability distribution between [0, 1] from logits, which is used to control the contribution of each output tensor feature in the current Transformer.

[0114] Step 3, based on the dynamic gate module, derive the final aggregated hierarchical visual features V by multiplying the gating probability with the multi-scale visual features of each layer and connecting them g to match the l-th layer of the image encoder in the Transformer, resulting in the visual prefix feature Specifically, the following steps are included:

[0115] Step 3.1, the dynamic gate g derives the final aggregated visual prefix feature to match the l-th layer in the Transformer, denoted as:

[0116]

[0117] where, is the gating probability of each layer of visual prefix feature , indicating the weighted importance of the feature, is the visual feature from different scales of the l-th layer;

[0118] Step 3.2, formally, the visual prefix feature corresponding to the l-th layer of the Transformer is obtained by the following connection operation:

[0119]

[0120] where, is the fusion result of multi-scale visual features, is the visual prefix feature, the weighted and gated visual prefix feature is obtained by the dynamic gate module, and the visual prefix feature will be used to enhance the hierarchical representation of the text modality through the visual prefix-based attention mechanism.

[0121] Step 4, the visual prefix feature obtained in step 3 is input as the visual prefix, and the visual prefix feature Each layer of the Transformer takes as input the visual prefix feature of the input Step-by-step enhancement and conversion are performed to refine the fine image feature representation layer by layer, and the final visual feature h is obtained v , specifically comprising the following steps: given the input visual prefix feature The following processing is performed:

[0122] Step 4.1, the visual prefix feature sequence generated from step 3 forms an input sequence, where n represents the number of layers of visual features, n≤l, each represents the visual prefix feature of the i-th layer, and each layer of the Transformer includes the calculation process of query (Q), key (K), and value (v):

[0123] Q (l) =X input W l Q , K (l) =X input W l K , V (l) =X input W l V

[0124] where W l Q , W l K , W l V are learned linear transformation matrices, representing the weight parameters of the Transformer layer. X input is the visual feature sequence input to the Transformer, Q (l) is the query matrix, K (l) is the key matrix, and V (l) is the value matrix;

[0125] Step 4.2, the self-attention mechanism calculation of each layer of the Transformer aggregates information through the weighted relationship between the query (Q), key (K), and value (v):

[0126]

[0127] where d is the feature dimension; is used to scale the size of the inner product to avoid large calculation values.

[0128] Step 4.3, after the self-attention mechanism calculation, the feature Z (l) is input to the feedforward neural network after residual connection and layer normalization, and further nonlinear transformation is performed:

[0129] FFN(Z (l) )=ReLU(Z (l) W1+b1)W2+b2

[0130] where W1, W2 are weight matrices of the feed-forward network, respectively, for the first and second linear transformation, and b1 and b2 are bias terms of the feed-forward network;

[0131] Step 4.4, visual prefix features extracted from the original image Step 4.5, the final visual feature h v is obtained by aggregating and enhancing layer by layer through the Transformer

[0132]

[0133] This final feature representation will be the basis for image understanding tasks, providing a deep understanding of the image content.

[0134] Step 5, given the input text sequence W = {W1, W2, ···, W m}, m is the length of the text sequence, the final text feature h e is calculated by the text encoder, the dynamic gate control strategy g' is used to realize cross-modal information fusion, and the text-aware visual representation M is obtained, the final text feature h e is combined with the text-aware visual representation M to generate the final cross-modal C.

[0135] Specifically, the following steps are included:

[0136] Step 5.1, the final text feature h

[0137] h e =Text-Enconder{W1, W2, ···, W m}

[0138] where d t is the dimension of the text, and Text-Enconder(·) represents processing the text sequence through a text encoder or similar model to generate the text feature h e captures the semantic information of the text sequence;

[0139] Step 5.2, a dynamic gate control strategy g' generated by a LeakyReLU activation function is used:

[0140]

[0141] where, and are the key and value vector projections obtained by linear transformation of the textual and visual representations, respectively, and the LeakyReLU activation function is used to generate a non-linear gating signal g' that adjusts the weight of visual features during fusion.

[0142] Step 5.3, cross-modal information fusion is implemented using the dynamic gate control strategy g', during cross-modal fusion, the final visual feature h e is used as the query (Q), and the final visual feature h v obtained in step 4 is used as the key (K) and value (v). By calculating the similarity between the query (Q) and the key (K), the attention weight is generated, and the generated attention weight is applied to the final visual feature h v , thereby obtaining the text-aware visual representation M:

[0143]

[0144] where Q is the query from the text, i.e., h e is the query vector obtained by appropriate linear transformation; K and V are the key and value vectors obtained by linear transformation of the final visual feature h v , respectively, and softmax(·) is used to calculate the attention weight, ensuring that the weighted sum is 1.

[0145] Step 5.4, the final textual feature h e is combined with the text-aware visual representation M based on the visual prefix to generate the final cross-modal representation C through a weighted sum:

[0146] C = h e + g' · M.

[0147] where g' is the dynamic gating signal that controls the degree of fusion of text and visual information. In this way, the combination ratio of visual features and textual features can be dynamically adjusted, so that the final cross-modal representation can more accurately capture the semantic relationship between images and text.

[0148] where the weighting coefficient is controlled by the dynamic gate control strategy g':

[0149] Step 6, the text decoder generates the output structure in an autoregressive manner, encodes and decodes the cross-modal C obtained in step 5, and repeats step 6. In the i-th step, the text decoder represents the state conditioned on the final cross-modal C and the previous state .

[0150] In this step, the text decoder is composed of multiple layers of transformers, in which each layer usually includes two main sub-layers: a multi-head self-attention layer and a feed-forward network layer. The text decoder additionally inserts a third sub-layer of the transformer, which is an additional multi-head attention layer inserted in the standard transformer structure, which performs multi-head attention on the cross-modal C of the dynamic gating strategy g'.

[0151] Previous state is:

[0152]

[0153] Based on the state The image and the final visual feature h v are decoded by linear projection and softmax function, and the text sequence is generated. e The dynamic integration automatically generates the information extraction results by the text decoder.

[0154] The method of the present application is compared with HVPNeT, ITA and MoRe, and the results are shown in Table 1 below.

[0155] Table 1

[0156]

[0157] As can be seen from the above table, the present application achieves equal or better performance. In particular, there are very high results compared with the best model in each data set, which indicates the effectiveness and universality of the present method in processing various multi-modal information extraction tasks, and also proves the success of the proposed visual encoder and dynamic gating module.

[0158] The above description is only an embodiment of the present application and is not used to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A generative multimodal information extraction method based on visual prefixes, characterized in that: The generative multimodal information extraction method specifically includes the following steps: Step 1: Starting with a visualization backbone of a deep learning model, for images in a given dataset, the deep learning model is used to process them, extracting hierarchical, multi-scale visual features from the input images. Step 2: Use the dynamic gate module to predict the gating probability of multi-scale visual features at each layer. This represents the degree vector of the i-th multi-scale visual feature when the l-th layer is executed. The value of the degree vector reflects the importance of each multi-scale visual feature to each layer in the Transformer. Step 3: Based on the dynamic gate module, by controlling the gate probability... Multiplying the multi-scale visual features of each layer and concatenating them, yields the final aggregated hierarchical visual feature V. g By matching the l-th layer of the image encoder in the Transformer, visual prefix features are obtained. Step 4: Apply the visual prefix features obtained in Step 3. As visual prefix input, and visual prefix features are used as input. The visual prefix features of the input are processed by a self-attention mechanism and a feedforward neural network at each layer of the Transformer. Through progressive enhancement and transformation, a refined image feature representation is extracted layer by layer to obtain the final visual feature h. v ; Step 5: Given a sequence of input text W = {W1, W2, ..., W...} m }, where m is the length of the text sequence, and the final text features h are calculated using a text encoder. e Using dynamic gate control strategy g ′ Achieving cross-modal information fusion to obtain the visual representation M of text perception, and then combining the final text features h e Combining the text-aware visual representation M with the final cross-modal representation C, the process includes the following steps: Step 5.1: Calculate the final text features using a text encoder (Transformer). h e =Text-Enconder{W1,W2,···,W m } Where, d t Let h be the dimension of the text. Text-Encoder(·) represents the text feature h generated by processing the text sequence through a text encoder. e It captures the semantic information of the text sequence; Step 5.2: Use the dynamic gate control strategy g generated by a LeakyReLU activation function. ′ : in, and These are the key and value vector projections obtained from the text and visual representations through linear transformations, respectively. Step 5.3: Use the dynamic gate control strategy g ′ To achieve cross-modal information fusion, in the process of cross-modal fusion, text features h e Used as query (Q), the final visual feature h obtained in step 4 v Using the query (Q) as the key (K) and the value (v), attention weights are generated by calculating the similarity between the query (Q) and the key (K), and these attention weights are then applied to the final visual feature h. v Thus, the visual representation M of text perception is obtained: Where Q is the query from the text, and K and V are the final visual features h, respectively. v The vector of keys and values ​​obtained after linear transformation is used to calculate attention weights using softmax(·), ensuring that the weighted sum is 1. Step 5.4: Transfer the final text features h e The visual representation M, which is based on visual prefixes and is perceived by text, is combined with a weighted sum to generate the final cross-modal representation C: C=h e +g ′ ·M; Step 6: The text decoder generates the output structure in an autoregressive manner, encoding and decoding the cross-modal C obtained in Step 5. Step 6 is repeated. In step i, the text decoder represents the final cross-modal C and the previous state. The state of being conditional.

2. The generative multimodal information extraction method based on visual prefixes according to claim 1, characterized in that: In step 1, ResNet is used as the Transformer for image processing. The Transformer includes an input layer, an encoder, and a decoder. Input layer: The input image is resized to standard dimensions H×W×C and preprocessed to generate the input feature tensor I. I=Preprocess(I raw ) Among them, I raw This represents the original RGB input image with dimensions H×W×C; Encoder: The encoder consists of an initial convolutional layer, a multi-resolution extraction module, and a feature fusion layer. Initial convolutional layer: Uses grouped convolution with a kernel size of k×k and c1 channels. The GELU activation function is used to improve nonlinear representation capabilities. Specifically: In init =GELU(Conv(I,k=7,c1=64,s=2)) Among them, V init The output tensor is s, where s is the stride. The multi-resolution extraction module includes a multi-scale pyramid module and a channel attention module. Each branch in the multi-scale pyramid module has different convolutional kernel sizes k1, k2, and k3, outputting features at different resolutions. The outputs of different branches are concatenated using the channel dimensions of the channel attention module to generate a multi-scale feature V. ms Used to extract multi-scale features and capture semantic information at different resolutions: In ms =Concat(Conv(V init ,k1=3,c1=32),Conv(V init ,k2=5,c2=32),Conv(V init ,k3 ＝7,c3＝32)) Feature fusion layer: The depthwise separable convolution of the feature fusion layer reduces computational overhead while maintaining spatial resolution, and outputs tensor features V∈R. H′×W′×C′ : V=Attention(DepthwiseConv(V ms ,k=3,c2=128)) In depthwise separable convolution, the kernel k is 3 and the output channel c2 is 128. In the self-attention mechanism, matrix multiplication is used to extract global contextual information and enhance the global consistency of features. Decoder: Includes a global context fusion module, which utilizes a self-attention mechanism to enhance the output tensor feature V representation, outputting hierarchical features. Each layer corresponds to a specific scale, with layered features. Embedding time step position encoding forms a hierarchical multi-scale visual feature representation V1, V2, ..., V n : {V1,V2,V3,V n }=HierarchicalSplit(V) The output tensor feature V is processed in layers to generate multi-scale output, with the resolution of each multi-scale visual feature decreasing. Finally, global context enhancement is performed to obtain the context-aware output feature V. context : V context =Attention(GAP(V)) Here, GAP stands for Global Average Pooling, and Attention stands for Self Attention.

3. The generative multimodal information extraction method based on visual prefixes according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Generate the logits of the gate signal. Where f(·) is the activation function ReLU, w i For each output tensor feature V i The calculated weighting coefficients, P(V) i ) is the feature obtained by pooling the i-th output tensor feature; Step 2.2, each output tensor feature V i It will be assigned a weight w based on its spatial location in the image. i ; Step 2.3: Based on the logits calculated in Step 2.1 The Softmax function is used for normalization to obtain the gating probability of each layer. A probability distribution between [0,1] is generated based on logits, which is used to control the contribution of each output tensor feature in the current Transformer.

4. The generative multimodal information extraction method based on visual prefixes according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1: The dynamic gate g derives the final aggregated visual prefix features. To match the l-th layer in a Transformer, it is represented as: in, For each layer of visual prefix features The gating probability, These are visual features from different scales in the l-th layer; Step 3.2: Visual prefix features corresponding to the l-th layer of the Transformer Obtained through the following connection operation: in, It is the result of the fusion of multi-scale visual features. It is a visual prefix feature.

5. The generative multimodal information extraction method based on visual prefixes according to claim 1, characterized in that: In step 4, given the visual prefix features of the input Perform the following processing: Step 4.1: Visual prefix feature sequence generated in Step 3 The input sequence is formed, where n represents the number of visual feature layers, n≤l, and each layer represents a visual prefix feature of the i-th layer. Each layer of the Transformer includes the calculation process of query (Q), key (K), and value (v): Q (l) =X input W l Q ,K (l) =X input W l K ,V (l) =X input W l V Among them, W l Q W l K W l V X is a linear transformation matrix, representing the weight parameters of the Transformer layer. input Given the visual feature sequence input to the Transformer, Q (l) For querying the matrix, K (l) V is the key matrix. (l) It is a value matrix; Step 4.2: The self-attention mechanism of each layer of the Transformer calculates and aggregates information by weighting the relationships between queries (Q), keys (K), and values ​​(v). Where d is the feature dimension; Step 4.3: Calculate the feature Z after the self-attention mechanism. (l) After residual connections and layer normalization, the result is used as input to the feedforward neural network for nonlinear transformation: FFN(Z (l) )=ReLU(Z (l) W1+b1)W2+b2 Where W1 and W2 are the weight matrices of the feedforward network, respectively, used for the first linear transformation and the second linear transformation, and b1 and b2 are the bias terms of the feedforward network. Step 4.4: Visual prefix features extracted from the original image The features are aggregated and then enhanced and aggregated layer by layer using a Transformer to obtain the final visual features h. v :

6. The generative multimodal information extraction method based on visual prefixes according to claim 1, characterized in that: In step 6, the text decoder consists of multiple transformers, with an additional third sub-layer inserted into the Transformer. This sub-layer controls the dynamic gate control strategy g. ′ The output of cross-modal C performs multi-head attention, prior to the state. for: Based on state The text sequence is decoded using linear projection and the softmax function, and the image and final visual features h are then analyzed. v These features are used as characteristics, and the final text features h are used to define them. e Dynamic integration: The text decoder automatically regresses to generate information extraction results.

Images