Information extraction method, device and program product for image and text feature alignment and fusion

Abstract

The present invention provides an information extraction method, device and program product for image-text feature alignment and fusion. The method comprises: inputting a target image into an image feature extraction model to obtain image features; detecting and identifying text blocks in the target image to obtain text block recognition results; inputting the text block recognition results into a text feature extraction model to obtain text features; constructing text block features based on the text block recognition results, and bridging and aligning the text features with image features; inputting the text features and image features into a feature fusion model for splicing to obtain a comprehensive feature vector; the feature fusion model comprises an autoencoder and a transformer encoder; classifying the comprehensive feature vector into a preset classifier, performing a semantic entity recognition task, determining the classification results corresponding to the text blocks, and performing a relationship extraction task based on the classification results; the method can solve the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, device, and program product for information extraction by aligning and fusing image and text features. Background Art

[0002] With the development of artificial intelligence technology, image processing and natural language processing (NLP) technologies have made significant progress. Among numerous application areas, key information extraction has attracted much attention due to its ability to simultaneously process and understand image and text information. This refers to extracting and presenting text blocks with key information connections from an image containing text content. This is of great value in providing richer and more accurate user interaction, especially in scenarios requiring the processing of complex image content and diverse text information.

[0003] However, existing key information extraction methods face numerous challenges, such as inaccuracy in image feature extraction, inaccuracy in text feature extraction, difficulty in aligning image and text features, low feature fusion efficiency, and slow system response speed. These challenges are particularly pronounced when processing images with complex backgrounds and multiple elements, leading to inaccurate information extraction. Therefore, there is an urgent need to develop a novel key information extraction method that can process image and text data more accurately, efficiently, and quickly, thereby improving the overall performance of multimodal data processing. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an information extraction method, apparatus, and program product for image-text feature alignment and fusion, to eliminate or improve one or more defects existing in the prior art. It can solve the problem of inaccurate information extraction when processing images containing complex backgrounds and multiple elements.

[0005] One aspect of the present invention provides an information extraction method for image-text feature alignment and fusion, the method comprising the following steps:

[0006] The target image is input into a pre-trained image feature extraction model to obtain image features. The image feature extraction model is a residual learning framework model, which includes a convolutional layer module, and the convolutional layer module includes several convolutional layers connected in sequence.

[0007] Detect and identify text blocks in the target image to obtain text block recognition results;

[0008] The text block recognition results are input into a pre-trained text feature extraction model to obtain text features;

[0009] Text block features are constructed based on the text block recognition results, and the text features are bridged and aligned with image features;

[0010] A pre-trained feature fusion model is obtained. Text features and image features are input into the feature fusion model and concatenated to obtain a comprehensive feature vector. The feature fusion model includes an autoencoder and a transformer encoder. The autoencoder is used to perform dimensionality unification processing on the input features. The transformer encoder includes several stacked transformer encoder layers, which are connected by residuals. Each transformer encoder layer contains a multi-head self-attention mechanism, which is used to concatenate the features after dimensionality unification processing.

[0011] The comprehensive feature vector classification is input into a preset classifier to perform a semantic entity recognition task, and the classification result corresponding to the text block is determined, including question category, answer category or other category;

[0012] Based on the classification results, a relation extraction task is performed to determine the connection relationship between each pair of question-answer sample pairs in the question classification and answer classification.

[0013] Optionally, image features include image semantic features and image spatial features; text block features include text block visual features and text block index features;

[0014] For semantic recognition tasks, the visual features of text blocks include the ratio of the x-coordinate of the center point of the text block to the width of the target image, the ratio of the y-coordinate of the center point of the text block to the height of the target image, the ratio of the height of the text block to the height of the target image, and the ratio of the width of the text block to the width of the target image; the index features of text blocks include the order in which the text blocks are arranged from top to bottom and then from left to right in all text blocks of the target image, and the total number of all text blocks.

[0015] For the relation extraction task, a question-answer sample pair includes a question category text block and an answer category text block. The visual features of the text blocks also include the ratio of the absolute value between the x-coordinates of the center points of the question category text block and the answer category text block to the width of the target image, the ratio of the absolute value to the height of the target image, and the positional relationship between the question category text block and the answer category text block. The text block index features also include the difference between the sequence number of the question category text block and the sequence number of the answer category text block.

[0016] Optionally, the autoencoder includes a first autoencoder corresponding to image semantic features, a second autoencoder corresponding to image spatial features, a third autoencoder corresponding to text block visual features, a fourth autoencoder corresponding to text block index features, and a fifth autoencoder corresponding to text features.

[0017] Optionally, before obtaining a pre-trained feature fusion model and concatenating text and image features into it to obtain a comprehensive feature vector, the process further includes: training the parameters of the autoencoder using a preset reconstruction loss function; the preset reconstruction loss function is expressed by the following formula:

[0018]

[0019] In the formula, n is the total number of samples; y i y' represents the i-th input sample of the autoencoder; i This represents the i-th output sample of the autoencoder.

[0020] Optionally, before inputting the comprehensive feature vector classification into the preset classifier, the method further includes: training the parameters of the preset classifier using a preset classification loss function; the preset classification loss function is expressed by the following formula:

[0021]

[0022] In the formula, N represents the total number of training samples; L I y is the loss value of the i-th training sample; M represents the number of categories; ic The value can be either 0 or 1; it takes the value 1 when the class of training sample i is c, and 0 otherwise. ic This represents the probability that the predicted training sample i belongs to class c.

[0023] Optionally, the image feature extraction model also includes a feature pyramid network module, which includes several feature extraction network branches. Each feature extraction network branch is connected to a convolutional layer and is used to receive the output of the corresponding convolutional layer as input and output image features.

[0024] Optionally, each feature extraction network branch includes a two-dimensional convolutional layer, a spatial attention layer, and a global average pooling layer.

[0025] Another aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the electronic device implements the steps of the aforementioned image-text feature alignment and fusion information extraction method.

[0026] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the aforementioned image-text feature alignment and fusion information extraction method.

[0027] Another aspect of the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the aforementioned image-text feature alignment and fusion information extraction method.

[0028] The image-text feature alignment and fusion information extraction method, device, and program products of this invention can solve the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements. They extract image features from target images using an image feature extraction model with a residual learning framework. This model includes several sequentially connected convolutional layers. By introducing residual learning, the image feature extraction model effectively solves the degradation problem in deep networks, making model training more efficient. Furthermore, its deep network structure improves the ability to extract image features from complex images, thus enhancing the accuracy of information extraction. Simultaneously, by identifying text block features, image features and text features are bridged and aligned, strengthening the semantic association between image and text features, and better aligning features of different modalities, further improving the accuracy of information extraction.

[0029] Furthermore, by integrating a feature pyramid network structure and a spatial attention mechanism, as well as applying a smoothing layer, the image feature extraction model can more effectively process complex image content and provide richer and more accurate image features for key information extraction. This enhances the ability to identify small targets in the target image and optimizes the extraction of spatial features, thereby improving the accuracy of image feature extraction and thus the accuracy of information extraction. At the same time, the spatial attention layer enhances the model's attention to specific regions by weighting the fused feature map at the pixel level, improving the overall recognition ability of the image feature extraction model for spatial features.

[0030] In addition, by introducing image modalities, the image feature extraction model can distinguish target images of different styles, improve the accuracy of distinguishing text blocks of different styles of text content on the same target image, and thus improve the accuracy of information extraction.

[0031] Furthermore, the text feature extraction model trained based on the ELECTRA encoder includes a generator and a discriminator. The generator focuses on predicting mask markers in the input, while the discriminator determines whether the markers have been replaced by the generator. This enables the ELECTRA encoder to more accurately identify and classify markers when processing text data, thereby improving the performance and efficiency of the text feature extraction model. At the same time, the discriminator of the ELECTRA encoder not only needs to understand the original vocabulary but also needs to identify which words have been replaced by the generator. This promotes a broader understanding of the entire vocabulary by the text feature extraction model, not just the mask part. Therefore, it can enhance the breadth and depth of vocabulary learning.

[0032] In addition, by extracting the core information of each modal feature through an autoencoder, the dimensionality of each feature is unified, reducing the computational burden caused by dimensionality. At the same time, modal feature vectors of the same length ensure that they have the same initial importance in subsequent transformer encoders, allowing the transformer encoder to learn the importance of different modes on its own. This improves the feature fusion model's ability to process complex data, enhances its understanding and application of different types of data, and thus improves the performance of the feature fusion model.

[0033] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0034] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0036] Figure 1 This is a schematic diagram of the structure of a convolutional layer module in an image feature pre-extraction model provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the residual structure in an image feature pre-extraction model provided in an embodiment of the present invention;

[0038] Figure 3 A flowchart of an information extraction method for image-text feature alignment and fusion provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the structure of an image feature pre-extraction model provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the structure of a text feature extraction model provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of the structure of a feature fusion model provided in an embodiment of the present invention;

[0042] Figure 7 This is a schematic diagram of the structure of the autoencoder in a feature fusion model provided in an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of the transformer encoder layer in a feature fusion model provided in an embodiment of the present invention;

[0044] Figure 9 This is a block diagram of an information extraction device for image-text feature alignment and fusion according to an embodiment of the present invention;

[0045] Figure 10 A block diagram of an electronic device provided in yet another embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0047] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0048] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0049] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0050] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0051] The terms involved in this invention will be explained below.

[0052] Residual Network 50 (ResNet50) model: As an advanced deep convolutional neural network, it has been widely used in the field of image recognition and processing. Its deep network structure can effectively extract high-level features of images, which is particularly important for understanding complex image content.

[0053] The core innovation of ResNet50 lies in its residual learning framework. In deep networks, the learning objective of each layer can be represented as H(x), where x is the input of that layer. Layers in traditional networks typically attempt to learn this objective function directly. However, the ResNet50 model changes this by having each layer learn a residual function F(x) = H(x) - x. Therefore, the original learning objective can be redefined as H(x) = F(x) + x. This residual structure can be expressed by the formula y = F(x, {W i The expression is F(x,{W)+x. Here, y is the output of the layer, and F(x,{W)+x is the expression. i}) indicates that it has weight W i The residual mapping function. By introducing residual learning, the ResNet50 model effectively solves the degradation problem in deep networks, making network training more efficient. Furthermore, its deep network structure enables the ResNet50 model to excel in extracting complex image features. The structure of ResNet50 and its internal residual structure are as follows: Figure 1 and Figure 2 As shown, for a 224×224×3 input image, the ResNet50 model extracts features layer by layer to obtain a 7×7×2048 feature map. Then, through global average pooling at the feature channel level, a 2048-dimensional image modality feature vector is obtained.

[0054] Feature Pyramid Network (FPN) is an improved feature extraction network that significantly enhances the ability to identify small targets by combining bottom-up and top-down feature flow mechanisms.

[0055] ELECTRA encoder (Efficiently Learning an Encoder that Classifies TokenReplacements Accurately):

[0056] The following section provides a detailed description of the information extraction method for image-text feature alignment and fusion provided in this application.

[0057] Optionally, the execution subject of the image-text feature alignment and fusion information extraction method provided in this application is an electronic device, which can be a terminal such as a computer, mobile phone, or tablet computer, or it can be a server. This embodiment does not limit the implementation method of the electronic device.

[0058] This embodiment provides an information extraction method for image-text feature alignment and fusion, such as... Figure 3 As shown, the information extraction method for image-text feature alignment and fusion includes at least steps S101 to S107:

[0059] Step S101: Input the target image into a pre-trained image feature extraction model to obtain image features.

[0060] The image feature extraction model is a residual learning framework model, which includes a convolutional layer module, and the convolutional layer module includes several convolutional layers connected in sequence; the target image is a document image containing text blocks, including images obtained by scanning documents or images obtained by cameras, etc.

[0061] like Figure 1 As shown, optionally, the convolutional layer module includes five sequentially connected convolutional layers. In actual implementation, the number of convolutional layers in the convolutional layer module can be adjusted according to the actual situation, such as four sequentially connected convolutional layers, six sequentially connected convolutional layers, etc. This embodiment does not limit the number of convolutional layers.

[0062] In this embodiment, the image feature extraction model is a ResNet50-Spatial FPN (ResNet50-SF) model that incorporates a spatial attention mechanism and a Feature Pyramid Network (FPN).

[0063] The Residual Network 50 (ResNet50) model is a deep convolutional neural network that has been widely used in image recognition and processing. However, while ResNet50 performs well in image feature extraction, it has limitations in some aspects. For example, its efficiency in recognizing small targets is relatively low because ResNet50 tends to extract salient features, which may not be ideal when processing small targets in images. Furthermore, its spatial feature extraction performance may also be limited when processing images with complex spatial structures. Although ResNet50 is an effective choice as the basis for image modalities in key information extraction systems, in specific application scenarios such as small target detection and image processing with complex spatial structures, high-level features, while semantically rich, have low spatial resolution, which is not conducive to the accurate localization of small targets. Conversely, low-level features, although having high spatial resolution, lack sufficient semantic information to distinguish small targets in complex backgrounds. Therefore, further optimization or combination with other techniques is needed to improve its performance.

[0064] In this embodiment, the target image is a document image containing text blocks, which are often small in size and distributed in specific areas of the document image. The ResNet50 model has limitations in feature extraction for such images.

[0065] To address the aforementioned technical challenges, this embodiment further optimizes the ResNet50 model by combining a feature pyramid network structure and a spatial attention mechanism. By integrating the feature pyramid network structure and spatial attention mechanism, and applying a smoothing layer, the image feature extraction model can more effectively process complex target images and provide richer and more accurate image features for extracting key information.

[0066] The feature pyramid network integrates features across different levels, combining high-level semantic information with low-level detail. This integration ensures that even small targets retain sufficient semantic information and spatial detail in the feature map. Simultaneously, a new top-down path propagates deep, semantically rich features downwards, enhancing the semantic expressiveness of low-level features. This flow of information from high-semantic information to low-semantic information but high-resolution feature maps makes the model more sensitive to small targets.

[0067] Furthermore, the smoothing layers integrated into the feature pyramid network further enhance the visual quality of the fused features. These smoothing layers ensure that the feature maps maintain high visual continuity and consistency after fusing features at different scales, which is particularly crucial for highlighting small target features.

[0068] Spatial attention layers, integrated into each level of the feature pyramid network, help image feature extraction models focus on specific key regions in target images, especially small but important targets. By weighting the fused feature maps at the pixel level, the spatial attention layers enhance the model's focus on specific regions. This not only improves the ability to locate small targets but also optimizes the overall spatial feature recognition capability of the image feature extraction model.

[0069] Specifically, refer to Figure 4 The image feature extraction model also includes a feature pyramid network module, which comprises several feature extraction network branches. Each feature extraction network branch includes a two-dimensional convolutional layer, a spatial attention layer, and a global average pooling layer. Each feature extraction network branch is connected to a convolutional layer and is used to receive the output of the corresponding convolutional layer as input to output image features.

[0070] For example, taking a convolutional layer module consisting of 5 sequentially connected convolutional layers as an example, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5 are each connected to a feature extraction network branch; the feature extraction network branches are connected sequentially as a 2D convolutional layer, a spatial attention layer, a 2D convolutional layer, a spatial attention layer, and a global average pooling layer; convolutional layer 1 takes a 224×224×3 target image as input, and features are extracted layer by layer by convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5 input the extracted features into the corresponding feature extraction network branches, and finally the global average pooling layer outputs the image features.

[0071] In addition, in this embodiment, by introducing image modalities, the image feature extraction model can distinguish target images of different styles, which lays the foundation for further accurate differentiation of text blocks with different styles of text content on the same target image and improves the accuracy of differentiating text blocks with different styles of text content on the same target image.

[0072] Step S102: Detect and identify text blocks in the target image to obtain text block recognition results.

[0073] In this embodiment, the text blocks in the target image are detected and identified by the Optical Character Recognition (OCR) model to ensure that the text content on the target image can be extracted completely and accurately.

[0074] In this embodiment, before detecting and recognizing text blocks in the target image and obtaining the text block recognition result, the method further includes: acquiring a training dataset, including sample images and corresponding sample target images with text blocks annotated; acquiring a preset neural network model; inputting the sample images into the preset neural network model to obtain sample text block recognition results; inputting the sample text block recognition results and the sample target images with text blocks annotated into a preset loss function to obtain loss function results; and iterating the preset neural network model based on the loss function results to obtain an optical character recognition model.

[0075] The preset neural network models include, but are not limited to, convolutional neural networks (CNN), recurrent neural networks (RNN), attention mechanisms, or transformers; the preset loss functions include, but are not limited to, connectionist temporal classification loss (CTC) or beam search loss (BeamSearch Loss).

[0076] Step S103: Input the text block recognition result into the pre-trained text feature extraction model to obtain text features.

[0077] In this embodiment, the text feature extraction model is based on a model trained using an ELECTRA encoder. The ELECTRA encoder includes a generator (G) and a discriminator (D), such as... Figure 5 As shown, the generator is a small masked language model (MLM) focused on predicting masked tokens in the input; the discriminator's task is to determine whether a token has been replaced by the generator. This unique structural design enables the ELECTRA encoder to more accurately identify and classify tokens when processing text data, thereby improving the model's performance and efficiency.

[0078] Traditional text feature extraction models are trained based on Bidirectional Encoder Representations from Transformers (BERT). However, BERT uses a masked language model (MLM) task, in which BERT randomly masks a subset of words (approximately 15%) of the input text and then attempts to predict these masked words. This approach only allows the model to learn from the small number of masked words, resulting in relatively little learning of non-masked tokens and thus slow training of text feature extraction models. Furthermore, in the process of recovering the masked words, BERT focuses on deeply learning the specific masked words, leading to limited breadth and depth of word learning.

[0079] To address the aforementioned technical issues, this embodiment utilizes a text feature extraction model trained using an ELECTRA encoder. The model structure of the text feature extraction model is as follows: Figure 6 As shown, the ELECTRA encoder uses a generator model to replace some input tokens and a discriminator model to determine whether each token has been replaced. This method is called "Replaced Token Detection" (RTD). The discriminator of the ELECTRA encoder participates in training when processing each input token, not just a few masked tokens. This makes the ELECTRA encoder more efficient than BERT when processing the same number of input tokens, thereby improving training efficiency and increasing the training speed of text feature extraction models. At the same time, the discriminator of the ELECTRA encoder not only needs to understand the original words, but also needs to identify which words have been replaced by the generator. This method promotes the model's broad understanding of the entire vocabulary, not just the masked part. Therefore, it can improve the breadth and depth of vocabulary learning.

[0080] Furthermore, the ELECTRA encoder's RTD task requires the discriminator to distinguish between real and fake tokens, which enhances the model's sensitivity to natural and unnatural (artificially generated) language patterns in the text. This ability is crucial for understanding and generating more natural and fluent text.

[0081] In this embodiment, the training of the ELECTRA encoder includes two phases: a pre-training phase on a large amount of text and a fine-tuning phase for a specific task. In the pre-training phase, firstly, unsupervised learning is used to pre-train the model on a large-scale text dataset. Secondly, by replacing a portion of the tokens in the original input text, the model is required to predict the replaced tokens. This replacement process is accomplished by sampling a replacement word pair. Through this replacement process, the model learns to distinguish between the real tokens and the replaced tokens, improving the model's generation and generalization abilities. In the fine-tuning phase, firstly, fine-tuning is performed on a labeled dataset for a specific task to obtain the target dataset. Secondly, supervised learning methods are used to fine-tune the model's parameters by performing backpropagation and gradient descent on the target dataset to maximize the model's performance on the specific task. This pre-training and fine-tuning approach enables the ELECTRA encoder to generate a 768-dimensional text feature vector for the input text string, effectively capturing and representing the complex text features of the text data.

[0082] During the training of the ELECTRA encoder, the generator aims to maximize the log-likelihood of mask positions in the input sequence. This approach enables the generator to learn semantic information in the text more effectively. The discriminator, on the other hand, focuses on distinguishing between the original and replaced tokens. This discriminator mechanism allows it to accurately identify subtle changes in the text, thereby improving the model's recognition capabilities.

[0083] In this embodiment, the training objective of the generator is represented by the following formula:

[0084] L G =E x～Data,t～Mask(x) [logP G (x t |x \t )]

[0085] In the formula, x represents the input sequence and t represents the masked mark position.

[0086] The training objective of the discriminator is expressed by the following formula:

[0087]

[0088] Where, x t It is the t-th label in the sequence, original t It is a binary tag representing xt Is it an original tag?

[0089] Step S104: Construct text block features based on the text block recognition results, and bridge and align the text features with the image features.

[0090] Because text blocks with specific content often have a specific shape and are distributed in specific locations within the target image, and text blocks with QA matching connections in the target image often have specific relative positions, orientations, and sequence numbers, this embodiment, in addition to the image features of the target image where the text block is located and the text features of the text content itself, also uses the text block features of the text block itself, including the position, shape, and sequence number of the text block, as well as the relative positions and sequence numbers between text blocks, as a bridge connecting image features and text features to better align different modal features.

[0091] Step S105: Obtain the pre-trained feature fusion model, input the text block features, text features and image features into the feature fusion model and concatenate them to obtain the comprehensive feature vector.

[0092] Since different features are often strongly correlated, it is particularly important to enable different features to fully interact and integrate, and learn their relationships.

[0093] For example, text blocks such as "name", "Zhang San", and "home address" (text features) often appear on the target image in the style of an ID card (image semantic features) and are distributed in specific locations (image spatial features, text block visual features). In the target image in the style of an ID card, all text blocks have a specific order (text block index features). The text blocks "name" and "Zhang San" with question-answer connection relationship are often adjacent in position (image spatial features, text block visual features, text block index features).

[0094] Alternatively, text blocks such as "unit price", "¥10.00", and "tax amount" (text features) often appear on the target image of the invoice format (image semantic features) and are distributed in specific locations (image spatial features, text block visual features). In the target image of the invoice format, all text blocks have a specific order (text block index features), and the fields "unit price" and "¥10.00" with question-answer connection relationships are often adjacent in position (image spatial features, text block visual features, text block index features).

[0095] Simultaneously, a feature fusion model is used to fuse and combine text block features, text features, and image features. While high-information-density text features, image features, and text block features provide rich information, this can also bring challenges in feature interaction, alignment, and fusion. Therefore, to enable the feature fusion model to better learn the correlations between different features and to perform more efficient feature interaction and fusion, image features are further subdivided into 1024-dimensional image semantic features and 3136-dimensional image spatial features, and text block features are further subdivided into text block visual features and text block index features. That is, image features include image semantic features and image spatial features; text block features include text block visual features and text block index features. Image semantic features represent the overall semantic information of the target image, while image spatial features contain fine-grained features at the pixel level of the target image.

[0096] Image features can be represented as:

[0097] Image Feature=Image Semantic Feature 1×1024 +Image Spatial Feature 1×3136

[0098] In the formula, Image Feature represents image features; Image Semantic Feature 1×1024 Represents 1024-dimensional image semantic features; Image Spatial Feature 1×3136 This represents the spatial features of a 3136-dimensional image.

[0099] refer to Figure 4 In image feature extraction models, feature rearrangement is a crucial step used to extract and separate semantic and spatial features to better support the extraction of key information. The image semantic features consist of four feature vectors i1, i2, i3, and i4, which have undergone top-down information flow in the image feature extraction model and have been smoothed and processed using spatial attention mechanisms. Each layer has 256 channels, and global average pooling is performed at the channel level before concatenation to form a 1024-dimensional image semantic feature vector. This can be expressed by the following formula:

[0100] Image Semantic Feature 1×1024 =Concatenate(

[0101] Feature 1 1×256 Feature2 1×256 Feature3 1×256 Feature4 1×256 )

[0102] In the formula, Image Semantic Feature 1×1024 Represents 1024-dimensional image semantic features;

[0103] Feature 1 1×256 Characteristic 1 refers to Global Average Pooling on channel (i1 56×56×256 ), where i1 56×56×256 Features received by the global average pooling layer;

[0104] Feature2 1×256 Feature 2 refers to Global Average Pooling on channel (i2 56×56×256 ), where i2 56×56×256 Features received by the global average pooling layer;

[0105] Feature 3 1×256 Feature 3 refers to Global Average Pooling on channel (i3 56×56×256 ), where i3 56×56×256 Features received by the global average pooling layer;

[0106] Feature 4 1×256 Feature 4, namely Global Average Pooling on channel (i4 56×56×256 ), where i4 56×56×256 These are the features received by the global average pooling layer.

[0107] The image spatial features are the lowest-level 56×56×256 feature maps of the image feature extraction model. This feature map contains the richest spatial information. After being added to a learnable location encoding vector of the same size, global average pooling is performed at each pixel level to convert the feature map size to 56×56. After flattening, a 3136-dimensional feature vector is obtained, which can be expressed by the following formula:

[0108] Image Spatial Feature 1×3136 =Flatten(Global Average Pooling on pixel (i1 56×56×256 +

[0109] Position Encoding 56×56×256 ))

[0110] In the formula, Image Spatial Feature 1×3136 Represents 3136-dimensional image spatial features; i1 56×56×256 Features received by the global average pooling layer; Position Encoding 56×56×256 This represents the positional encoding vector.

[0111] Text block features are further subdivided into text block visual features and text block index features, which can be represented by the following formula:

[0112] Text Block Feature =

[0113] Text Block Visual Feature 4(for SER),8(for RE) +Text Block Index Feature 2(for SER),4(for RE)

[0114] In the formula, Text Block Feature represents text block features; Text Block Visual Feature... 4(for SER),8(for RE) This means that for semantic entity recognition tasks, the visual features of text blocks are 4-dimensional, while for relation extraction tasks, the visual features of text blocks are 8-dimensional.

[0115] Text Block Index Feature 2(for SER),4(for RE) This means that for semantic entity recognition tasks, text block index features are 2-dimensional, while for relation extraction tasks, text block index features are 4-dimensional.

[0116]

[0117] Specifically, for semantic entity recognition tasks, the classification results obtained include question category, answer category, or other categories. The visual features of the text block include the ratio of the horizontal coordinate of the text block's center point to the width of the target image, the ratio of the vertical coordinate of the text block's center point to the height of the target image, the ratio of the height of the text block to the height of the target image, and the ratio of the width of the text block to the width of the target image, which can be expressed by the following formula:

[0118]

[0119] In the formula, This represents the ratio of the x-coordinate of the center point of the text block to the width of the target image. This represents the ratio of the ordinate of the center point of the text block to the height of the target image. This represents the ratio of the height of the text block to the height of the target image. This represents the ratio of the width of the text block to the width of the target image.

[0120] The text block index features include the sequence number of the text block in all text blocks of the target image, arranged from top to bottom and then from left to right, and the total number of text blocks corresponding to it. This can be represented by the following formula:

[0121] Block Index Feature=(index,total_blocks)

[0122] In the formula, index represents the sequence number of the text block in the target image, arranged from top to bottom and then from left to right; total_blocks represents the total number of text blocks.

[0123] For the relation extraction task, the connection relationship between each pair of question-answer sample pairs in question classification and answer classification is determined. Each pair of question-answer sample pairs includes a question category text block and an answer category text block. Accordingly, in addition to the ratio of the x-coordinate of the center point of the question category text block to the width of the target image, the ratio of the y-coordinate of the center point of the question category text block to the height of the target image, the ratio of the x-coordinate of the center point of the answer category text block to the width of the target image, and the ratio of the y-coordinate of the center point of the answer category text block to the height of the target image, the visual features of the text blocks also include the ratio of the absolute value of the x-coordinate of the center point of the question category text block and the center point of the answer category text block to the width of the target image, the ratio of the absolute value of the x-coordinate of the center point of the question category text block to the height of the target image, and the positional relationship between the question category text block and the answer category text block, including the vertical positional relationship and the horizontal positional relationship.

[0124] In addition to the order of question-type text blocks in the target image from top to bottom and then from left to right, and the order of answer-type text blocks in the target image from top to bottom and then from left to right, and the corresponding number of all text blocks, the text block index features also include the difference between the order of question-type text blocks and the order of answer-type text blocks.

[0125] In this embodiment, feature rearrangement distinguishes features with different functions, which helps the subsequent feature fusion model learn the interactions of different features. Through the above optimization, it is possible to handle small targets and complex spatial structures more effectively and achieve more efficient feature fusion. This improved framework provides stronger support for processing various types of multimodal data and is an important supplement and improvement to existing technologies.

[0126] In this embodiment, the feature fusion model includes an autoencoder and a transformer encoder.

[0127] Among them, the autoencoder is used to unify the dimensions of the input features, transforming the dimensions of the input features to 512 dimensions; for example... Figure 5 As shown, the autoencoder includes an autoencoder corresponding to image semantic features, an autoencoder corresponding to image spatial features, an autoencoder corresponding to text block visual features, an autoencoder corresponding to text block index features, and an autoencoder corresponding to text features.

[0128] Specifically, the autoencoder corresponding to the image semantic features is used to transform the 1024-dimensional image semantic features into 312-dimensional encoded pixel features, as expressed by the following formula:

[0129] Encoded Image Semantic Feature=AutoEncoder image semantic (F image semantic )

[0130] In the formula, Encoded Image Semantic Feature represents the encoded pixel feature; AutoEncoder image semantic F represents the autoencoder corresponding to the semantic features of the image; image semantic Represents the semantic features of an image.

[0131] The autoencoder corresponding to the image spatial features is used to transform the 3136-dimensional image spatial features into 512-dimensional encoded pixel features, as expressed by the following formula:

[0132] Encoded Image Spatial Feature=AutoEncoder image spatial (F image spatial )

[0133] In the formula, Encoded Image Spatial Feature represents the encoded semantic feature; AutoEncoder image spatial F represents the autoencoder corresponding to the spatial features of the image; image spatial Represents the spatial features of an image.

[0134] The autoencoder corresponding to the visual features of text blocks is used to transform 4- or 8-dimensional visual features of text blocks into 512-dimensional encoded visual features of text blocks, expressed by the following formula:

[0135] Encoded Text Block Visual Feature=AutoEncoder text block visual (F text block visual )

[0136] In the formula, Encoded Text Block Visual Feature represents the visual features of the encoded text block; AutoEncoder text block visual F represents the autoencoder corresponding to the visual features of the text block; text block visual Represents the visual features of text blocks.

[0137] The autoencoder corresponding to the text block index features is used to transform 2- or 4-dimensional text block index features into 512-dimensional encoded text block index features, expressed by the following formula:

[0138] Encoded Text Block Index Feature=AutoEncoder text block index (F text block index )

[0139] In the formula, Encoded Text Block Index Feature represents the index feature of the encoded text block; AutoEncoder text block index F represents the autoencoder corresponding to the text block index feature; text block index This indicates the characteristics of a text block index.

[0140] The autoencoder corresponding to the text features is used to transform the 768-dimensional text features into 512-dimensional encoded text features, expressed by the following formula:

[0141] Encoded Text Feature=AutoEncoder text (F text )

[0142] In the formula, Encoded Text Feature represents the encoded text feature; AutoEncoder text F represents the autoencoder corresponding to the text features. text Represents text features.

[0143] In this embodiment, the purpose of the autoencoder is to extract the core information of each modal feature, reducing the computational burden caused by dimensionality. Simultaneously, modal feature vectors of the same length ensure that they have the same initial importance in subsequent transformer encoders, allowing the transformer encoder to learn the importance of different modes on its own. The structure of the autoencoder is as follows: Figure 7 As shown.

[0144] In this embodiment, the structure of the transformer encoder is as follows: Figure 8 As shown, the transformer encoder comprises several stacked transformer encoder layers, connected by residuals. Each transformer encoder layer includes a multi-head self-attention mechanism to concatenate the features after dimensionality unification processing, resulting in a comprehensive feature vector. The input X... embedding After multiple stacking transformations of the residual connections through a multi-head self-attention mechanism, the output X is obtained. hidden The process is as follows Figure 7 As shown.

[0145] The comprehensive feature vector is represented by the following formula:

[0146] Integrated Feature=Concat(Encoded Text Feature, Encoded ImageSemantic Feature, Encoded Image Spatial Feature, Encoded Text Block VisualFeature, Encoded Text Block Index Feature)

[0147] In the formula, Integrated Feature represents the integrated feature vector; Encoded Text Feature represents the encoded text feature; Encoded Image Semantic Feature represents the encoded pixel feature; Encoded Image Spatial Feature represents the encoded semantic feature; Encoded Text Block Visual Feature represents the encoded text block visual feature; and Encoded Text Block Index Feature represents the encoded text block index feature.

[0148] In this embodiment, the transformer encoder achieves interactive learning between input features through a series of stacked transformer encoder layers. Each transformer encoder layer contains a multi-head self-attention mechanism, which can process information from different subspaces in parallel, thereby capturing subtle correlations between input features. After the multi-head self-attention mechanism, a series of linear transformations and normalization processes are performed to further refine the feature representation.

[0149] In this embodiment, each transformer encoder layer employs residual connections, which alleviates the gradient vanishing problem in deep networks and ensures effective information transfer within the network's deeper layers. In this way, each transformer encoder layer not only learns new feature representations but also retains information from the previous layer, enhancing the feature integration capability. After processing through multiple transformer encoder layers, the input features are effectively fused, forming a comprehensive feature vector representation with stronger overall performance.

[0150] Through the aforementioned fusion mechanism, efficient integration of features from different modalities is achieved, providing a robust feature representation foundation for the key information extraction system. This fusion approach not only improves the feature fusion model's ability to process complex data but also enhances its understanding and application of different types of data, thereby improving the model's performance.

[0151] In addition, before obtaining the pre-trained feature fusion model and concatenating the text and image features into it to obtain the comprehensive feature vector, the process also includes: training the parameters of the autoencoder using a preset reconstruction loss function; the preset reconstruction loss function is expressed by the following formula:

[0152]

[0153] In the formula, n is the total number of samples; y i y' represents the i-th input sample of the autoencoder; i This represents the i-th output sample of the autoencoder.

[0154] For example, when the autoencoder is an autoencoder corresponding to text features, during the training process, the input of the pre-defined reconstruction loss function includes the sample text features and the corresponding sample encoded text features.

[0155] Step S106: Input the comprehensive feature vector classification into the preset classifier, perform the semantic entity recognition task, and determine the classification result corresponding to the text block, including question category, answer category, or other categories.

[0156] The preset classifier is a multilayer perceptron (MLP). The fused integrated feature vector is input into the preset classifier to perform semantic entity recognition (SER) and relation extraction (RE) tasks.

[0157] In this embodiment, the semantic entity recognition task refers to classifying the comprehensive feature vector input into a preset classifier to obtain the classification result, thereby determining the category of the text block corresponding to the comprehensive feature vector, including question category, answer category, or other categories. For example, text blocks such as "unit price" or "name" are question categories, and text blocks such as "¥10.00" are answer categories.

[0158] For the categorized text blocks, a relation extraction task is used to determine whether there is a connection between each pair of question-answer sample pairs. For example, there is a connection between the question-class text block "unit price" and the adjacent answer-class text block "￥10.00", while there is no connection between the question-class text block "name" and the answer-class text block "￥10.00".

[0159] In this embodiment, before inputting the comprehensive feature vector classification into the preset classifier, the parameters of the preset classifier need to be trained using a preset classification loss function. The preset classification loss function is expressed by the following formula:

[0160]

[0161] In the formula, N represents the total number of training samples; L I y is the loss value of the i-th training sample; M represents the number of categories; ic The value can be either 0 or 1; it takes the value 1 when the class of training sample i is c, and 0 otherwise. ic This represents the probability that the predicted training sample i belongs to class c.

[0162] Specifically, before inputting the comprehensive feature vector classification into the preset classifier, the process also includes training the parameters of the preset classifier using a preset classification loss function.

[0163] Step S107: Based on the classification results, perform a relation extraction task to determine the connection relationship between each pair of question-answer sample pairs in the question classification and answer classification.

[0164] In this embodiment, based on the classification results of the text blocks obtained after the execution of the semantic entity recognition task, each pair of question-answer text blocks is enumerated to determine whether each pair of question-answer text blocks truly has a question-answer connection relationship.

[0165] In summary, the image-text feature alignment and fusion information extraction method provided in this embodiment obtains image features by inputting the target image into a pre-trained image feature extraction model; detects and identifies text blocks in the target image to obtain text block recognition results; inputs the text block recognition results into a pre-trained text feature extraction model to obtain text features; constructs text block features based on the text block recognition results and aligns the text features with the image features; obtains a pre-trained feature fusion model, and concatenates the text features and image features into the feature fusion model to obtain a comprehensive feature vector; the feature fusion model includes an autoencoder and a transformer encoder; the autoencoder is used to perform dimensionality unification processing on the input features; the transformer encoder includes several superimposed transformer encoder layers, which are connected by residuals; each transformer encoder layer contains a multi-head self-attention mechanism for concatenating the features after dimensionality unification processing; the comprehensive feature vector is classified and input into a preset classifier to execute the language. This system performs semantic entity recognition to determine the classification results corresponding to text blocks, including question categories, answer categories, or other categories. Based on the classification results, it performs relation extraction to determine the connection relationships between each pair of question-answer sample pairs in the question and answer categories. It addresses the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements. The system extracts image features from the target image using an image feature extraction model based on a residual learning framework. This model consists of several sequentially connected convolutional layers. By introducing residual learning, the image feature extraction model effectively solves the degradation problem in deep networks, making model training more efficient. Furthermore, its deep network structure improves the ability to extract image features from complex images, thus enhancing the accuracy of information extraction. Simultaneously, by bridging and aligning the identified text block features, it strengthens the semantic association between image and text features, better aligns features from different modalities, and further improves the accuracy of information extraction.

[0166] Furthermore, by integrating a feature pyramid network structure and a spatial attention mechanism, as well as applying a smoothing layer, the image feature extraction model can more effectively process complex image content and provide richer and more accurate image features for key information extraction. This enhances the ability to identify small targets in the target image and optimizes the extraction of spatial features, thereby improving the accuracy of image feature extraction and thus the accuracy of information extraction. At the same time, the spatial attention layer enhances the model's attention to specific regions by weighting the fused feature map at the pixel level, improving the overall recognition ability of the image feature extraction model for spatial features.

[0167] In addition, by introducing image modalities, the image feature extraction model can distinguish target images of different styles, improve the accuracy of distinguishing text blocks of different styles of text content on the same target image, and thus improve the accuracy of information extraction.

[0168] Furthermore, the text feature extraction model trained based on the ELECTRA encoder includes a generator and a discriminator. The generator focuses on predicting mask markers in the input, while the discriminator determines whether the markers have been replaced by the generator. This enables the ELECTRA encoder to more accurately identify and classify markers when processing text data, thereby improving the performance and efficiency of the text feature extraction model. At the same time, the discriminator of the ELECTRA encoder not only needs to understand the original vocabulary but also needs to identify which words have been replaced by the generator. This promotes a broader understanding of the entire vocabulary by the text feature extraction model, not just the mask part. Therefore, it can enhance the breadth and depth of vocabulary learning.

[0169] In addition, by extracting the core information of each modal feature through an autoencoder, the dimensionality of each feature is unified, reducing the computational burden caused by dimensionality. At the same time, modal feature vectors of the same length ensure that they have the same initial importance in subsequent transformer encoders, allowing the transformer encoder to learn the importance of different modes on its own. This improves the feature fusion model's ability to process complex data, enhances its understanding and application of different types of data, and thus improves the performance of the feature fusion model.

[0170] This embodiment provides an information extraction device for image-text feature alignment and fusion, such as... Figure 9 As shown, the device includes at least the following modules: image feature acquisition module 910, text block recognition module 920, text feature acquisition module 930, text block feature construction module 940, feature fusion and splicing module 950, first task execution module 960, and second task execution module 970.

[0171] The image feature acquisition module 910 is used to input the target image into a pre-trained image feature extraction model to obtain image features.

[0172] The text block recognition module 920 is used to detect and recognize text blocks in the target image and obtain the text block recognition result.

[0173] The text feature acquisition module 930 is used to input the text block recognition results into a pre-trained text feature extraction model to obtain text features.

[0174] The text block feature construction module 940 is used to construct text block features based on the text block recognition results and to bridge and align the text features with the image features.

[0175] The feature fusion and splicing module 950 is used to obtain a pre-trained feature fusion model. Text features and image features are input into the feature fusion model and spliced ​​to obtain a comprehensive feature vector.

[0176] The first task execution module 960 is used to input the comprehensive feature vector classification into the preset classifier, perform the semantic entity recognition task, and determine the classification result corresponding to the text block.

[0177] The second task execution module 970 is used to perform a relation extraction task based on the classification results, and to determine the connection relationship between each pair of question-answer sample pairs in the question classification and answer classification.

[0178] For relevant details, please refer to the methods and embodiments described above.

[0179] It should be noted that the image-text feature alignment and fusion information extraction device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image-text feature alignment and fusion information extraction device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image-text feature alignment and fusion information extraction device and the image-text feature alignment and fusion information extraction method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0180] This embodiment provides an electronic device, such as... Figure 10 As shown. The electronic device includes at least a processor 1010 and a memory 1020.

[0181] Processor 1010 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1010 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1010 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1010 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0182] The memory 1020 may include one or more computer-readable storage media, which may be non-transitory. The memory 1020 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1020 is used to store at least one instruction, which is executed by the processor 1010 to implement the image-text feature alignment and fusion information extraction method provided in the method embodiments of this application.

[0183] In some embodiments, the electronic device may also optionally include a peripheral device interface and at least one peripheral device. The processor 1010, memory 1020, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to, radio frequency circuits, touch displays, audio circuits, and power supplies.

[0184] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0185] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned image-text feature alignment and fusion information extraction method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0186] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned image-text feature alignment and fusion information extraction method.

[0187] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0188] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0189] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0190] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for information extraction through image-text feature alignment and fusion, characterized in that, The method includes the following steps: The target image is input into a pre-trained image feature extraction model to obtain image features; the image feature extraction model is a residual learning framework model, which includes a convolutional layer module, and the convolutional layer module includes several convolutional layers connected in sequence. Detect and identify text blocks in the target image to obtain text block recognition results; The text block recognition results are input into a pre-trained text feature extraction model to obtain text features; Based on the text block recognition results, text block features are constructed, and the text features are bridged and aligned with the image features; A pre-trained feature fusion model is obtained. The text block features, text features, and image features are input into the feature fusion model and concatenated to obtain a comprehensive feature vector. The feature fusion model includes an autoencoder and a transformer encoder. The autoencoder is used to perform dimensionality unification processing on the input features. The transformer encoder includes several stacked transformer encoder layers, which are connected by residuals. Each transformer encoder layer contains a multi-head self-attention mechanism for concatenating the features after dimensionality unification processing. The comprehensive feature vector is input into a preset classifier to perform a semantic entity recognition task, and the classification result corresponding to the text block is determined, including question category, answer category, or other category; Based on the classification results, a relation extraction task is performed to determine the connection relationship between each pair of question-answer sample pairs in the question classification and the answer classification; the image features include image semantic features and image spatial features; the text block features include text block visual features and text block index features. For the semantic entity recognition task, the visual features of the text block include the ratio of the horizontal coordinate of the center point of the text block to the width of the target image, the ratio of the vertical coordinate of the center point of the text block to the height of the target image, the ratio of the height of the text block to the height of the target image, and the ratio of the width of the text block to the width of the target image; the index features of the text block include the sequence number of the text block in all text blocks of the target image, arranged from top to bottom and then from left to right, and the number of all text blocks. For the relation extraction task, a question-answer sample pair includes a question category text block and an answer category text block. The visual features of the text blocks also include the ratio of the absolute value between the x-coordinates of the center points of the question category text block and the center points of the answer category text block to the width of the target image, the ratio of the absolute value to the height of the target image, and the positional relationship between the question category text block and the answer category text block. The text block index features also include the difference between the sequence number of the question category text block and the sequence number of the answer category text block.

2. The method according to claim 1, characterized in that, The autoencoder includes a first autoencoder corresponding to the image semantic features, a second autoencoder corresponding to the image spatial features, a third autoencoder corresponding to the text block visual features, a fourth autoencoder corresponding to the text block index features, and a fifth autoencoder corresponding to the text features.

3. The method according to claim 2, characterized in that, Before obtaining the pre-trained feature fusion model and concatenating the text features and image features to obtain the comprehensive feature vector, the method further includes: training the parameters of the autoencoder using a preset reconstruction loss function; the preset reconstruction loss function is expressed by the following formula: In the formula, n is the total number of samples; y i This represents the i-th input sample of the autoencoder; the y ' i This represents the i-th output sample of the autoencoder.

4. The method according to claim 1, characterized in that, Before inputting the comprehensive feature vector classification into the preset classifier, the method further includes: training the parameters of the preset classifier using a preset classification loss function; the preset classification loss function is expressed by the following formula: In the formula, N represents the total number of training samples; L I It is the loss value of the i-th training sample; M represents the number of categories; y ic The value can be 0 or 1; it takes the value 1 when the class of training sample i is c, and 0 otherwise. ic This represents the probability that the predicted training sample i belongs to class c.

5. The method according to claim 1, characterized in that, The image feature extraction model further includes a feature pyramid network module, which includes several feature extraction network branches. Each feature extraction network branch is connected to a convolutional layer and is used to receive the output of the corresponding convolutional layer as input and output the image features.

6. The method according to claim 5, characterized in that, Each feature extraction network branch includes a two-dimensional convolutional layer, a spatial attention layer, and a global average pooling layer.

7. An electronic device comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

Images