Image text feature alignment fusion information extraction method, device and program product

Abstract

The invention provides an information extraction method and device for image text feature alignment fusion and a program product. The method comprises the following steps: inputting a target image into an image feature extraction model to obtain image features; detecting and recognizing a text block in the target image to obtain a text block recognition result; inputting the text block recognition result into a text feature extraction model to obtain text features; text block features are constructed based on the text block recognition result, and the text features and the image features are subjected to bridging alignment; inputting the text features and the image features into a feature fusion model for splicing to obtain a comprehensive feature vector; the feature fusion model comprises an auto-encoder and a transformer encoder; inputting the comprehensive feature vectors into a preset classifier in a classified manner, executing a semantic entity recognition task, determining a classification result corresponding to the text block, and executing a relationship extraction task based on the classification result; the problem that information extraction is inaccurate when an image containing a complex background and multiple elements is processed can be solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an information extraction method, device and program product for image and text feature alignment and fusion. Background Art

[0002] With the development of artificial intelligence technology, image processing and natural language processing (NLP) technologies have made significant progress. Among many application fields, key information extraction has attracted much attention because it can process and understand image and text information at the same time. It means that for a picture with text content, the text blocks with key information extraction connection relationship are extracted and presented to the user, which is of great value in providing richer and more accurate user interaction, especially in scenarios that need to process complex image content and diverse text information.

[0003] However, the existing key information extraction faces many challenges, such as inaccuracy in image feature extraction, inaccuracy in text feature extraction, difficulty in aligning image features with text features, low feature fusion efficiency, low system response speed, etc. These challenges are particularly prominent when processing images with complex backgrounds and multiple elements, and there is a problem of inaccurate information extraction. Therefore, there is an urgent need to develop a new key information extraction method that can process image and text data more accurately, efficiently, and quickly, and improve the overall performance of multimodal data processing. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide an information extraction method, device and program product for image-text feature alignment fusion to eliminate or improve one or more defects in the prior art, and can solve the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements.

[0005] One aspect of the present invention provides an information extraction method for image-text feature alignment fusion, the method comprising the following steps:

[0006] Input the target image into a pre-trained image feature extraction model to obtain image features; the image feature extraction model is a model of a residual learning framework, including a convolutional layer module, and the convolutional layer module includes a number of convolutional layers connected in sequence;

[0007] Detect and recognize text blocks in the target image to obtain text block recognition results;

[0008] Input the text block recognition result into the pre-trained text feature extraction model to obtain text features;

[0009] Construct text block features based on text block recognition results, and bridge and align text features with image features;

[0010] Obtain a pre-trained feature fusion model, input text features and image features into the feature fusion model for splicing, and obtain a comprehensive feature vector; the feature fusion model includes an autoencoder and a transformer encoder; the autoencoder is used to unify the dimensions of the input features; the transformer encoder includes several superimposed transformer encoder layers, and the transformer encoder layers are connected through residuals; each transformer encoder layer contains a multi-head self-attention mechanism, which is used to splice the features after the dimensionality is unified;

[0011] Classify the comprehensive feature vector and input it into a preset classifier to perform a semantic entity recognition task and determine the classification result corresponding to the text block, including question category, answer category or other category;

[0012] Based on the classification results, the relationship extraction task is performed to determine the connection relationship between each pair of question-answer samples in the question classification and answer classification.

[0013] Optionally, the image features include image semantic features and image spatial features; the text block features include text block visual features and text block index features;

[0014] For the semantic recognition task, the text block visual features include the ratio of the horizontal coordinate of the center point of the text block to the width of the target image, the ratio of the vertical coordinate of the center point of the text block to the height of the target image, the ratio of the height of the text block to the height of the target image, and the ratio of the width of the text block to the width of the target image; the text block index features include the order of the text block in all the text blocks of the target image, which are arranged from top to bottom and then from left to right, and the number of all the text blocks;

[0015] For the relationship extraction task, a pair of question-answer sample pairs includes a question category text block and an answer category text block. The text block visual features also include the ratio of the absolute value between the horizontal coordinates of the center point of the question category text block and the center point of the answer category text block to the width of the target image, the ratio of the absolute value to the height of the target image, and the positional relationship between the question category text block and the answer category text block; the text block index features also include the difference between the sequence number of the question category text block and the sequence number of the answer category text block.

[0016] Optionally, the autoencoder includes a first autoencoder corresponding to image semantic features, a second autoencoder corresponding to image spatial features, a third autoencoder corresponding to text block visual features, a fourth autoencoder corresponding to text block index features, and a fifth autoencoder corresponding to text features.

[0017] Optionally, before obtaining a pre-trained feature fusion model and inputting text features and image features into the feature fusion model for splicing to obtain a comprehensive feature vector, the method further includes: training parameters of the autoencoder by a preset reconstruction loss function; the preset reconstruction loss function is represented by the following formula:

[0018]

[0019] Where n is the total number of samples; y i Represents the i-th input sample of the autoencoder; y' i Represents the i-th output sample of the autoencoder.

[0020] Optionally, before classifying the comprehensive feature vector into a preset classifier, the method further includes: training parameters of the preset classifier by using a preset classification loss function; the preset classification loss function is represented by the following formula:

[0021]

[0022] Where N is the total number of training samples; L I is the loss value of the i-th training sample; M represents the number of classification categories; y ic The value is 0 or 1. It takes 1 when the category of training sample i is c, otherwise it takes 0. ic Represents the probability that the predicted training sample i belongs to category c.

[0023] Optionally, the image feature extraction model also includes a feature pyramid network module, which includes a plurality of feature extraction network branches, each of which is connected to a convolutional layer for receiving the output of the corresponding convolutional layer as input and outputting image features.

[0024] Optionally, each feature extraction network branch includes a two-dimensional convolution layer, a spatial attention layer, and a global average pooling layer.

[0025] Another aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the electronic device implements the steps of the aforementioned image-text feature alignment and fusion information extraction method.

[0026] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the aforementioned information extraction method for image-text feature alignment and fusion.

[0027] Another aspect of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned information extraction method for image-text feature alignment and fusion.

[0028] The information extraction method, device and program product of the image-text feature alignment fusion of the present invention can solve the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements; the image features of the target image are extracted by an image feature extraction model of a residual learning framework, the image feature extraction model includes a number of convolutional layers connected in sequence, and by introducing residual learning, the image feature extraction model effectively solves the degradation problem in the deep network, making the model training more efficient, and in addition, its deep network structure can improve the ability to extract image features of complex images and improve the accuracy of information extraction; at the same time, by identifying the obtained text block features, the image features and text features are bridged and aligned, the semantic association between the image features and the text features is enhanced, and the alignment of features of different modalities is better performed, which can further improve the accuracy of information extraction.

[0029] In addition, the image feature extraction model can process complex image content more effectively and provide richer and more accurate image features for key information extraction by integrating the feature pyramid network structure and the spatial attention mechanism, as well as applying a smoothing layer. It improves the recognition ability of small targets in the target image and optimizes the extraction of image spatial features, thereby improving the accuracy of image feature extraction and thus improving the accuracy of information extraction. At the same time, the spatial attention layer enhances the model's attention to specific areas by weighting the fused feature map at the pixel level, thereby improving the overall recognition ability of the image feature extraction model for spatial features.

[0030] In addition, by introducing image modality, the image feature extraction model can distinguish target images of different styles, improve the accuracy of distinguishing text blocks with different styles of text content on the same target image, and thus improve the accuracy of information extraction.

[0031] In addition, the text feature extraction model is trained based on the ELECTRA encoder. The ELECTRA encoder includes a generator and a discriminator. The generator focuses on predicting the masked tokens in the input; the discriminator's task is to determine whether the token is replaced by the generator, so that the ELECTRA encoder can more accurately identify and classify tokens when processing text data, thereby improving the performance and efficiency of the text feature extraction model; at the same time, the discriminator of the ELECTRA encoder not only needs to understand the original vocabulary, but also needs to identify which vocabulary is replaced by the generator, which can promote the text feature extraction model's extensive understanding of the entire vocabulary, not just the masked part, and therefore, can improve the breadth and depth of vocabulary learning.

[0032] In addition, the core information of each modal feature is extracted through the autoencoder, the dimensions of each feature are unified, and the computational burden brought by the dimensions is reduced; at the same time, the modal feature vectors of the same length ensure that they have the same initial importance in the subsequent transformer encoder, allowing the transformer encoder to learn the importance of different modalities by itself, thereby improving the feature fusion model's ability to process complex data, and enhancing the feature fusion model's ability to understand and apply different types of data, thereby enhancing the performance of the feature fusion model.

[0033] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0034] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0036] Figure 1 A schematic diagram of the structure of a convolutional layer module in an image feature pre-extraction model provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of the residual structure in the image feature pre-extraction model provided by one embodiment of the present invention;

[0038] Figure 3 A flowchart of an information extraction method for image and text feature alignment and fusion provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an image feature pre-extraction model provided by an embodiment of the present invention;

[0040] Figure 5 A structural diagram of a text feature extraction model provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of a feature fusion model provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the structure of an autoencoder in a feature fusion model provided by an embodiment of the present invention;

[0043] Figure 8 A schematic diagram of the structure of a transformer encoder layer in a feature fusion model provided by an embodiment of the present invention;

[0044] Fig. 9 A block diagram of an information extraction device for image and text feature alignment and fusion provided by an embodiment of the present invention;

[0045] Fig.10 A block diagram of an electronic device provided according to another embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0047] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0048] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0049] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0050] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0051] The terms involved in the present invention are explained below.

[0052] Residual Network 50 (ResNet50) model: As an advanced deep convolutional neural network, it has been widely used in the field of image recognition and processing. Its deep network structure can effectively extract high-level features of images, which is particularly critical for understanding complex image content.

[0053] The core innovation of ResNet50 lies in its residual learning framework. In deep networks, the learning objective of each layer can be expressed as H(x). Here, x is the input of the layer. Layers in traditional networks usually try to learn this objective function directly. However, in the ResNet50 model, this pattern is changed by letting each layer learn the residual function F(x)=H(x)-x. Therefore, the original learning objective can be redefined as H(x)=F(x)+x. This residual structure can be obtained by the formula y=F(x,{W i})+x. Where y is the output of the layer, F(x,{W i}) indicates that the i The residual mapping function of ResNet50. By introducing residual learning, the ResNet50 model effectively solves the degradation problem in deep networks and makes network training more efficient. In addition, its deep network structure makes the ResNet50 model excellent in extracting complex image features. The structure of ResNet50 and its internal residual structure are shown in Figure 1. Figure 1 and Figure 2 For a 224×224×3 input image, the ResNet50 model extracts features layer by layer to obtain a 7×7×2048 feature map, which is then subjected to global average pooling at the feature channel level to obtain a 2048-dimensional image modality feature vector.

[0054] Feature Pyramid Network (FPN): is an improved feature extraction network that significantly improves the recognition ability of small targets by combining bottom-up and top-down feature flow mechanisms.

[0055] ELECTRA encoder (Efficiently Learning an Encoder that Classifies TokenReplacements Accurately):

[0056] The following is a detailed introduction to the information extraction method for image-text feature alignment and fusion provided by this application.

[0057] Optionally, the executor of the information extraction method for image-text feature alignment and fusion provided in the present application is an electronic device, which may be a terminal such as a computer, a mobile phone, a tablet computer, or a server. This embodiment does not limit the implementation method of the electronic device.

[0058] This embodiment provides an information extraction method for image-text feature alignment fusion, such as Figure 3 As shown, the information extraction method of image-text feature alignment fusion includes at least steps S101 to S107:

[0059] Step S101, input the target image into a pre-trained image feature extraction model to obtain image features.

[0060] Among them, the image feature extraction model is a model of the residual learning framework, including a convolutional layer module, and the convolutional layer module includes several convolutional layers connected in sequence; the target image is a document image containing a text block, including an image obtained by scanning a document or an image taken by a camera, etc.

[0061] like Figure 1 As shown, optionally, the convolutional layer module includes 5 sequentially connected convolutional layers. In actual implementation, the convolutional layer module can adjust the number of convolutional layers according to actual conditions, such as 4 sequentially connected convolutional layers, 6 sequentially connected convolutional layers, etc. This embodiment does not limit the number of convolutional layers.

[0062] In this embodiment, the image feature extraction model is a residual network 50 (ResNet50-Spatial FPN, ResNet50-SF) model that introduces a spatial attention mechanism (Spatial) and a feature pyramid network (FPN).

[0063] The Residual Network 50 (ResNet50) model is a deep convolutional neural network that has been widely used in the field of image recognition and processing. However, although the ResNet50 model performs well in image feature extraction, it has limitations in some aspects. For example, its recognition efficiency for small objects is low because the ResNet50 model prefers to extract salient features, which may not work well when processing small objects in the image. In addition, its spatial feature extraction performance may also be limited when processing images with complex spatial structures. Although ResNet50 is an effective choice as the basis of image modality in key information extraction systems, in specific application scenarios such as small object detection and image processing of complex spatial structures, high-level features are semantically rich but have low spatial resolution, which is not conducive to the precise positioning of small objects, while low-level features, despite their high spatial resolution, lack sufficient semantic information to distinguish small objects in complex backgrounds. Therefore, its performance needs to be improved through further optimization or combined with other techniques.

[0064] In this embodiment, the target image is a document image containing text blocks, wherein the text blocks are often small in size and distributed in specific areas of the document image. The ResNet50 model has limitations for feature extraction of such images.

[0065] In order to solve the above technical problems, in this embodiment, the performance of the ResNet50 model is improved by further optimization, combining the feature pyramid network structure and the spatial attention mechanism. By integrating the feature pyramid network structure and the spatial attention mechanism, and applying a smoothing layer, the image feature extraction model can more effectively process target images with complex content and provide richer and more accurate image features for the extraction of key information.

[0066] Among them, the feature pyramid network combines high-level semantic information and low-level detail information by fusing features between different levels. This fusion ensures that even small-sized objects can maintain sufficient semantic information and spatial details in the feature map. At the same time, the newly added top-down path passes deep, semantically rich features downward, enhancing the semantic expression ability of low-level features. This high-semantic information flows to the feature map with low semantic information but rich resolution features, making the model more sensitive to small objects.

[0067] In addition, the smoothing layers integrated in the feature pyramid network further improve the visual quality of the fused features. These smoothing layers ensure that the feature maps can maintain high visual continuity and consistency after fusing features of different scales, which is particularly important for highlighting small object features.

[0068] The spatial attention layer is integrated into each level of the feature pyramid network, which helps the image feature extraction model focus on specific key areas in the target image, especially small and important objects. The spatial attention layer enhances the model's attention to specific areas by weighting the fused feature map at the pixel level. This not only improves the ability to locate small objects, but also optimizes the overall recognition ability of the image feature extraction model for spatial features.

[0069] Specifically, refer to Figure 4 The image feature extraction model also includes a feature pyramid network module, which includes several feature extraction network branches, each of which includes a two-dimensional convolutional layer, a spatial attention layer and a global average pooling layer; each feature extraction network branch is connected to a convolutional layer to receive the output of the corresponding convolutional layer as input and output image features.

[0070] For example: taking the convolutional layer module including 5 convolutional layers connected in sequence as an example, convolutional layer 2, convolutional layer 3, convolutional layer 4 and convolutional layer 5 are respectively connected to a feature extraction network branch; the feature extraction network branch connections are two-dimensional convolutional layer, spatial attention layer, two-dimensional convolutional layer, spatial attention layer and global average pooling layer in sequence; convolutional layer 1 inputs a 224×224×3 model target image, and convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4 and convolutional layer 5 perform feature extraction layer by layer, and convolutional layer 2, convolutional layer 3, convolutional layer 4 and convolutional layer 5 input the extracted features into the corresponding feature extraction network branches, and finally the global average pooling layer outputs the image features.

[0071] In addition, in this embodiment, by introducing image modality, the image feature extraction model can distinguish target images of different styles, laying a good foundation for further accurately distinguishing text blocks with different styles of text content on the same target image, thereby improving the accuracy of distinguishing text blocks with different styles of text content on the same target image.

[0072] Step S102: Detect and recognize text blocks in the target image to obtain text block recognition results.

[0073] In this embodiment, the text blocks in the target image are detected and recognized by an optical character recognition (OCR) model, so as to ensure that the text content on the target image can be completely and accurately extracted.

[0074] In this embodiment, before detecting and identifying text blocks in a target image and obtaining text block recognition results, the method also includes: obtaining a training data set, including sample images and corresponding sample target images with text blocks annotated; obtaining a preset neural network model; inputting the sample images into the preset neural network model to obtain sample text block recognition results; inputting the sample text block recognition results and the sample target images with text blocks annotated into a preset loss function to obtain loss function results; and iterating the preset neural network model based on the loss function results to obtain an optical character recognition model.

[0075] Among them, the preset neural network models include but are not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), attention mechanisms (Attention) or transcribers (Transformer), etc.; the preset loss functions include but are not limited to CTC loss functions (Connectionist Temporal Classification Loss) or beam search loss functions (BeamSearch Loss), etc.

[0076] Step S103, inputting the text block recognition result into a pre-trained text feature extraction model to obtain text features.

[0077] In this embodiment, the text feature extraction model is a model obtained by training the ELECTRA encoder. The ELECTRA encoder includes a generator (G) and a discriminator (D). Figure 5 As shown in the figure, the generator is a smaller masked language model (MLM) that focuses on predicting the masked tokens in the input; the discriminator's task is to determine whether the token is replaced by the generator. This unique structural design enables the ELECTRA encoder to more accurately identify and classify tokens when processing text data, thereby improving the performance and efficiency of the model.

[0078] The traditional text feature extraction model is based on the training of Bidirectional Encoder Representations from Transformers (BERT). However, BERT uses the Masked Language Model (MLM) task, in which BERT randomly masks a portion of the words in the input text (about 15%) and then tries to predict these masked words. This method only allows the model to learn from a small number of masked words, and relatively little learning of non-masked tags, resulting in the problem of slow training of text feature extraction models; at the same time, in the process of restoring masked words, BERT focuses on in-depth learning of specific masked words, resulting in the problem of small breadth and depth of vocabulary learning.

[0079] In order to solve the above technical problems, in this embodiment, based on the text feature extraction model obtained by training the ELECTRA encoder, the model structure of the text feature extraction model is as follows: Figure 6 As shown. Since the generator model of the ELECTRA encoder replaces some input tokens and uses a discriminator model to determine whether each token is replaced, this method is called "Replaced Token Detection" (RTD); the discriminator of the ELECTRA encoder participates in the training when processing each input token, not just a few masked tokens, which makes the ELECTRA encoder more efficient than BERT when processing the same number of input tokens, thereby improving the training efficiency and speeding up the training of the text feature extraction model; at the same time, the discriminator of the ELECTRA encoder must not only understand the original vocabulary, but also identify which vocabulary is replaced by the generator. This method promotes the model's extensive understanding of the entire vocabulary, not just the masked part, and therefore can improve the breadth and depth of vocabulary learning.

[0080] In addition, the RTD task of the ELECTRA encoder requires the discriminator to distinguish between real and fake tokens, which enhances the model's sensitivity to natural and unnatural (artificially generated) language patterns in text. This ability is critical for understanding and generating more natural and fluent text.

[0081] In this embodiment, the training of the ELECTRA encoder includes two stages: a stage of pre-training on a large amount of text and a stage of fine-tuning for a specific task. In the pre-training stage, first, pre-training is performed on a large amount of text using an unsupervised learning method. Secondly, by replacing a part of the tags in the original input text, the model is required to predict the replaced tags. This replacement process is completed by sampling a replacement word pair. Through this replacement process, the model learns to distinguish between the real tags and the replaced tags, and improves the generation and generalization capabilities of the model; in the fine-tuning stage, first, fine-tuning is performed on a labeled data set for a specific task to obtain a target data set. Secondly, using a supervised learning method, back propagation and gradient descent are performed on the target data set to fine-tune the parameters of the model to maximize the performance of the model on a specific task. Through this pre-training and fine-tuning method, the ELECTRA encoder can generate a 768-dimensional text feature vector for the input text string, effectively capturing and expressing the complex text features of the text data.

[0082] During the training of the ELECTRA encoder, the goal of the generator is to maximize the log-likelihood of the mask position in the input sequence. This approach enables the generator to learn semantic information in the text more effectively. The discriminator is committed to distinguishing between the original tokens and the replaced tokens. This distinction mechanism enables the discriminator to accurately identify subtle changes in the text, thereby improving the recognition ability of the model.

[0083] In this embodiment, the training objective of the generator is expressed by the following formula:

[0084] L G =E x～Data,t～Mask(x) [logP G (x t |x \t )]

[0085] Where x represents the input sequence and t represents the masked marker position.

[0086] The training objective of the discriminator is expressed as follows:

[0087]

[0088] Among them, x t is the tth token in the sequence, original t is a binary label, indicating that xt Whether it is an original tag.

[0089] Step S104, constructing text block features based on the text block recognition results, and bridging and aligning the text features with the image features.

[0090] Since text blocks with specific text content often have specific shapes and are distributed in specific position areas of the target image. Text blocks with QA matching connection relationships in the target image often have specific relative positions, relative orientations, and relative serial numbers. Based on this, in this embodiment, in addition to the image features of the target image where the text block is located and the text features of the text block's own text content, the text block features of the text block itself are also used, including the position, shape, and serial number of the text block, as well as the relative position and relative serial number between text blocks, as a bridge connecting image features and text features, so as to better align features of different modalities.

[0091] Step S105, obtaining a pre-trained feature fusion model, inputting text block features, text features and image features into the feature fusion model for splicing, and obtaining a comprehensive feature vector.

[0092] Since different features are often strongly correlated, it is particularly important to fully interact and integrate different features and learn their correlation.

[0093] For example, text blocks such as "name", "Zhang San", and "home address" (text features) often appear on the target image in the ID card style (image semantic features) and are distributed in specific locations (image spatial features, text block visual features). In the target image in the ID card style, all text blocks have a specific order (text block index features), and the "name" and "Zhang San" text blocks with question-answer connection relationships are often adjacent in position (image spatial features, text block visual features, text block index features);

[0094] Alternatively, text blocks such as "unit price", "￥10.00", and "tax amount" (text features) often appear on invoice-style target images (image semantic features) and are distributed in specific locations (image spatial features, text block visual features). In the invoice-style target image, all text blocks have a specific order (text block index features), and the "unit price" and "￥10.00" fields with a question-answer connection relationship are often adjacent in position (image spatial features, text block visual features, text block index features).

[0095] At the same time, the fusion and splicing of text block features, text features and image features are realized through the feature fusion model. Although high information density text features, image features and text block features provide rich information, this may also bring challenges in interaction, alignment and fusion between features. Therefore, in order for the feature fusion model to better learn the association between different features and perform more efficient feature interaction and fusion, the image features are further subdivided into 1024-dimensional image semantic features and 3136-dimensional image spatial features, and the text block features are further subdivided into text block visual features and text block index features, that is, image features include image semantic features and image spatial features; text block features include text block visual features and text block index features. Image semantic features represent the overall semantic information of the target image, and image spatial features contain fine-grained features at the pixel level of the target image.

[0096] Among them, the image features can be expressed as:

[0097] Image Feature=Image Semantic Feature 1×1024 +Image Spatial Feature 1×3136

[0098] In the formula, Image Feature represents image features; Image Semantic Feature 1×1024 Represents 1024-dimensional image semantic features; Image Spatial Feature 1×3136 Represents 3136-dimensional image space features.

[0099] refer to Figure 4 In the image feature extraction model, feature rearrangement is a key step to extract and separate image semantic features and image spatial features in order to better support the needs of key information extraction. Image semantic features are composed of four layers of feature vectors i1, i2, i3, and i4 that have passed through the top-down information flow in the image feature extraction model and have undergone smoothing and spatial attention mechanisms. The number of channels in each layer is 256, and global average pooling is performed at the channel level, and then spliced ​​to form a 1024-dimensional vector of image semantic features. It can be expressed by the following formula:

[0100] Image Semantic Feature 1×1024 =Concatenate(

[0101] Feature1 1×256 ,Feature2 1×256 ,Feature3 1×256 ,Feature4 1×256 )

[0102] In the formula, Image Semantic Feature 1×1024 Represents 1024-dimensional image semantic features;

[0103] Feature1 1×256 Represents feature 1, namely Global Average Pooling on channel (i1 56×56×256 ), where i1 56×56×256 It is the feature received by the global average pooling layer;

[0104] Feature2 1×256 Represents feature 2, namely Global Average Pooling on channel (i2 56×56×256 ), where i2 56×56×256 It is the feature received by the global average pooling layer;

[0105] Feature3 1×256 Represents feature 3, namely Global Average Pooling on channel (i3 56×56×256 ), where i3 56×56×256 It is the feature received by the global average pooling layer;

[0106] Feature4 1×256 Represents feature 4, namely Global Average Pooling on channel (i4 56×56×256 ), where i4 56×56×256 is the feature received by the global average pooling layer.

[0107] The image spatial feature is the lowest layer 56×56×256 feature map of the image feature extraction model. This feature map contains the richest spatial information. After adding the learnable position encoding vector of the same size, global average pooling is performed at each pixel level to convert the feature map size to 56×56, and then flattened to obtain a 3136-dimensional feature vector, which can be expressed by the following formula:

[0108] Image Spatial Feature 1×3136 =Flatten(Global Average Pooling on pixel (i1 56×56×256 +

[0109] Position Encoding 56×56×256 ))

[0110] In the formula, Image Spatial Feature 1×3136 Represents 3136-dimensional image space features; i1 56×56×256 Features received by the global average pooling layer; Position Encoding 56×56×256 Represents a position encoding vector.

[0111] The text block features are further subdivided into text block visual features and text block index features, which can be expressed as follows:

[0112] Text Block Feature=

[0113] Text Block Visual Feature 4(for SER),8(for RE) +Text Block Index Feature 2(for SER),4(for RE)

[0114] In the formula, Text Block Feature represents the text block feature; Text Block Visual Feature 4(for SER),8(for RE) It means that for the semantic entity recognition task, the visual features of the text block are 4-dimensional, and for the relationship extraction task, the visual features of the text block are 8-dimensional, that is,

[0115] Text Block Index Feature 2(for SER),4(for RE) It means that for the semantic entity recognition task, the text block index feature is 2-dimensional, and for the relationship extraction task, the text block index feature is 4-dimensional, that is,

[0116]

[0117] Specifically, for the semantic entity recognition task, the classification results obtained by recognition include question categories, answer categories or other categories, and the text block visual features include the ratio of the horizontal coordinate of the center point of the text block to the width of the target image, the ratio of the vertical coordinate of the center point of the text block to the height of the target image, the ratio of the height of the text block to the height of the target image, and the ratio of the width of the text block to the width of the target image, which can be expressed by the following formula:

[0118]

[0119] In the formula, Indicates the ratio of the horizontal coordinate of the center point of the text block to the width of the target image; Indicates the ratio of the ordinate of the center point of the text block to the height of the target image; Indicates the ratio of the height of the text block to the height of the target image; Indicates the ratio of the width of the text block to the width of the target image.

[0120] The text block index feature includes the order of the text block in all the text blocks of the target image, which is arranged from top to bottom and then from left to right, and the number of corresponding text blocks, which can be expressed by the following formula:

[0121] Block Index Feature=(index,total_blocks)

[0122] Where index represents the order of the text block in all the text blocks of the target image, arranged from top to bottom and then from left to right; total_blocks represents the number of all text blocks.

[0123] For the relationship extraction task, it is used to determine the connection relationship between each pair of question-answer sample pairs in question classification and answer classification, wherein a pair of question-answer sample pairs includes a question category text block and an answer category text block; accordingly, in addition to the ratio of the horizontal coordinate of the center point of the question category text block to the width of the target image, the ratio of the vertical coordinate of the center point of the question category text block to the height of the target image, the ratio of the horizontal coordinate of the center point of the answer category text block to the width of the target image, and the ratio of the vertical coordinate of the center point of the answer category text block to the height of the target image, the text block visual features also include the ratio of the absolute value between the horizontal coordinates of the center point of the question category text block and the center point of the answer category text block to the width of the target image, the ratio of the absolute value to the height of the target image, and the positional relationship between the question category text block and the answer category text block, including the up-and-down positional relationship and the left-and-right positional relationship.

[0124] The text block index feature includes, in addition to the sequence numbers of question-type text blocks in all text blocks of the target image arranged from top to bottom and then from left to right, the sequence numbers of answer-type text blocks in all text blocks of the target image arranged from top to bottom and then from left to right, and the corresponding number of all text blocks, the difference between the sequence numbers of question-type text blocks and the sequence numbers of answer-type text blocks.

[0125] In this embodiment, features with different functions are distinguished by feature rearrangement, which helps the subsequent feature fusion model to interactively learn different features. Through the above optimization, small targets and complex spatial structures can be processed more effectively, and more efficient feature fusion can be achieved. This improved framework provides stronger support for processing various types of multimodal data, which is an important supplement and improvement to the existing technology.

[0126] In this embodiment, the feature fusion model includes an autoencoder (AutoEnocder) and a transformer encoder (Transformer Encoder).

[0127] Among them, the autoencoder is used to unify the dimensions of the input features and transform the dimensions of the input features to 512 dimensions; Figure 5 As shown, the autoencoder includes an autoencoder corresponding to image semantic features, an autoencoder corresponding to image spatial features, an autoencoder corresponding to text block visual features, an autoencoder corresponding to text block index features, and an autoencoder corresponding to text features.

[0128] Specifically, the autoencoder corresponding to the image semantic features is used to transform the 1024-dimensional image semantic features into 312-dimensional encoded pixel features, which is expressed by the following formula:

[0129] Encoded Image Semantic Feature=AutoEncoder image semantic (F image semantic )

[0130] In the formula, Encoded Image Semantic Feature represents the encoded pixel feature; AutoEncoder image semantic Represents the autoencoder corresponding to the semantic features of the image; F image semantic Represents the semantic features of the image.

[0131] The autoencoder corresponding to the image spatial features is used to transform the 3136-dimensional image spatial features into 512-dimensional encoded pixel features, which can be expressed as follows:

[0132] Encoded Image Spatial Feature=AutoEncoder image spatial (F image spatial )

[0133] In the formula, Encoded Image Spatial Feature represents the encoded semantic features; AutoEncoder image spatial Represents the autoencoder corresponding to the image spatial features; F image spatial Represents the image space features.

[0134] The autoencoder corresponding to the text block visual feature is used to transform the 4- or 8-dimensional text block visual feature into the 512-dimensional encoded text block visual feature, which is expressed by the following formula:

[0135] Encoded Text Block Visual Feature=AutoEncoder text block visual (F text block visual )

[0136] Where, Encoded Text Block Visual Feature represents the visual feature of the encoded text block; AutoEncoder text block visual Represents the autoencoder corresponding to the visual features of the text block; F text block visual Represents the visual characteristics of a text block.

[0137] The autoencoder corresponding to the text block index feature is used to transform the 2- or 4-dimensional text block index feature into a 512-dimensional encoded text block index feature, which is expressed by the following formula:

[0138] Encoded Text Block Index Feature=AutoEncoder text block index (F text block index )

[0139] In the formula, Encoded Text Block Index Feature represents the encoded text block index feature; AutoEncoder text block index Represents the autoencoder corresponding to the text block index feature; F text block index Represents a text block index feature.

[0140] The autoencoder corresponding to the text feature is used to transform the 768-dimensional text feature into a 512-dimensional encoded text feature, which is expressed as follows:

[0141] Encoded Text Feature=AutoEncoder text (F text )

[0142] In the formula, Encoded Text Feature represents the encoded text feature; AutoEncoder text Represents the autoencoder corresponding to the text feature; F text Represents text features.

[0143] In this embodiment, the purpose of the autoencoder is to extract the core information of each modal feature and reduce the computational burden caused by the dimension; at the same time, the modal feature vectors of the same length ensure that they have the same initial importance in the subsequent transformer encoder, allowing the transformer encoder to learn the importance of different modalities by itself. The structure of the autoencoder is as follows Figure 7 shown.

[0144] In this embodiment, the structure of the transformer encoder is as follows: Figure 8 As shown in Figure 1, the transformer encoder includes several stacked transformer encoder layers, and the transformer encoder layers are connected through residual connections; each transformer encoder layer contains a multi-head self-attention mechanism, which is used to concatenate the features after the dimension is uniformly processed to obtain a comprehensive feature vector. embedding After the multi-head self-attention mechanism, the multiple stacking of residual connections is transformed into the output X hidden The process is as Figure 7 shown.

[0145] The comprehensive feature vector is expressed as follows:

[0146] Integrated Feature=Concat(Encoded Text Feature, Encoded ImageSemantic Feature, Encoded Image Spatial Feature, Encoded Text Block VisualFeature, Encoded Text Block Index Feature)

[0147] In the formula, Integrated Feature represents the comprehensive feature vector; Encoded Text Feature represents the encoded text feature; Encoded Image Semantic Feature represents the encoded pixel feature; Encoded Image SpatialFeature represents the encoded semantic feature; Encoded Text Block Visual Feature represents the encoded text block visual feature; Encoded Text Block Index Feature represents the encoded text block index feature.

[0148] In this embodiment, the transformer encoder realizes interactive learning between input features through a series of superimposed transformer encoder layers. Each transformer encoder layer contains a multi-head self-attention mechanism, which can process information in different subspaces in parallel to capture subtle correlations between input features. After the multi-head self-attention mechanism, a series of linear transformations and normalization processes are performed to further refine the feature representation.

[0149] In this embodiment, each transformer encoder layer uses residual connection, which can alleviate the gradient vanishing problem in the deep network and ensure that information is effectively transmitted in the deep layer of the network. In this way, each transformer encoder layer not only learns new feature representations, but also retains the information of the previous layer, enhancing the ability to integrate features. After multiple layers of transformer encoder processing, the input features are effectively fused to form a comprehensive feature vector representation with stronger comprehensive performance.

[0150] Through the above fusion mechanism, efficient integration of different modal features is achieved, providing a powerful feature representation foundation for the key information extraction system. This fusion method not only improves the feature fusion model's ability to process complex data, but also enhances the feature fusion model's ability to understand and apply different types of data, thereby enhancing the performance of the feature fusion model.

[0151] In addition, before obtaining a pre-trained feature fusion model, inputting text features and image features into the feature fusion model for splicing, and obtaining a comprehensive feature vector, it also includes: training the parameters of the autoencoder through a preset reconstruction loss function; the preset reconstruction loss function is expressed by the following formula:

[0152]

[0153] Where n is the total number of samples; y i Represents the i-th input sample of the autoencoder; y' i Represents the i-th output sample of the autoencoder.

[0154] For example, when the autoencoder is an autoencoder corresponding to text features, during the training process, the input of the preset reconstruction loss function includes sample text features and corresponding sample encoded text features.

[0155] Step S106, classify the comprehensive feature vector and input it into a preset classifier, perform the semantic entity recognition task, and determine the classification result corresponding to the text block, including question category, answer category or other category.

[0156] The preset classifier is a multilayer perceptron (MLP). The fused comprehensive feature vector is input into the preset classifier to perform the tasks of speech emotion recognition (SER) and relation extraction (RE).

[0157] In this embodiment, the semantic entity recognition task refers to classifying the comprehensive feature vector input into the preset classifier to obtain a classification result to determine the category of the text block corresponding to the comprehensive feature vector, including question category, answer category or other category. For example, a text block such as "unit price" or "name" is a question category, and a text block such as "￥10.00" is an answer category.

[0158] For the classified text blocks, the relationship extraction task is used to determine whether there is a connection relationship between each pair of question-answer samples. For example, there is a connection relationship between the question-type text block "unit price" and the adjacent answer-type text block "￥10.00", but there is no connection relationship between the question-type text block "name" and the answer-type text block "￥10.00".

[0159] In this embodiment, before the comprehensive feature vector is classified and input into the preset classifier, the parameters of the preset classifier need to be trained by the preset classification loss function. The preset classification loss function is represented by the following formula:

[0160]

[0161] Where N is the total number of training samples; L I is the loss value of the i-th training sample; M represents the number of classification categories; y ic The value is 0 or 1. It takes 1 when the category of training sample i is c, otherwise it takes 0. ic Represents the probability that the predicted training sample i belongs to category c.

[0162] Specifically, before classifying the comprehensive feature vector into a preset classifier, the method further includes: training parameters of the preset classifier through a preset classification loss function.

[0163] Step S107, based on the classification results, execute the relationship extraction task to determine the connection relationship between each pair of question-answer sample pairs in the question classification and answer classification.

[0164] In this embodiment, based on the classification results of the text blocks obtained after the execution of the semantic entity recognition task, each pair of question-answer text blocks is enumerated to determine whether each pair of question-answer text blocks actually has a question-answer connection relationship.

[0165] In summary, the information extraction method for image-text feature alignment and fusion provided in the present embodiment obtains image features by inputting the target image into a pre-trained image feature extraction model; detects and identifies the text blocks in the target image to obtain text block recognition results; inputs the text block recognition results into a pre-trained text feature extraction model to obtain text features; constructs text block features based on the text block recognition results, and bridges and aligns the text features with the image features; obtains a pre-trained feature fusion model, inputs the text features and the image features into the feature fusion model for splicing, and obtains a comprehensive feature vector; the feature fusion model includes an autoencoder and a transformer encoder; the autoencoder is used to unify the dimensions of the input features; the transformer encoder includes a number of superimposed transformer encoder layers, and the transformer encoder layers are connected by residuals; each transformer encoder layer includes a multi-head self-attention mechanism, which is used to splice the features after the dimensionality is unified; the comprehensive feature vector is classified and input into a preset classifier, and the language is executed The method can solve the problem of inaccurate information extraction when processing images with complex backgrounds and multiple elements. The image feature extraction model of the residual learning framework is used to extract the image features of the target image. The image feature extraction model includes several convolutional layers connected in sequence. By introducing residual learning, the image feature extraction model effectively solves the degradation problem in the deep network, making the model training more efficient. In addition, its deep network structure can improve the ability to extract image features of complex images and improve the accuracy of information extraction. At the same time, by identifying the obtained text block features, the image features and text features are bridged and aligned, which enhances the semantic association between image features and text features, and better aligns features of different modalities, which can further improve the accuracy of information extraction.

[0166] In addition, the image feature extraction model can process complex image content more effectively and provide richer and more accurate image features for key information extraction by integrating the feature pyramid network structure and the spatial attention mechanism, as well as applying a smoothing layer. It improves the recognition ability of small targets in the target image and optimizes the extraction of image spatial features, thereby improving the accuracy of image feature extraction and thus improving the accuracy of information extraction. At the same time, the spatial attention layer enhances the model's attention to specific areas by weighting the fused feature map at the pixel level, thereby improving the overall recognition ability of the image feature extraction model for spatial features.

[0167] In addition, by introducing image modality, the image feature extraction model can distinguish target images of different styles, improve the accuracy of distinguishing text blocks with different styles of text content on the same target image, and thus improve the accuracy of information extraction.

[0168] In addition, the text feature extraction model is trained based on the ELECTRA encoder. The ELECTRA encoder includes a generator and a discriminator. The generator focuses on predicting the masked tokens in the input; the discriminator's task is to determine whether the token is replaced by the generator, so that the ELECTRA encoder can more accurately identify and classify tokens when processing text data, thereby improving the performance and efficiency of the text feature extraction model; at the same time, the discriminator of the ELECTRA encoder not only needs to understand the original vocabulary, but also needs to identify which vocabulary is replaced by the generator, which can promote the text feature extraction model's extensive understanding of the entire vocabulary, not just the masked part, and therefore, can improve the breadth and depth of vocabulary learning.

[0169] In addition, the core information of each modal feature is extracted through the autoencoder, the dimensions of each feature are unified, and the computational burden brought by the dimensions is reduced; at the same time, the modal feature vectors of the same length ensure that they have the same initial importance in the subsequent transformer encoder, allowing the transformer encoder to learn the importance of different modalities by itself, thereby improving the feature fusion model's ability to process complex data, and enhancing the feature fusion model's ability to understand and apply different types of data, thereby enhancing the performance of the feature fusion model.

[0170] This embodiment provides an information extraction device for aligning and fusing image and text features. Fig. 9 As shown, the device includes at least the following modules: an image feature acquisition module 910, a text block recognition module 920, a text feature acquisition module 930, a text block feature construction module 940, a feature fusion and splicing module 950, a first task execution module 960 and a second task execution module 970.

[0171] The image feature acquisition module 910 is used to input the target image into a pre-trained image feature extraction model to obtain image features.

[0172] The text block recognition module 920 is used to detect and recognize text blocks in the target image and obtain text block recognition results.

[0173] The text feature acquisition module 930 is used to input the text block recognition result into a pre-trained text feature extraction model to obtain text features.

[0174] The text block feature construction module 940 is used to construct text block features based on the text block recognition results, and bridge and align the text features with the image features.

[0175] The feature fusion and splicing module 950 is used to obtain a pre-trained feature fusion model, input text features and image features into the feature fusion model for splicing, and obtain a comprehensive feature vector.

[0176] The first task execution module 960 is used to classify the comprehensive feature vector and input it into a preset classifier, execute the semantic entity recognition task, and determine the classification result corresponding to the text block.

[0177] The second task execution module 970 is used to execute the relationship extraction task based on the classification result, and determine the connection relationship between each pair of question-answer sample pairs in the question classification and the answer classification.

[0178] For relevant details, refer to the above methods and embodiments.

[0179] It should be noted that: when the information extraction device for image-text feature alignment and fusion provided in the above-mentioned embodiment performs information extraction by image-text feature alignment and fusion, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the information extraction device for image-text feature alignment and fusion is divided into different functional modules to complete all or part of the functions described above. In addition, the information extraction device for image-text feature alignment and fusion provided in the above-mentioned embodiment belongs to the same concept as the information extraction method embodiment for image-text feature alignment and fusion. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0180] This embodiment provides an electronic device, such as Fig.10 The electronic device at least includes a processor 1010 and a memory 1020 .

[0181] The processor 1010 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1010 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1010 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1010 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0182] The memory 1020 may include one or more computer-readable storage media, which may be non-transitory. The memory 1020 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1020 is used to store at least one instruction, which is used to be executed by the processor 1010 to implement the information extraction method for image text feature alignment fusion provided in the method embodiment of the present application.

[0183] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 1010, the memory 1020 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0184] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0185] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the information extraction method for image-text feature alignment fusion are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0186] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned information extraction method for image-text feature alignment and fusion.

[0187] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0188] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0189] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An information extraction method for image-text feature alignment fusion, characterized in that: The method comprises the following steps: Inputting the target image into a pre-trained image feature extraction model to obtain image features; the image feature extraction model is a model of a residual learning framework, including a convolutional layer module, and the convolutional layer module includes a plurality of convolutional layers connected in sequence; Detecting and identifying text blocks in the target image to obtain text block recognition results; Inputting the text block recognition result into a pre-trained text feature extraction model to obtain text features; Constructing text block features based on the text block recognition results, and bridging and aligning the text features with the image features; A pre-trained feature fusion model is obtained, and the text features and the image features are input into the feature fusion model for splicing to obtain a comprehensive feature vector; the feature fusion model includes an autoencoder and a transformer encoder; the autoencoder is used to perform dimension unification processing on the input features; the transformer encoder includes a plurality of superimposed transformer encoder layers, and the transformer encoder layers are connected by residuals; each transformer encoder layer includes a multi-head self-attention mechanism for splicing the features after dimension unification processing; Classify the comprehensive feature vector and input it into a preset classifier to perform a semantic entity recognition task to determine the classification result corresponding to the text block, including question category, answer category or other category; Based on the classification result, a relationship extraction task is performed to determine the connection relationship between each pair of question-answer samples in the question classification and the answer classification.

2. The method according to claim 1, characterized in that The image features include image semantic features and image spatial features; the text block features include text block visual features and text block index features; For the semantic recognition task, the text block visual features include a ratio of a horizontal coordinate of a text block center point to a width of the target image, a ratio of a vertical coordinate of a text block center point to a height of the target image, a ratio of a height of the text block to a height of the target image, and a ratio of a width of the text block to a width of the target image; The text block index feature includes the order of the text block in all the text blocks of the target image, which is arranged from top to bottom and then from left to right, and the number of all the text blocks corresponding to the text blocks; For the relationship extraction task, a pair of question-answer sample pairs includes a question category text block and an answer category text block, and the text block visual features also include the ratio of the absolute value between the horizontal coordinates of the center point of the question category text block and the center point of the answer category text block to the width of the target image, the ratio of the absolute value to the height of the target image, and the positional relationship between the question category text block and the answer category text block; the text block index feature also includes the difference between the sequence number of the question category text block and the sequence number of the answer category text block.

3. The method according to claim 2, characterized in that The autoencoder includes a first autoencoder corresponding to the image semantic features, a second autoencoder corresponding to the image spatial features, a third autoencoder corresponding to the text block visual features, a fourth autoencoder corresponding to the text block index features, and a fifth autoencoder corresponding to the text features.

4. The method according to claim 3, characterized in that The method further includes: training the parameters of the autoencoder by using a preset reconstruction loss function before obtaining a pre-trained feature fusion model, inputting the text feature and the image feature into the feature fusion model for splicing, and obtaining a comprehensive feature vector; the preset reconstruction loss function is represented by the following formula: Where n is the total number of samples; y is i Represents the i-th input sample of the autoencoder; the y' i Represents the i-th output sample of the autoencoder.

5. The method according to claim 1, characterized in that Before the comprehensive feature vector is classified and input into the preset classifier, the method further includes: training the parameters of the preset classifier by using a preset classification loss function; the preset classification loss function is represented by the following formula: Wherein, N represents the total number of training samples; L I is the loss value of the i-th training sample; M represents the number of classification categories; y ic The value is 0 or 1. When the category of training sample i is c, it takes 1, otherwise it takes 0. ic Represents the probability that the predicted training sample i belongs to category c.

6. The method according to claim 1, characterized in that The image feature extraction model also includes a feature pyramid network module, which includes a plurality of feature extraction network branches, each of which is connected to a convolutional layer and is used to receive the output of the corresponding convolutional layer as input and output the image features.

7. The method according to claim 6, characterized in that Each feature extraction network branch includes a two-dimensional convolution layer, a spatial attention layer and a global average pooling layer.

8. An electronic device comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images