Feature fusion method, device, electronic device and computer-readable storage medium

Through the self-attention and cross-attention processing of the feature fusion network, image features and text features are fused, which solves the problem of poor feature fusion effect in the prior art, and improves the accuracy of model performance and classification results.

CN115601620BActive Publication Date: 2024-11-12BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211304730.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-24
Publication Date
2024-11-12
Estimated Expiration
2042-10-24

AI Technical Summary

Technical Problem

When the prior art combines different features into new features, the effect is poor, affecting the model performance. Especially in document image classification, the correlation between different modal features is not fully utilized, resulting in poor accuracy of classification results.

Method used

Feature fusion network, including self-attention sub-network and cross-attention sub-network, is adopted to fuse image features and text features through self-attention processing and cross-attention processing, generate cross-attention features, and share attention weight parameters to enhance Feature fusion effect, and the fused features are further processed through the feature fusion layer.

Benefits of technology

The effect of feature fusion is improved and the model performance is improved. Especially in document image classification, the semantic confusion problem is reduced, the semantic expression ability is enhanced, and the accuracy of classification results is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115601620B_ABST
    Figure CN115601620B_ABST
Patent Text Reader

Abstract

The present disclosure provides a feature fusion method, device, electronic device and computer-readable storage medium, which relate to the field of artificial intelligence technology, specifically to the field of deep learning, image processing, large models, and computer vision technology, and can be applied to scenarios such as optical character recognition. The specific implementation scheme is: obtaining a first input feature and a second input feature, and the correlation between the first input feature and the second input feature and the target analysis object meets a preset correlation condition; the first input feature and the second input feature are input into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature. Based on the feature fusion network provided by this scheme, the first input feature and the second input feature are subjected to feature fusion processing, and the first cross-attention feature and the second cross-attention feature that are fused with the characteristics of the first input feature and the characteristics of the second input feature are obtained, which can improve the effect of feature fusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of deep learning, image processing, large models, and computer vision technology, and can be applied to scenarios such as optical character recognition (OCR). Specifically, the present disclosure relates to a feature fusion method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Feature fusion refers to fusing different features into new features, thereby better utilizing the characteristics of different features to improve model performance.

[0003] The effect of feature fusion will have a significant impact on model performance. Therefore, how to improve the effect of feature fusion when fusing different features into new features to ensure model performance has become an important technical issue in related fields. Summary of the invention

[0004] In order to solve at least one of the above-mentioned defects, the present disclosure provides a feature fusion method, device, electronic device and computer-readable storage medium.

[0005] According to a first aspect of the present disclosure, a feature fusion method is provided, the method comprising:

[0006] Acquire a first input feature and a second input feature, wherein the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition;

[0007] Inputting the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0008] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0009] The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature;

[0010] The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature;

[0011] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0012] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0013] According to a second aspect of the present disclosure, a feature fusion device is provided, the device comprising:

[0014] A feature acquisition module, used to acquire a first input feature and a second input feature, wherein the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition;

[0015] A feature fusion module, used for inputting the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0016] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0017] The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature;

[0018] The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature;

[0019] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0020] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] A memory in communication with the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the feature fusion method.

[0025] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned feature fusion method.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0028] Figure 1 is a flowchart of a feature fusion method provided by an embodiment of the present disclosure;

[0029] Figure 2 It is a flowchart of a specific implementation of the feature fusion method provided in the embodiment of the present disclosure;

[0030] Figure 3 It is a flowchart of a document image classification method provided by an embodiment of the present disclosure;

[0031] Figure 4 is a structural schematic diagram of a feature fusion device provided by an embodiment of the present disclosure;

[0032] Figure 5 is a structural schematic diagram of a document image classification device provided by an embodiment of the present disclosure;

[0033] Figure 6 It is a block diagram of an electronic device used to implement the feature fusion method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] When different features are fused into new features for model processing, the effect of feature fusion has an important impact on model performance. Therefore, how to improve the effect of feature fusion has become an important technical issue in related fields.

[0036] When classifying document images, the related art generally collects the image features and text features of the document images separately, and then determines the document classification results based on the image features and text features. For example, the classification results are obtained based on the image features and text features respectively, and then the final classification result is determined from the two classification results based on voting and other rules. This method does not take into account the correlation between different modal features, resulting in poor accuracy of the final classification results.

[0037] There are also some feature fusion methods in the related art, which are used to fuse image features and text features, but the effect of feature fusion through these feature fusion methods may be poor, which will also affect the accuracy of classification results. For example, the new features obtained by fusing image features with text features may not have a good effect on semantic expression, and there may be semantic confusion problems, which will affect the accuracy of classification results.

[0038] The feature fusion method, device, electronic device and computer-readable storage medium provided in the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0039] Figure 1 A flow chart of a feature fusion method provided by an embodiment of the present disclosure is shown. Figure 1 As shown in , the method can mainly include:

[0040] Step S110: Acquire a first input feature and a second input feature, wherein the correlations between the first input feature and the second input feature and the target analysis object satisfy a preset correlation condition.

[0041] The target analysis object may be an object that needs to be subjected to feature analysis based on fused features obtained through feature fusion processing.

[0042] The first input feature and the second input feature are both features related to the target analysis object, and the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition. The correlation condition can be set according to actual needs to ensure that the first input feature and the second input feature have a strong correlation with the target analysis object, so that the fused feature obtained by fusion of the first input feature and the second input feature is more effective.

[0043] As an example, the target analysis object is a document image to be analyzed, the first input feature is a visual feature extracted based on the document image, and the second input feature is a text feature extracted based on the target text contained in the document image. The correlation condition is that both the visual feature and the text feature are extracted based on the same document image.

[0044] As an example, the target analysis object is a target video to be analyzed, and the first input feature and the second input feature are any one of the multimodal features extracted from the target video. The correlation condition is that the first input feature and the second input feature are both extracted based on the same target video. Specifically, the multimodal features include video features, image features, audio features, and text features extracted based on the target video.

[0045] As an example, the target analysis object is the target video to be analyzed, the first input feature is any one of the multimodal features extracted from the target video, the second input feature is other features associated with the target video, and the correlation condition is that both the first input feature and the second input feature are related to the target video. The multimodal features include video features, image features, audio features, and text features extracted based on the target video. The other features include user behavior features of the associated users of the target video, user social features of the associated users of the target video, and advertising features of the target video. Associated users include but are not limited to the uploader and forwarder of the target video.

[0046] Step S120: inputting the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0047] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0048] The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature;

[0049] The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature;

[0050] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0051] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0052] In the disclosed embodiment, the feature fusion network is used to fuse the features of the first input feature and the second input feature to obtain the first cross-attention feature and the second cross-attention feature. The first cross-attention feature and the second cross-attention feature both fuse the characteristics of the first input feature and the characteristics of the second input feature.

[0053] In the feature network provided by the embodiment of the present disclosure, a first self-attention sub-network is used to perform self-attention processing on a first input feature to obtain a first self-attention feature, and a second self-attention sub-network is used to perform self-attention processing on a second input feature to obtain a second self-attention feature, and then a first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and a second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature, and a second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0054] In the implementation of the present disclosure, the second attention weight parameter is the attention weight parameter used by the second self-attention subnetwork for self-attention processing of the second input feature. Specifically, the second input feature can be converted to obtain query features, key features and content features, respectively, so as to learn the second attention weight parameter according to the query features, key features and content features.

[0055] As an example, the attention weight parameter can be expressed by the following formula 1:

[0056] w=softmax(QK T) ……(Formula 1)

[0057] Among them, w represents the attention weight parameter, softmax is the normalized exponential function, Q represents the query feature, and K represents the key value feature. In this example, both Q and K can be converted from the second input feature.

[0058] In the implementation of the present disclosure, the first attention weight parameter is the attention weight parameter used by the first self-attention subnetwork to perform self-attention processing on the first input feature. Specifically, the first input feature can be converted to obtain query features, key features and content features, respectively, so as to learn the first attention weight parameter according to the query features, key features and content features.

[0059] As an example, the formulaic expression of the first attention weight parameter can refer to the above formula 1. In this example, both Q and K can be obtained by converting the first input feature.

[0060] In the disclosed embodiment, since the first cross-attention feature is obtained by cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network, it is equivalent to guiding the reorganization of the first self-attention feature through the second attention weight parameter of the second attention sub-network, so that the first cross-attention feature can integrate the characteristics of the second input feature on the basis of having the characteristics of the first input feature.

[0061] In the disclosed embodiment, since the second cross-attention feature is obtained by cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network, it is equivalent to guiding the reorganization of the second self-attention feature through the first attention weight parameter of the first attention sub-network, so that the second cross-attention feature can integrate the characteristics of the first input feature on the basis of having the characteristics of the second input feature.

[0062] The method provided by the embodiment of the present disclosure obtains a first input feature and a second input feature, and the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition; the first input feature and the second input feature are input into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network includes a first self-attention subnetwork, a second self-attention subnetwork, a first cross-attention subnetwork and a second cross-attention subnetwork; the first self-attention subnetwork is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; the second self-attention subnetwork is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; the first cross-attention subnetwork is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention subnetwork to obtain the first cross-attention feature; the second cross-attention subnetwork is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention subnetwork to obtain the second cross-attention feature. In this solution, the first input feature and the second input feature are fused based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the first input feature and the characteristics of the second input feature, which can improve the effect of feature fusion and improve model performance.

[0063] In the disclosed embodiment, by making the first cross-attention sub-network and the second self-attention sub-network share attention weight parameters, the self-attention mechanism and the cross-attention mechanism can be better combined, thereby improving the feature fusion effect, reducing the number of model parameters in the feature fusion network, and reducing the complexity of model operations.

[0064] In the embodiment of the present disclosure, in order to improve the feature fusion effect, a multi-layer feature fusion network can be set according to actual needs, and each layer of the feature fusion network can be connected in sequence.

[0065] In an optional manner of the present disclosure, when the first cross-attention sub-network performs cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain the first cross-attention feature, it is specifically used to:

[0066] Convert the first self-attention feature into the first content feature;

[0067] Performing cross attention processing based on the first content feature and a second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross attention feature;

[0068] When the second cross-attention sub-network performs cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain the second cross-attention feature, it is specifically used to:

[0069] Convert the second self-attention feature into a second content feature;

[0070] Cross-attention processing is performed based on the second content feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0071] In the implementation of the present disclosure, when cross-attention processing is performed based on the first cross-attention sub-network, the first self-attention feature can be converted into a first content feature, and then cross-attention processing is performed based on the first content feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain the first cross-attention feature.

[0072] The second attention weight parameter is determined based on the query features and key-value features obtained by transforming the second input features, which is equivalent to guiding the reorganization of the first content features through the query features and key-value features of the second attention sub-network, thereby realizing the effective combination of the second self-attention sub-network and the first cross-attention sub-network.

[0073] In the implementation of the present disclosure, when cross-attention processing is performed based on the second cross-attention sub-network, the second self-attention feature can be converted into a second content feature, and then cross-attention processing is performed based on the second content feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0074] The first attention weight parameter is determined based on the query feature and key-value feature obtained by transforming the first input feature, which is equivalent to guiding the reorganization of the second content feature through the query feature and key-value feature of the second attention sub-network, thereby realizing the effective combination of the first self-attention sub-network and the second cross-attention sub-network.

[0075] In an optional manner of the present disclosure, the feature fusion network further includes a feature fusion layer. After the first input feature and the second input feature are input into a preset feature fusion network to obtain the first cross attention feature and the second cross attention feature, the method further includes:

[0076] The first cross-attention feature and the second cross-attention feature are input into a feature fusion layer to determine a fused feature based on the first cross-attention feature and the second cross-attention feature.

[0077] In the disclosed embodiment, the feature fusion network also includes a feature fusion layer, which is connected to the first cross-attention sub-network and the second cross-attention sub-network, and is used to perform feature fusion on the first cross-attention feature output by the first cross-attention sub-network and the second cross-attention feature output by the second cross-attention sub-network to obtain a fused feature.

[0078] The fused feature is obtained based on the first cross-attention feature and the second cross-attention feature, has the characteristics of both, and can be used for subsequent analysis and processing.

[0079] In an optional manner of the present disclosure, determining a fused feature based on the first cross-attention feature and the second cross-attention feature includes:

[0080] The first cross-attention feature and the second cross-attention feature are fused to obtain a fused feature.

[0081] In the embodiment of the present disclosure, a specific processing method of the feature fusion layer may be to directly fuse the first cross-attention feature with the second cross-attention feature to obtain a fused feature.

[0082] As an example, the first cross-attention feature and the second cross-attention feature may be concatenated to obtain a fused feature.

[0083] In an optional manner of the present disclosure, determining a fused feature based on the first cross-attention feature and the second cross-attention feature includes:

[0084] Perform feature fusion on the first cross attention feature and the first input feature to obtain a first sub-fusion feature;

[0085] Perform feature fusion on the second cross attention feature and the second input feature to obtain a second sub-fusion feature;

[0086] The first sub-fusion feature is fused with the second sub-fusion feature to obtain a fused feature.

[0087] In the disclosed embodiment, the specific processing method of the feature fusion layer can also be to perform feature fusion on the first cross-attention feature and the first input feature to obtain a first sub-fusion feature, and to perform feature fusion on the second cross-attention feature and the second input feature to obtain a second sub-fusion feature, and then to perform feature fusion on the first sub-fusion feature and the second sub-fusion feature to obtain a fused feature.

[0088] Since both the first cross-attention feature and the second cross-attention feature have undergone cross-attention processing, confusion may occur between the features, and the characteristics of the first input feature contained in the first cross-attention feature and the characteristics of the second input feature contained in the second cross-attention feature may also be averaged. By performing feature fusion of the first cross-attention feature and the first input feature, the first sub-fusion feature can fuse the characteristics of the second input feature on the basis of taking into account the characteristics of the first input feature. By performing feature fusion of the second cross-attention feature and the second input feature, the second sub-fusion feature can fuse the characteristics of the first input feature on the basis of taking into account the characteristics of the second input feature. As a result, the fused feature obtained based on the fusion of the first sub-fusion feature and the second sub-fusion feature has a better feature fusion effect.

[0089] In the disclosed embodiment, by fusing the cross features after cross-attention processing with the original input features in the feature fusion layer, it is possible to increase the data transmission link in the feature fusion network, increase the complexity of the network, and improve the feature fusion effect of the feature fusion network.

[0090] As an example, a specific method of fusing the first cross-attention feature with the first input feature may be to add the feature matrix corresponding to the first cross-attention feature to the feature matrix corresponding to the first input feature, and a specific method of fusing the second cross-attention feature with the second input feature may be to add the feature matrix corresponding to the second cross-attention feature to the feature matrix corresponding to the second input feature.

[0091] As an example, the first sub-fusion feature and the second sub-fusion feature may be concatenated to obtain a fused feature.

[0092] In an optional manner of the present disclosure, the target analysis object is a document image containing target text, the first input feature is a visual feature, the second input feature is a text feature, and the correlation condition is that both the visual feature and the text feature are extracted based on the document image.

[0093] In the disclosed embodiments, the target analysis object is a document image containing target text, and the corresponding application scenario is to perform feature analysis on the document image based on feature fusion features, such as classifying the document image, performing layout analysis on the document image, identifying named entities in the target text, etc.

[0094] The correlation condition is that both the visual features and the text features are extracted based on the document image, that is, the feature fusion is limited to the visual features and text features extracted from the same document image. There is generally a certain correlation between the visual features and text features extracted from the same document image. For example, the title area in the document image is one of the key features to distinguish the category to which the document image belongs, and the text content of the title can be mined based on the image features of the document image. Therefore, the two have a certain correlation.

[0095] The feature fusion network provided in this solution is used to fuse text features and visual features, which has a good feature fusion effect. The fused features have a better effect on semantic expression, can avoid semantic confusion problems, and ensure the analysis effect of document images.

[0096] In an optional manner of the present disclosure, the above method further includes:

[0097] Extracting image features of the document image and position features of pixel points in the document image;

[0098] The image features and position features are fused to obtain visual features.

[0099] In the disclosed embodiment, image features can be extracted from the document image, and the position features of the pixels in the document image can be extracted, and the image features and the position features can be fused to obtain visual features. Since the position information in the document image is of great significance for the analysis of the document image, by fusing the position features into the visual features, it is equivalent to introducing the position information in the document image, which helps to improve the subsequent analysis effect of the document image.

[0100] As an example, the position feature may be incorporated into the channel dimension of the image feature to achieve the feature of the image feature and the position feature.

[0101] In an optional manner of the present disclosure, extracting image features of a document image includes:

[0102] Image features of document images are extracted based on the preset Convolutional Neural Networks (CNN).

[0103] In the disclosed embodiment, a layer of convolutional neural network may be preset to extract image features of the document image.

[0104] In an optional manner of the present disclosure, the above method further includes:

[0105] Perform optical character recognition (OCR) on the document image to obtain the target text;

[0106] Perform feature extraction on the target text to obtain text features.

[0107] In the disclosed embodiment, the target text can be extracted from the document image by OCR. Then, each character in the target text is encoded by a word vector (Word2Vector) model to obtain a word vector, and then the word vector is converted by a Transformer-based bidirectional encoder representation (Bidirectional Encoder Representations from Transformer, BERT) model to obtain text features.

[0108] In an optional manner of the present disclosure, the feature fusion network is connected to a fully connected layer, and the method further includes:

[0109] The fused features are input to the fully connected layer to obtain any of the following outputs of the fully connected layer:

[0110] Classification results of document images;

[0111] Named entity recognition results in the target text;

[0112] Layout analysis results of document images.

[0113] In the disclosed embodiment, the feature fusion network is connected to a fully connected layer, and the fused features are input into the fully connected layer to obtain the output result of the fully connected layer, that is, the analysis result of the document image.

[0114] Specifically, the output of the fully connected layer can include any of the following:

[0115] Classification results of document images;

[0116] Named entity recognition results in the target text;

[0117] Layout analysis results of document images.

[0118] As an example, the present disclosure provides a document image classification method, and the specific process is as follows:

[0119] 1. Feature extraction

[0120] First, obtain the document image I, I∈R h×w×3 , the width of the document image is w pixels and the height is h pixels. The document image can have three channels: red, green, and blue (redgreenblue, RGB).

[0121] The document image is processed through a layer of convolutional neural network. The convolution kernel of the convolutional neural network is K, K∈R r×r×3×(d-1) , where d-1 is the number of convolution kernels. After convolution processing, the image feature F is obtained.

[0122] According to the position feature P of each pixel in the document image, Among them, P i.j Represents the i-th row and j-th column of the image feature F, whose value is equal to

[0123] Combine the image feature F with the position feature P to obtain the complete visual feature: V = {F, P}, where

[0124] Extract n lines of target text from the document image through OCR processing. Use the word2vector model to encode each word in the target text into a d-dimensional vector. Then use the BERT model to encode the feature vector sequence of each line to obtain the text feature T, T = {BERT (t a ),a∈[1,n]},T∈R n×d , where BERT represents the loss function of the BERT model and a represents any line of the target text.

[0125] 2. Feature Fusion

[0126] Reduce the first two dimensions of the visual feature V to obtain the visual feature V'. This makes the dimensions of visual features equal to those of text features, facilitating cross-attention processing.

[0127] The feature fusion network provided by the present disclosure is used to fuse the visual feature V' and the text feature T to obtain the first cross-attention feature V x And get the second cross attention feature T x .

[0128] The first cross-attention feature V x And get the second cross attention feature T xAfter splicing, we get the fused feature M, M = {V x ,T x},

[0129] The fused feature I, I = concat (V, T), where concat represents the loss function of the feature fusion network.

[0130] 3. Classification of Document Images

[0131] After the feature fusion network, the fully connected layer (FC) is connected to map the feature M to a vector of the corresponding number of categories. The softmax function is used to map the vector into a probability distribution score, scores = softmax(FC(M)), where softmax represents the softmax function and FC represents the loss function of the fully connected layer. The category with the highest probability value can be taken as the predicted category cls of the document. cls = argmax(scores), where argmax represents the argmax function, which is used to find the maximum value of the probability distribution.

[0132] As an example, Figure 2 A flow chart of a specific implementation of the feature fusion method provided in an embodiment of the present disclosure is shown in FIG.

[0133] like Figure 2 As shown in , the visual features will be converted into query features Vq, key features Vk and content features Vv. Based on the query features Vq, key features Vk and content features Vv, self-attention processing is performed in the first self-attention sub-network to obtain the first self-attention feature Vn. The first self-attention weight parameter Mv is the attention weight parameter when performing self-attention processing in the first self-attention sub-network. The first self-attention weight parameter Mv can be calculated based on the query features Vq and the key features Vk.

[0134] The text features will be converted into query features Tq, key features Tk and content features TT. Based on the query features Tq, key features Tk and content features TT, the self-attention processing in the second self-attention sub-network is performed to obtain the second self-attention feature Tn. The second self-attention weight parameter Mt is the attention weight parameter when the self-attention processing is performed in the second self-attention sub-network. The second self-attention weight parameter Mt can be calculated based on the query features Tq and the key features Tk.

[0135] Based on the first self-attention feature Vn and the second self-attention weight parameter Mt, cross-self-attention processing is performed in the first cross-attention sub-network to obtain the first cross-attention feature Vo.

[0136] Based on the second attention feature Tn and the first self-attention weight parameter Mv, cross-self-attention processing is performed in the second cross-attention sub-network to obtain the second cross-attention feature To.

[0137] By fusing the first cross-attention feature Vo with the visual feature, the first sub-fusion feature Vf can be obtained.

[0138] By fusing the second cross-attention feature To with the text feature, the second sub-fusion feature Tf can be obtained.

[0139] By fusing the first sub-fusion feature Vf with the second sub-fusion feature Tf, the fused feature I can be obtained.

[0140] Figure 3 A schematic diagram of a document image classification method provided by an embodiment of the present disclosure is shown. Figure 3 As shown in , the method can mainly include:

[0141] Step S310: extracting image features of the document image and position features of pixel points in the document image, and fusing the image features with the position features to obtain visual features;

[0142] Step S320: performing OCR processing on the document image to obtain target text, and performing feature extraction on the target text to obtain text features;

[0143] Step S330: inputting the visual features and the text features into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0144] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0145] The first self-attention sub-network is used to perform self-attention processing on the visual features to obtain the first self-attention features;

[0146] The second self-attention sub-network is used to perform self-attention processing on the text features to obtain the second self-attention features;

[0147] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0148] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature;

[0149] Step S340: Fusing the first cross-attention feature with the visual feature to obtain a first sub-fusion feature;

[0150] Step S350: Fusing the second cross-attention feature with the text feature to obtain a second sub-fusion feature;

[0151] Step S360: Fusing the first sub-fusion feature with the second sub-fusion feature to obtain a fused feature;

[0152] Step S370: Input the fused features into the fully connected layer to obtain the classification result of the document image output by the fully connected layer.

[0153] In the embodiment of the present disclosure, visual features and text features can be respectively extracted from a document image containing a target text, and the visual features and text features are fused based on the feature fusion network provided in the embodiment of the present disclosure to obtain fused features. The fused features fuse the characteristics of the first input features and the characteristics of the second input features, so that feature analysis of the document image can be performed based on the fused features to classify the document image.

[0154] In the disclosed embodiment, a layer of convolutional neural network may be preset to extract image features of the document image.

[0155] In the disclosed embodiment, image features can be extracted from the document image, and the position features of the pixels in the document image can be extracted, and the image features and the position features can be fused to obtain visual features. Since the position information in the document image is of great significance for the analysis of the document image, by fusing the position features into the visual features, it is equivalent to introducing the position information in the document image, which helps to improve the subsequent analysis effect of the document image.

[0156] As an example, the position feature may be incorporated into the channel dimension of the image feature to achieve the feature of the image feature and the position feature.

[0157] In the disclosed embodiment, the target text can be extracted from the document image by OCR, and then each character in the target text is encoded by the Word2Vector model to obtain a word vector, and then the word vector is converted by the BERT model to obtain text features.

[0158] In the feature network provided by the embodiment of the present disclosure, the first self-attention sub-network is used to perform self-attention processing on the visual feature to obtain the first self-attention feature, and the second self-attention sub-network is used to perform self-attention processing on the text feature to obtain the second self-attention feature, and then the first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain the first cross-attention feature, and the second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain the second cross-attention feature.

[0159] In the implementation of the present disclosure, the second attention weight parameter is the attention weight parameter used by the second self-attention subnetwork for self-attention processing of text features. Specifically, the text features can be converted to obtain query features, key features and content features, respectively, so as to learn the second attention weight parameter according to the query features, key features and content features.

[0160] In the implementation of the present disclosure, the first attention weight parameter is the attention weight parameter used by the first self-attention subnetwork for self-attention processing of visual features. Specifically, the visual features can be converted to obtain query features, key features and content features, respectively, so as to learn the first attention weight parameter according to the query features, key features and content features.

[0161] In the disclosed embodiment, since the first cross-attention feature is obtained by cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network, it is equivalent to guiding the reorganization of the first self-attention feature through the second attention weight parameter of the second attention sub-network, so that the first cross-attention feature can integrate the characteristics of text features on the basis of the characteristics of visual frequency features.

[0162] In the disclosed embodiment, since the second cross-attention feature is obtained by cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network, it is equivalent to guiding the reorganization of the second self-attention feature through the first attention weight parameter of the first attention sub-network, so that the second cross-attention feature can integrate the characteristics of visual features on the basis of the characteristics of text features.

[0163] The method provided by the embodiment of the present disclosure extracts visual features and text features from the document image respectively, inputs the visual features and text features into a preset feature fusion network, and obtains a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network includes a first self-attention subnetwork, a second self-attention subnetwork, a first cross-attention subnetwork, and a second cross-attention subnetwork; the first self-attention subnetwork is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; the second self-attention subnetwork is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; the first cross-attention subnetwork is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention subnetwork to obtain the first cross-attention feature; the second cross-attention subnetwork is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention subnetwork to obtain the second cross-attention feature. In this scheme, feature fusion processing is performed on visual features and text based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the visual features and the characteristics of the text features, which can improve the effect of feature fusion and help improve the accuracy of the subsequent classification results of the document image.

[0164] In the disclosed embodiment, by making the first cross-attention sub-network and the second self-attention sub-network share attention weight parameters, the self-attention mechanism and the cross-attention mechanism can be better combined, thereby improving the feature fusion effect, reducing the number of model parameters in the feature fusion network, and reducing the complexity of model operations.

[0165] In the embodiment of the present disclosure, in order to improve the feature fusion effect, a multi-layer feature fusion network can be set according to actual needs, and each layer of the feature fusion network can be connected in sequence.

[0166] In the embodiment of the present disclosure, the specific processing method of the feature fusion layer can be to perform feature fusion of the first cross-attention feature with the visual feature to obtain a first sub-fusion feature, and to perform feature fusion of the second cross-attention feature with the text feature to obtain a second sub-fusion feature, and then perform feature fusion of the first sub-fusion feature with the second sub-fusion feature to obtain a fused feature.

[0167] Since both the first cross-attention feature and the second cross-attention feature have undergone cross-attention processing, confusion may occur between the features, and the characteristics of the visual features contained in the first cross-attention feature and the characteristics of the text features contained in the second cross-attention feature may also be averaged. By fusing the first cross-attention feature with the visual feature, the first sub-fusion feature can fuse the characteristics of the text feature on the basis of taking into account the characteristics of the visual feature. By fusing the second cross-attention feature with the second input feature, the second sub-fusion feature can fuse the characteristics of the visual feature on the basis of taking into account the characteristics of the text feature. As a result, the fused feature obtained based on the fusion of the first sub-fusion feature and the second sub-fusion feature has a better feature fusion effect.

[0168] In the disclosed embodiment, by fusing the cross features after cross-attention processing with the original input features in the feature fusion layer, it is possible to increase the data transmission link in the feature fusion network, increase the complexity of the network, and improve the feature fusion effect of the feature fusion network.

[0169] As an example, a specific method for fusing the first cross-attention feature with the visual feature may be to add the feature matrix corresponding to the first cross-attention feature to the feature matrix corresponding to the visual feature, and a specific method for fusing the second cross-attention feature with the text feature may be to add the feature matrix corresponding to the second cross-attention feature to the feature matrix corresponding to the text feature.

[0170] As an example, the first sub-fusion feature and the second sub-fusion feature may be concatenated to obtain a fused feature.

[0171] In the disclosed embodiment, the feature fusion network is connected to a fully connected layer, and the fused features are input to the fully connected layer to obtain the classification result of the document image output by the fully connected layer.

[0172] Based on Figure 1 The same principle as shown in the method, Figure 4 A structural schematic diagram of a feature fusion device provided by an embodiment of the present disclosure is shown. Figure 4 As shown, the feature fusion device 30 may include:

[0173] The feature acquisition module 410 is used to acquire a first input feature and a second input feature, wherein the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition;

[0174] A feature fusion module 420, configured to input the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0175] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0176] The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature;

[0177] The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature;

[0178] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0179] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0180] The device provided by the embodiment of the present disclosure obtains a first input feature and a second input feature, and the correlation between the first input feature and the second input feature and the target analysis object meets a preset correlation condition; the first input feature and the second input feature are input into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network and a second cross-attention sub-network; the first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain the first self-attention feature; the second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain the second self-attention feature; the first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain the first cross-attention feature; the second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain the second cross-attention feature. In this solution, the first input feature and the second input feature are fused based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the first input feature and the characteristics of the second input feature, which can improve the effect of feature fusion and improve model performance.

[0181] Optionally, when the first cross-attention sub-network performs cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain the first cross-attention feature, it is specifically used to:

[0182] Convert the first self-attention feature into the first content feature;

[0183] Cross-attention processing is performed based on the first content feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature.

[0184] When the second cross-attention sub-network performs cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain the second cross-attention feature, it is specifically used to:

[0185] Convert the second self-attention feature into a second content feature;

[0186] Cross-attention processing is performed based on the second content feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature.

[0187] Optionally, the feature fusion network further includes a feature fusion layer, and the above device further includes:

[0188] The feature fusion layer processing module is used to input the first input feature and the second input feature into a preset feature fusion network to obtain the first cross-attention feature and the second cross-attention feature, and then input the first cross-attention feature and the second cross-attention feature into the feature fusion layer to determine the fused feature based on the first cross-attention feature and the second cross-attention feature.

[0189] Optionally, when determining the fused feature based on the first cross-attention feature and the second cross-attention feature, the feature fusion layer processing module is specifically used to:

[0190] The first cross-attention feature and the second cross-attention feature are fused to obtain a fused feature.

[0191] Optionally, when determining the fused feature based on the first cross-attention feature and the second cross-attention feature, the feature fusion layer processing module is specifically used to:

[0192] Perform feature fusion on the first cross attention feature and the first input feature to obtain a first sub-fusion feature;

[0193] Perform feature fusion on the second cross attention feature and the second input feature to obtain a second sub-fusion feature;

[0194] The first sub-fusion feature is fused with the second sub-fusion feature to obtain a fused feature.

[0195] Optionally, the target analysis object is a document image containing target text, the first input feature is a visual feature, the second input feature is a text feature, and the correlation condition is that both the visual feature and the text feature are extracted based on the document image.

[0196] Optionally, the above device further comprises a visual feature extraction module, and the visual feature extraction module is used to:

[0197] Extracting image features of the document image and position features of pixel points in the document image;

[0198] The image features and position features are fused to obtain visual features.

[0199] Optionally, when extracting image features of a document image, the visual feature extraction module is specifically used to:

[0200] The image features of the document image are extracted based on the preset convolutional neural network.

[0201] Optionally, the above device further includes a text feature extraction module, and the text feature extraction module is used to:

[0202] Perform OCR processing on the document image to obtain the target text;

[0203] Perform feature extraction on the target text to obtain text features.

[0204] Optionally, the feature fusion network is connected to a fully connected layer, and the apparatus further comprises:

[0205] The fully connected layer processing module is used to input the fused features into the fully connected layer to obtain any of the following outputs of the fully connected layer:

[0206] Classification results of document images;

[0207] Named entity recognition results in the target text;

[0208] Layout analysis results of document images.

[0209] It can be understood that the above modules of the feature fusion device in the embodiment of the present disclosure have the function of realizing Figure 1 The functions of the corresponding steps of the feature fusion method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above feature fusion device, please refer to Figure 1 The corresponding description of the feature fusion method in the embodiment shown in will not be repeated here.

[0210] Based on Figure 3 The same principle as shown in the method, Figure 5 A schematic diagram of the structure of a document image classification device provided by an embodiment of the present disclosure is shown. Figure 5 As shown, the feature fusion device 50 may include:

[0211] The feature extraction module 510 is used to extract the image features of the document image and the position features of the pixels in the document image, and to fuse the image features with the position features to obtain visual features; and to perform OCR processing on the document image to obtain the target text, and to perform feature extraction on the target text to obtain text features;

[0212] A feature fusion module 520, used for inputting the visual features and the text features into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature;

[0213] The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network;

[0214] The first self-attention sub-network is used to perform self-attention processing on the visual features to obtain the first self-attention features;

[0215] The second self-attention sub-network is used to perform self-attention processing on the text features to obtain the second self-attention features;

[0216] The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature;

[0217] The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature;

[0218] The feature fusion layer processing module 530 is used to perform feature fusion on the first cross-attention feature and the visual feature to obtain a first sub-fusion feature, perform feature fusion on the second cross-attention feature and the text feature to obtain a second sub-fusion feature, and perform feature fusion on the first sub-fusion feature and the second sub-fusion feature to obtain a fused feature;

[0219] The fully connected layer processing module 540 is used to input the fused features into the fully connected layer to obtain the classification result of the document image output by the fully connected layer.

[0220] The method provided by the embodiment of the present disclosure extracts visual features and text features from the document image respectively, inputs the visual features and text features into a preset feature fusion network, and obtains a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network includes a first self-attention subnetwork, a second self-attention subnetwork, a first cross-attention subnetwork, and a second cross-attention subnetwork; the first self-attention subnetwork is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; the second self-attention subnetwork is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; the first cross-attention subnetwork is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention subnetwork to obtain the first cross-attention feature; the second cross-attention subnetwork is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention subnetwork to obtain the second cross-attention feature. In this scheme, feature fusion processing is performed on visual features and text based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the visual features and the characteristics of the text features, which can improve the effect of feature fusion and help improve the accuracy of the subsequent classification results of the document image.

[0221] It can be understood that the above modules of the document image classification device in the embodiment of the present disclosure have the function of realizing Figure 3 The functions of the corresponding steps of the document image classification method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above feature fusion device, please refer to Figure 3 The corresponding description of the document image classification method in the embodiment shown in will not be repeated here.

[0222] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0223] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0224] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the feature fusion method provided in the embodiment of the present disclosure.

[0225] Compared with the prior art, the electronic device obtains a first input feature and a second input feature, and the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition; the first input feature and the second input feature are input into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network includes a first self-attention subnetwork, a second self-attention subnetwork, a first cross-attention subnetwork and a second cross-attention subnetwork; the first self-attention subnetwork is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; the second self-attention subnetwork is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; the first cross-attention subnetwork is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention subnetwork to obtain the first cross-attention feature; the second cross-attention subnetwork is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention subnetwork to obtain the second cross-attention feature. In this solution, the first input feature and the second input feature are fused based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the first input feature and the characteristics of the second input feature, which can improve the effect of feature fusion and improve model performance.

[0226] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the feature fusion method provided in the embodiment of the present disclosure.

[0227] Compared with the prior art, the readable storage medium obtains the first input feature and the second input feature, and the correlation between the first input feature and the second input feature and the target analysis object satisfies the preset correlation condition; the first input feature and the second input feature are input into a preset feature fusion network to obtain the first cross-attention feature and the second cross-attention feature; wherein the feature fusion network includes a first self-attention subnetwork, a second self-attention subnetwork, a first cross-attention subnetwork and a second cross-attention subnetwork; the first self-attention subnetwork is used to perform self-attention processing on the first input feature to obtain the first self-attention feature; the second self-attention subnetwork is used to perform self-attention processing on the second input feature to obtain the second self-attention feature; the first cross-attention subnetwork is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention subnetwork to obtain the first cross-attention feature; the second cross-attention subnetwork is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention subnetwork to obtain the second cross-attention feature. In this solution, the first input feature and the second input feature are fused based on the feature fusion network to obtain the first cross-attention feature and the second cross-attention feature that fuse the characteristics of the first input feature and the characteristics of the second input feature, which can improve the effect of feature fusion and improve model performance.

[0228] Figure 6 A schematic block diagram of an example electronic device 60 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0229] like Figure 6 As shown, the electronic device 60 includes a computing unit 610, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 620 or a computer program loaded from a storage unit 680 into a random access memory (RAM) 630. In the RAM 630, various programs and data required for the operation of the device 60 can also be stored. The computing unit 610, the ROM 620, and the RAM 630 are connected to each other via a bus 640. An input / output (I / O) interface 650 is also connected to the bus 640.

[0230] A number of components in the device 60 are connected to the I / O interface 650, including: an input unit 660, such as a keyboard, a mouse, etc.; an output unit 670, such as various types of displays, speakers, etc.; a storage unit 680, such as a disk, an optical disk, etc.; and a communication unit 690, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 690 allows the device 60 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0231] The computing unit 610 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 610 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 610 performs the feature fusion method provided in the embodiment of the present disclosure. For example, in some embodiments, the feature fusion method provided in the embodiment of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 680. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 60 via the ROM 620 and / or the communication unit 690. When the computer program is loaded into the RAM 630 and executed by the computing unit 610, one or more steps of the feature fusion method provided in the embodiment of the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 610 may be configured to perform the feature fusion method provided in the embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0232] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0234] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0235] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0236] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0237] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0238] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0239] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A feature fusion method, comprising: Obtaining a first input feature and a second input feature, wherein the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition; Inputting the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network further includes a feature fusion layer; Inputting the first cross-attention feature and the second cross-attention feature into the feature fusion layer to determine a fused feature based on the first cross-attention feature and the second cross-attention feature; The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network; wherein the first self-attention sub-network is connected to the second cross-attention sub-network, and the second self-attention sub-network is connected to the first cross-attention sub-network; The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature; the first cross-attention sub-network is specifically used to convert the first self-attention feature into a first content feature; based on the first content feature and the second attention weight parameter corresponding to the second self-attention sub-network, cross-attention processing is performed to obtain the first cross-attention feature; wherein the second attention weight parameter is determined based on the query feature and the key-value feature obtained by converting the second input feature; The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature; the second cross-attention sub-network is specifically used to convert the second self-attention feature into a second content feature; based on the second content feature and the first attention weight parameter corresponding to the first self-attention sub-network, cross-attention processing is performed to obtain a second cross-attention feature; wherein the first attention weight parameter is determined based on the query feature and the key-value feature obtained by converting the first input feature; The target analysis object is a document image containing target text, the first input feature is a visual feature, the second input feature is a text feature, and the correlation condition is that both the visual feature and the text feature are extracted based on the document image.

2. The method according to claim 1, wherein The determining of a fused feature based on the first cross-attention feature and the second cross-attention feature includes: The first cross-attention feature and the second cross-attention feature are fused to obtain a fused feature.

3. The method according to claim 1, wherein The determining of a fused feature based on the first cross-attention feature and the second cross-attention feature includes: Performing feature fusion on the first cross-attention feature and the first input feature to obtain a first sub-fusion feature; Fusing the second cross-attention feature with the second input feature to obtain a second sub-fusion feature; The first sub-fusion feature and the second sub-fusion feature are fused to obtain a fused feature.

4. The method according to any one of claims 1 to 3, further comprising: Extracting image features of the document image and position features of pixels in the document image; The image feature and the position feature are fused to obtain the visual feature.

5. The method according to claim 4, wherein The extracting image features of the document image includes: The image features of the document image are extracted based on a preset convolutional neural network.

6. The method according to any one of claims 1 to 3, further comprising: Performing optical character recognition (OCR) processing on the document image to obtain the target text; Feature extraction is performed on the target text to obtain the text features.

7. The method according to any one of claims 1 to 3, wherein The feature fusion network is connected to a fully connected layer, and the method further includes: Input the fused features into the fully connected layer to obtain any of the following outputs of the fully connected layer: a classification result of the document image; Named entity recognition results in the target text; The layout analysis result of the document image.

8. A feature fusion device, comprising: A feature acquisition module, configured to acquire a first input feature and a second input feature, wherein the correlation between the first input feature and the second input feature and the target analysis object satisfies a preset correlation condition; a feature fusion module, configured to input the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature; wherein the feature fusion network further includes a feature fusion layer; a feature fusion layer processing module, configured to, after inputting the first input feature and the second input feature into a preset feature fusion network to obtain a first cross-attention feature and a second cross-attention feature, input the first cross-attention feature and the second cross-attention feature into the feature fusion layer, so as to determine a fused feature based on the first cross-attention feature and the second cross-attention feature; The feature fusion network includes a first self-attention sub-network, a second self-attention sub-network, a first cross-attention sub-network, and a second cross-attention sub-network; wherein the first self-attention sub-network is connected to the second cross-attention sub-network, and the second self-attention sub-network is connected to the first cross-attention sub-network; The first self-attention sub-network is used to perform self-attention processing on the first input feature to obtain a first self-attention feature; The second self-attention sub-network is used to perform self-attention processing on the second input feature to obtain a second self-attention feature; The first cross-attention sub-network is used to perform cross-attention processing based on the first self-attention feature and the second attention weight parameter corresponding to the second self-attention sub-network to obtain a first cross-attention feature; the first cross-attention sub-network is specifically used to convert the first self-attention feature into a first content feature; based on the first content feature and the second attention weight parameter corresponding to the second self-attention sub-network, cross-attention processing is performed to obtain the first cross-attention feature; wherein the second attention weight parameter is determined based on the query feature and the key-value feature obtained by converting the second input feature; The second cross-attention sub-network is used to perform cross-attention processing based on the second self-attention feature and the first attention weight parameter corresponding to the first self-attention sub-network to obtain a second cross-attention feature; the second cross-attention sub-network is specifically used to convert the second self-attention feature into a second content feature; based on the second content feature and the first attention weight parameter corresponding to the first self-attention sub-network, cross-attention processing is performed to obtain a second cross-attention feature; wherein the first attention weight parameter is determined based on the query feature and the key-value feature obtained by converting the first input feature; The target analysis object is a document image containing target text, the first input feature is a visual feature, the second input feature is a text feature, and the correlation condition is that both the visual feature and the text feature are extracted based on the document image.

9. The device according to claim 8, wherein When determining a fused feature based on the first cross-attention feature and the second cross-attention feature, the feature fusion layer processing module is specifically configured to: The first cross-attention feature and the second cross-attention feature are fused to obtain a fused feature.

10. The device according to claim 8, wherein When determining a fused feature based on the first cross-attention feature and the second cross-attention feature, the feature fusion layer processing module is specifically configured to: Performing feature fusion on the first cross-attention feature and the first input feature to obtain a first sub-fusion feature; Fusing the second cross-attention feature with the second input feature to obtain a second sub-fusion feature; The first sub-fusion feature and the second sub-fusion feature are fused to obtain a fused feature.

11. The apparatus according to any one of claims 8 to 10, further comprising a visual feature extraction module, wherein the visual feature extraction module is configured to: Extracting image features of the document image and position features of pixels in the document image; The image feature and the position feature are fused to obtain the visual feature.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Image definition recognition method and device, electronic equipment and storage medium

    CN113837965A

  • Entity linking method and device and electronic equipment

    CN114638231A

  • X-ray chest radiography diagnosis report generation method based on multi-task multi-mode deep learning

    CN115223678A

  • Audio-visual fusion with cross-modal attention for video action recognition

    WO2021184026A1