Multimodal text page classification method based on decoupled feature guidance

Abstract

The present invention relates to a multimodal text page classification method based on decoupled feature guidance, comprising the following steps: obtaining electronic text page data to construct a training set, and training a dual-stream end-to-end Transformer encoding network based on the training set, wherein the dual-stream end-to-end Transformer encoding network comprises a visual feature extractor, a language feature extractor, and a cross-modal fusion encoder; inputting text data of the electronic text page to be classified into the language feature extractor to obtain deep language features; inputting image data of the electronic text page to be classified into the visual feature extractor to obtain deep visual features; inputting the obtained deep language features and deep visual features into the cross-modal fusion encoder to obtain multimodal features, and using the classification network for classification. The present invention can effectively and accurately classify electronic text pages through a multimodal method.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition and computer vision, and in particular to a multimodal text page classification method based on decoupling feature guidance. Background Art

[0002] In recent years, with the widespread popularity of smart devices such as tablets and mobile phones, electronic text pages have gradually become one of the most important information media. The various information displayed on electronic text pages are in different formats, types, and sources, making it difficult for a single modality method to effectively analyze comprehensive features. In addition, the document image classification task in current real-world scenarios is the most similar visual task to the electronic text page classification, but the electronic text page image includes the types of documents and is not limited to this. Previous researchers also used multimodal methods to classify document image classification tasks by extracting text from documents, but there is currently little research on electronic text pages.

[0003] Although computer vision technology has made great progress, it still faces many challenges in electronic text page scenarios, such as the lack of information relevance and complementarity of single modal features, and the cost problem of model learning image detail features. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a multimodal text page classification method based on decoupling feature guidance, which can effectively and accurately classify electronic text pages through a multimodal method.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A multimodal text page classification method based on decoupled feature guidance includes the following steps:

[0007] Acquire electronic text page data to construct a training set, and train a dual-stream end-to-end Transformer encoding network based on the training set, wherein the dual-stream end-to-end Transformer encoding network includes a visual feature extractor, a language feature extractor, and a cross-modal fusion encoder;

[0008] Inputting text data of the electronic text page to be classified into a language feature extractor to obtain deep language features;

[0009] Inputting image data of the electronic text page to be classified into a visual feature extractor to obtain deep visual features;

[0010] The obtained deep language features and deep visual features are jointly input into the cross-modal fusion encoder to obtain multimodal features, and are used in the classification network for classification.

[0011] Furthermore, the electronic text page data includes electronic text page images and text data in the categories of economy, military, sports, food, technology, and tourism.

[0012] Furthermore, the language feature extractor includes a second decoupled representation network module and a language encoder; the method for processing data by the language feature extractor is as follows:

[0013] The text data of the electronic text page to be classified is first input into the second decoupled representation network module to obtain the shallow language feature E head and the remaining text features E tail ;

[0014] Then the shallow language feature E head The text features are further extracted through the 10-layer Transformer structure language encoder network to obtain the deep language features F t .

[0015] Furthermore, let Text = {t1, t2, …, t M} is a text data sequence processed by word segmentation, t j Represents the jth text descriptor and divides it into the head text T head and the remaining text T tail ;

[0016] Use large-scale pre-trained Bert to encode text language features; embed text sequences into language feature space The specific process is expressed as:

[0017]

[0018] Where Bert(·) is the pre-trained Bert weight, is the text sequence to be embedded, The positions and semantic types of the embedding vectors are added in the embedding process according to the Bert processing method respectively;

[0019] The text embedding method in S22 is used to embed the head text T head and the remaining text T tail Sharing the same Bert weight embedding feature space to obtain is a shallow language feature. are the remaining text features;

[0020] The remaining features and shallow language features are respectively input into the guided alignment module and the language encoding module for processing.

[0021] Furthermore, the visual feature extractor includes a first decoupled representation network module, a guided alignment module and a visual encoder; the method for processing data by the visual feature extractor is as follows:

[0022] The image data of the electronic text page to be classified is processed by the first decoupled representation network module to decouple and obtain the shape feature f shape 、Color feature f hsv and pixel feature f gray ;

[0023] The obtained shape feature f shape 、Color feature f hsv , pixel feature f gray and the remaining text features E obtained in S22 tail Input the guided alignment module to perform guided alignment to obtain the shallow visual feature E v ;

[0024] The shallow visual feature E v Input the visual encoder for processing, and further extract the visual features to obtain the deep visual features F v ; The self-attention layer in the conventional visual encoder is replaced by a Transformer self-attention layer with a Gaussian kernel function to generate a mask. The Gaussian attention GAttn(·) is used to obtain visual modal features that pay more attention to the central area. The calculation formula is as follows:

[0025]

[0026] where μ r ,μ l ,σ is a hyperparameter,μ r ,μ l represents the mean of the Gaussian function, σ represents the standard deviation of the Gaussian function, W q ,W k ,W θ are the Q value and K value in the self-attention mechanism and the learnable weight parameter set for the Gaussian weight, r and l are the position indexes of the feature points in the feature map, and T is the matrix transpose symbol.

[0027] Furthermore, let I = {x1, x2, …, x N} is to obtain the electronic text page image training set from the Internet, x i For the i-th image, the text page image is sliced ​​to obtain the image segment sequence X i =Patch(x i ), Patch(·) is a convolution kernel with a size of 32×64 and a step size of (32, 64); and the image fragment sequence X iThe channel information is removed and the pixel feature sequence f is used as the two-dimensional image. gray ;

[0028] For image x i The image is processed by Gaussian kernels G(·) of different sizes, and a Gaussian difference pyramid DOG(·) is obtained; the SIFT method is used to extract the image extreme feature point descriptors keypints in the Gaussian difference pyramid, and the principal component analysis method PCA(·) is used to obtain the 128-dimensional eigenvalue SIFT val and the feature vector SIFT vec ; Input the feature value sequence into a fully connected network to extract it into a one-dimensional image shape feature sequence of a specific length f shape , and its calculation formula is as follows:

[0029] DOG(x i ,σ)=(G(x i ,i*σ)-G(x i ,σ))*x i

[0030] SIFT val ,SIFT vec = PCA(keypints{(x i ,y i )|∈DOG(I k ,σ)},128)

[0031] f shape =w sp SIFT val +b sp

[0032] Where σ is the Gaussian blur coefficient, W sp ,b sp is the weight of the fully connected layer;

[0033] By extracting the image x i The HSV features of the feature value sequence are mapped into a one-dimensional color feature sequence f of a specific length through a fully connected network. hsv , and its calculation formula is as follows:

[0034] HSV=CalcHist(H+S+V)

[0035] f hsv =w hsv HSV+b hsv

[0036] Where H, S, V represent the image hue, saturation, and brightness respectively, CalcHist(·) represents the color histogram statistical function, and w hsv ,b hsvis the weight of the fully connected layer;

[0037] The shape features, color features and pixel features are input into the guided alignment module for processing, and the decoupled features are used to guide the fusion to replace the traditional visual features.

[0038] Furthermore, the shape features f shape 、Color feature f hsv , pixel feature f gray Embed the feature space so that the pixel feature space is guided by the shape and color feature space, and the feature space E1 is expressed as:

[0039] E1=γ1*LN(W g ·f gray )+γ2*LN(W sh ·f shape )+γ3*LN(W c ·f hsv )

[0040] Where W g ,W sh ,W C They represent the MLP weights, γ1, γ2, and γ3 are f gray , f shape and f hsv Feature weighting hyperparameter, LN(·) represents a LayerNorm normalization layer;

[0041] The remaining text features E tail Through 1×1 channel convolution, the dimension of the visual feature E1 is aligned, and the input of a self-attention layer is concatenated with E1 for semantic alignment to obtain the shallow visual feature Ev to replace the existing merged feature. The calculation formula is as follows:

[0042] E v =selfAttn([E1,Conv 1x1 (E tail )])

[0043] Where selfAttn(·) is the standard self-attention module, Conv 1x1 (·) represents 1×1 channel convolution;

[0044] The shallow visual feature E v Input the visual encoder for processing.

[0045] Furthermore, the method for processing data by the cross-modal fusion encoder is as follows:

[0046] The obtained deep language feature F t and deep visual features F vThe cross-modal fusion encoder is input together to extract the cross-modal feature F through 6 layers of cross-attention encoding. x ;

[0047] The cross-modal feature F x Input the classification head network for classification; the classification head network consists of three fully connected layers with a dimension size of 1, and two fully connected layers are connected through the Mish activation function and the BatchNorm normalization layer. The classification head finally outputs a prediction sequence with a dimension of the number of categories through softmax, which represents the predicted probability of each category corresponding to the i-th electronic text page sample, and selects the category with the highest probability as the classification result of the sample.

[0048] Furthermore, the training of the dual-stream end-to-end Transformer encoding network specifically includes

[0049] (1): Obtain electronic text page data to build a training set;

[0050] (2): The visual modality feature F v and the language modality feature F t Calculate the inter-modal loss L through infoNCE loss CM , the shape feature f shape and color feature f hsv Used to calculate the intra-modal loss L IM , and its calculation formula is as follows:

[0051]

[0052]

[0053]

[0054] where F i ,F i ' represents the two features under the i-th picture, and F' represents the features of all pictures F i 'Collection of ', F' id represents the idth feature in F', sim(·) is the similarity calculation function of two feature sequences, F v ,F t Respectively represent the visual modality features and language modality features of the current image, Respectively represent the set of all image visual modality features and language modality features, f shape ,f hsv Respectively represent the shape features and color features of the current image, are the sets of shape features and color features of all images respectively, and e is the natural index;

[0055] (3): The classification probability output by the classification head is used to calculate the difference between the predicted classification probability and the true probability using the cross entropy loss function and obtain the loss value L BCE , and through the inter-modal loss L CM and the intra-modal loss L IM Weighted calculation of the loss of the entire network Loss = αL BCE +βL CM +γL IM , where α, β, and γ are hyperparameters; the network parameters are then updated using the gradient descent and back-propagation algorithms to complete the dual-stream end-to-end Transformer encoding network training.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention can effectively analyze the comprehensive features in the electronic text page and improve the accuracy of classification. By combining the self-supervised contrast loss to measure the feature space difference between the encoders of the two modalities and the difference between the shape features and color features in the visual modality, the loss optimization is used to shorten the feature distance between the modalities, and the shape features and color features are always guided to the visual features in a similar direction. The constraints can be gradually strengthened during the model training process, and the generalization of the features can be improved.

[0058] 2. Aiming at the problem of model learning the cost of detailed features of electronic text page images, the present invention proposes a decoupled representation module, which reduces the feature dimension and retains effective features by decoupling the strong correlation of features. In combination with the visual feature optimization method, a multimodal Transformer electronic text page classification method is proposed. The multimodal method overcomes the problem that single modal features lack the correspondence between different modalities and are difficult to identify difficult samples by combining information from different angles.

[0059] 3. To address the problem of lack of image detail information when summarizing visual features in text areas, the present invention proposes a method to guide the representation module to fuse shape features, color features and pixel features, enhance the qualitative understanding of image features for image areas and the quantitative understanding of text distribution, enhance the learning ability of the multimodal model in the visual part, and greatly enhance the ability to distinguish visual entity contours in electronic text page images.

[0060] 4. In order to solve the problem that it is difficult to fuse a large amount of text with visual features in electronic text page scenes, the present invention pre-aligns the remaining text features with the shared Bert weights in the guided representation module, solves the problem that the visual features lack text semantics, and enables the visual encoder to have key text perception characteristics. The optimization on Bert is affected by both text features and visual features, thereby improving the ability to visually understand the meaning of the text.

[0061] 5. The present invention uses Gaussian attention to extract visual modal features in visual Transformer coding, which increases the model's attention to the central area of ​​the image and effectively improves the classification effect. During training, visual feature masks that fit the Gaussian distribution are randomly generated to achieve the effect of data enhancement, enrich data diversity and increase model robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] Please refer to Figure 1 The present invention provides a multimodal text page classification method based on decoupling feature guidance, comprising the following steps: (steps S2 and S3 can be performed simultaneously)

[0065] Step S1: train a two-stream end-to-end Transformer encoding network through a multimodal electronic text page image and text training set and infer the electronic text page category through the trained network to detect the type of electronic text page browsed by the user;

[0066] In this embodiment, step S1 specifically includes the following steps:

[0067] Step S11: obtaining a training set of electronic text page images and text data classified into categories such as economy, military, sports, food, technology, and tourism from the Internet for end-to-end multimodal Transformer encoding network training;

[0068] Preferably, the end-to-end multimodal Transformer encoding network training specifically includes the following steps:

[0069] Step S111: obtaining a multimodal electronic text page training set from the Internet;

[0070] Step S112: The visual modality feature F obtained in S34 v and the language modality feature F obtained in S23 t Calculate the inter-modal loss L through infoNCE loss CM , the shape features f obtained by S312 and S313 shape and color feature f hsv Used to calculate the intra-modal loss L IM , and its calculation formula is as follows:

[0071]

[0072]

[0073]

[0074] Among them, F i ,F i ' represents the two features under the i-th picture, and F' represents the features of all pictures F i 'Collection of ', F' id represents the idth feature in F', sim(·) is the similarity calculation function of two feature sequences, F v ,F t Respectively represent the visual modality features and language modality features of the current image, Respectively represent the set of all image visual modality features and language modality features, f shape ,f hsv Respectively represent the shape features and color features of the current image, are the sets of shape features and color features of all images respectively, and e is the natural index;

[0075] Step S113: Use the cross entropy loss function to calculate the difference between the predicted classification probability and the true probability and obtain the loss value L BCE , and through the inter-modal loss L CM and the intra-modal loss L IM Weighted calculation of the loss of the entire network Loss = αL BCE +βL CM +γL IM , where α, β, and γ are hyperparameters; then the gradient descent and back propagation algorithms are used to update the network parameters to complete the neural network training.

[0076] Step S12: The dual-stream end-to-end Transformer encoding network in S11 will consist of three main modules: visual feature extractor, language feature extractor, and cross-modal fusion encoder, and will be connected to the classification head to complete the end-to-end data output;

[0077] Step S13: Input the electronic text page image and text information to be classified into the end-to-end multimodal Transformer encoding network trained in S11, output the classification probability corresponding to each category of the electronic text page, and use the category with the highest probability as the classification result of network reasoning, which is used to detect the type of electronic text page browsed by the user.

[0078] Step S2: Based on the network architecture in S1, the electronic text page text collected in S1 is input into the language feature extractor to obtain deep language features.

[0079] In this embodiment, step S2 specifically includes the following steps:

[0080] Step S21: Different from the general text feature encoding neural network, the language feature extractor structure in S12 is mainly composed of a second decoupled representation network module and a language encoder;

[0081] Step S22: Input the electronic text page text collected in S11 into the second decoupled representation network module to replace the traditional text truncation preprocessing method to obtain the shallow language feature E head and the remaining text features E tail ;

[0082] Preferably, step S22 specifically includes the following steps:

[0083] Step S221: Set Text = {t1, t2, ..., t M} is a text data sequence processed by word segmentation, t j Represents the jth text descriptor and divides it into the head text T head and the remaining text T tail ;

[0084] Step S222: Use large-scale pre-trained Bert to encode text language features; embed the text sequence into the language feature space The specific process is expressed as:

[0085]

[0086] Where Bert(·) is the pre-trained Bert weight, is the text sequence to be embedded, The positions and semantic types of the embedding vectors are added in the embedding process according to the Bert processing method respectively;

[0087] Step S223: Use the text embedding method in S22 to embed the header text T head and the remaining text T tail Sharing the same Bert weight embedding feature space to obtain is a shallow language feature. are the remaining text features;

[0088] Step S224: input the remaining features and shallow language features into the guided alignment module and the language encoding module respectively for processing.

[0089] Step S23: The shallow language feature E obtained in S21 head The text features are further extracted through the 10-layer Transformer structure language encoder network to obtain the deep language features F t .

[0090] Step S3: Based on the network architecture in S1, the electronic text page image collected in S1 is input into the visual feature extractor to obtain deep visual features;

[0091] In this embodiment, step S3 specifically includes the following steps:

[0092] Step S31: the visual feature extractor structure in S12 is mainly composed of a first decoupled representation network module, a guided alignment module and a visual encoder;

[0093] Step S32: Different from the simple visual feature extraction in the traditional multimodal method, the electronic text page image collected in S11 is first processed by the first decoupled representation network module to decouple the shape feature f shape 、Color feature f hsv and pixel feature f gray ;

[0094] Step S33: The shape feature f obtained in S31 shape 、Color feature f hsv , pixel feature f gray and the remaining text features E obtained in S22 tail Input the guided alignment module to perform guided alignment to obtain the shallow visual feature E v ,Compared with traditional methods, it enriches the visual modality features;

[0095] Step S34: The shallow visual feature E v Input the visual encoder for processing, and further extract the visual features to obtain the deep visual features F v ; The self-attention layer in the conventional visual encoder is replaced by a Transformer self-attention layer with a Gaussian kernel function to generate a mask. The Gaussian attention GAttn(·) is used to obtain visual modal features that pay more attention to the central area. The calculation formula is as follows:

[0096]

[0097] where μ r ,μ l ,σ is a hyperparameter,μ r ,μ l represents the mean of the Gaussian function, σ represents the standard deviation of the Gaussian function, W q ,W k ,W θ are the Q value and K value in the self-attention mechanism and the learnable weight parameter set for the Gaussian weight, r and l are the position indexes of the feature points in the feature map, and T is the matrix transpose symbol;

[0098] Step S4: Based on the network architecture in S1, the deep language features and deep visual features obtained in S2 and S3 are input into the cross-modal fusion encoder to obtain multimodal features, and used in the classification network for classification;

[0099] In this embodiment, step S4 specifically includes the following steps:

[0100] Step S41: Based on the network architecture in S1, the deep language features F obtained in S23 are t And the deep visual feature F obtained by S34 v The cross-modal fusion encoder is input together to extract the cross-modal feature F through 6 layers of cross-attention encoding. x ;

[0101] Step S42: Transform the cross-modal feature F in S41 x Input the classification head network for classification; the classification head network consists of three fully connected layers with a dimension size of 1, and two fully connected layers are connected through the Mish activation function and the BatchNorm normalization layer. The classification head finally outputs a prediction sequence with a dimension of the number of categories through softmax, which represents the predicted probability of each category corresponding to the i-th electronic text page sample, and selects the category with the highest probability as the classification result of the sample.

[0102] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A multimodal text page classification method based on decoupled feature guidance, characterized in that: The following steps are involved: Acquire electronic text page data to construct a training set, and train a dual-stream end-to-end Transformer encoding network based on the training set, wherein the dual-stream end-to-end Transformer encoding network includes a visual feature extractor, a language feature extractor, and a cross-modal fusion encoder; Inputting text data of the electronic text page to be classified into a language feature extractor to obtain deep language features; Inputting image data of the electronic text page to be classified into a visual feature extractor to obtain deep visual features; The obtained deep language features and deep visual features are input into the cross-modal fusion encoder to obtain multimodal features, and then used in the classification network for classification; The visual feature extractor includes a first decoupled representation network module, a guided alignment module and a visual encoder; the method for processing data by the visual feature extractor is as follows: The image data of the electronic text page to be classified is processed by the first decoupled representation network module to decouple and obtain the shape feature f shape 、Color feature f hsv and pixel feature f gray ; The obtained shape feature f shape 、Color feature f hsv , pixel feature f gray and the remaining text features E obtained in S22 tail Input the guided alignment module to perform guided alignment to obtain the shallow visual feature E v ; The shallow visual feature E v Input the visual encoder for processing, further extract the visual features to obtain the deep visual features F v ; The self-attention layer in the conventional visual encoder is replaced by the Transformer self-attention layer with a Gaussian kernel function to generate a mask. The Gaussian attention GAttn(·) is used to obtain the visual modal features that pay more attention to the central area. The calculation formula is as follows: where μ r ,μ l ,σ is a hyperparameter,μ r ,μ l represents the mean of the Gaussian function, σ represents the Gaussian blur coefficient, W q ,W k ,W θ are the Q value and K value in the self-attention mechanism and the learnable weight parameter set for the Gaussian weight, r and l are the position indexes of the feature points in the feature map, and T is the matrix transpose symbol; Let I = {x1, x2, …, x N } is to obtain the electronic text page image training set from the Internet, x i For the i-th image, the text page image is sliced ​​to obtain the image segment sequence X i =Patch(x i ), Patch(·) is a convolution kernel with a size of 32×64 and a step size of (32, 64); and the image fragment sequence X i The channel information is removed and the pixel feature sequence f is used as the two-dimensional image. gray ; For image x i The image is processed by Gaussian kernels G(·) of different sizes, and a Gaussian difference pyramid DOG(·) is obtained; the SIFT method is used to extract the image extreme feature point descriptors keypints in the Gaussian difference pyramid, and the principal component analysis method PCA(·) is used to obtain the 128-dimensional eigenvalue SIFT val and the feature vector SIFT vec ; Input the feature value sequence into a fully connected network to extract it into a one-dimensional image shape feature sequence of a specific length f shape , and its calculation formula is as follows: DOG(x i ,σ=G(x i ,i*σ-G(x i ,σ))*x i SIFT val ,SIFT vec =PCA(keypints{(x i ,y i )|∈DOG(I k ,σ)},128) f shape =w sp ·SIFT val +b sp Where σ is the Gaussian blur coefficient, w sp ,b sp is the weight of the fully connected layer; By extracting the image x i The HSV features of the feature value sequence are mapped into a one-dimensional color feature sequence f of a specific length through a fully connected network. hsv , and its calculation formula is as follows: HSV=CalcHist(H+S+V) f hsv =w hsv ·HSV+b hsv Where H, S, V represent the image hue, saturation, and brightness respectively, CalcHist(·) represents the color histogram statistical function, and w hsv ,b hsv is the weight of the fully connected layer; The shape features, color features and pixel features are input into the guided alignment module for processing, and the decoupled features are used to guide the fusion to replace the traditional visual features; The shape features f are respectively transformed through MLP shape 、Color feature f hsv , pixel feature f gray Embed the feature space so that the pixel feature space is guided by the shape and color feature space, and the feature space E1 is expressed as: E1=γ1*LN(W g ·f gray )+γ2*LN(W sh ·f shape )+γ3 *LN(W c f hsv ) Where W g ,W sh ,W C They represent the MLP weights, γ1, γ2, and γ3 are f gray , f shape and f hsv Feature weighting hyperparameter, LN(·) represents a LayerNorm normalization layer; The remaining text features E tail Through 1×1 channel convolution, the dimension of the visual feature E1 is aligned, and the input of a self-attention layer is concatenated with E1 for semantic alignment to obtain the shallow visual feature Ev to replace the existing merged feature. The calculation formula is as follows: E v =selfAttn([E1,Conv 1x1 Etail)]) Where selfAttn(·) is the standard self-attention module, Conv 1x1 (·) represents 1×1 channel convolution; The shallow visual feature E v Input the visual encoder for processing.

2. The multimodal text page classification method based on decoupled feature guidance according to claim 1 is characterized in that: The electronic text page data includes electronic text page images and text data in the categories of economy, military, sports, food, technology, and tourism.

3. The multimodal text page classification method based on decoupled feature guidance according to claim 1 is characterized in that: The language feature extractor includes a second decoupled representation network module and a language encoder; the method for processing data by the language feature extractor is as follows: The text data of the electronic text page to be classified is first input into the second decoupled representation network module to obtain the shallow language feature E head and the remaining text features E tail ; Then the shallow language feature E head The text features are further extracted through the 10-layer Transformer structure language encoder network to obtain the deep language features F t .

4. The multimodal text page classification method based on decoupling feature guidance according to claim 3 is characterized in that Text={t1,t2,…,t M } is a text data sequence processed by word segmentation, t j Represents the jth text descriptor and divides it into the head text T head and the remaining text T tail ; Use large-scale pre-trained Bert to encode text language features; Embedding text sequences into language feature space The specific process is expressed as: Where Bert(·) is the pre-trained Bert weight, is the text sequence to be embedded, The positions and semantic types of the embedding vectors are added in the embedding process according to the Bert processing method respectively; The text embedding method in S22 is used to embed the head text T head and the remaining text T tail Sharing the same Bert weight embedding feature space to obtain is a shallow language feature. are the remaining text features; The remaining features and shallow language features are respectively input into the guided alignment module and the language encoding module for processing.

5. The multimodal text page classification method based on decoupled feature guidance according to claim 1 is characterized in that: The method for processing data by the cross-modal fusion encoder is as follows: The obtained deep language feature F t and deep visual features F v The cross-modal fusion encoder is input together to extract the cross-modal feature F through 6 layers of cross-attention encoding. x ; The cross-modal feature F x Input the classification head network for classification; the classification head network consists of three fully connected layers with a dimension size of 1, and two fully connected layers are connected through the Mish activation function and the BatchNorm normalization layer. The classification head finally outputs a prediction sequence with a dimension of the number of categories through softmax, which represents the predicted probability of each category corresponding to the i-th electronic text page sample, and selects the category with the highest probability as the classification result of the sample.

6. The multimodal text page classification method based on decoupled feature guidance according to claim 1 is characterized in that: The training of the dual-stream end-to-end Transformer encoding network specifically includes (1): Obtain electronic text page data to build a training set; (2): The visual modality feature F v and language modality feature F t Calculate the inter-modal loss L through infoNCEloss CM , the shape feature f shape and color feature f hsv Used to calculate the intra-modal loss L IM , and its calculation formula is as follows: where F i ,F i ′ represents the two features under the i-th picture, and F′ represents the features of all pictures F i ′, F′ id represents the idth feature in F′, sim(·) is the similarity calculation function of two feature sequences, and F v ,F t Respectively represent the visual modality features and language modality features of the current image, Respectively represent the set of all image visual modality features and language modality features, f shape ,f hsv Respectively represent the shape features and color features of the current image, are the sets of shape features and color features of all images respectively, and e is the natural index; (3): The classification probability output by the classification head is used to calculate the difference between the predicted classification probability and the true probability using the cross entropy loss function and obtain the loss value L BCE , and through the inter-modal loss L CM and the intra-modal loss L IM Weighted calculation of the loss of the entire network Loss = αL BCE +βL CM +γL IM , where α, β, and γ are hyperparameters; the network parameters are then updated using the gradient descent and back-propagation algorithms to complete the dual-stream end-to-end Transformer encoding network training.

Images