A Long Text Classification Method Based on Multimodal Knowledge Enhancement

Abstract

This invention relates to the field of data processing, and more particularly to a long text classification method based on multimodal knowledge enhancement. The method includes: dividing extracted multimodal data into semantic feature units and performing feature extraction, integration, and projection to generate a unified feature representation for each modality; based on the vector set of semantic feature units for each modality, performing intra-modal association modeling and inter-modal association modeling to obtain intra-modal semantic association graphs and inter-modal association matrices, and optimizing the inter-modal association matrix; based on the intra-modal semantic association graphs and the optimized inter-modal association matrix, calculating the dynamic contribution of each modality, and integrating the unified feature representations of each modality to generate a comprehensive feature; and based on the comprehensive feature, predicting the category label of the long text. This method solves the problems of low classification accuracy and poor robustness in traditional text classification methods when processing multimodal long texts due to insufficient semantic understanding, missing logical connections, and opaque decision-making processes.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a long text classification method based on multimodal knowledge enhancement. Background Technology

[0002] In the field of natural language processing, text classification is a fundamental and crucial task, widely used in scenarios such as sentiment analysis, public opinion monitoring, news classification, question answering systems, and spam detection. Traditional text classification methods mainly rely on plain text information, achieving classification through feature extraction and pattern recognition. However, when processing long texts containing multiple modalities such as text, images, and tables, they suffer from low classification accuracy and poor robustness due to insufficient semantic understanding, lack of logical connections, and opaque decision-making processes.

[0003] Currently, pre-trained large models, multimodal alignment models, and knowledge graph technologies are constantly advancing, showing broad prospects in academic research and practical applications, and becoming one of the important research directions in the field of intelligent text processing. Summary of the Invention

[0004] This invention provides a long text classification method based on multimodal knowledge enhancement to solve the technical problems of low classification accuracy and poor robustness of traditional text classification methods when dealing with long texts containing multiple modalities such as text, images, and tables, due to insufficient semantic understanding, lack of logical connections, and opaque decision-making processes.

[0005] The present invention provides a long text classification method based on multimodal knowledge enhancement, which specifically includes the following technical solutions:

[0006] A long text classification method based on multimodal knowledge enhancement includes the following steps:

[0007] S1. Extract multimodal data from the long text to be classified, divide each modality into semantic feature units and extract features to obtain a vector set of semantic feature units for each modality; integrate the feature vectors of all semantic feature units of each modality to generate a global feature vector for each modality; project the global feature vector of each modality onto a unified semantic space to generate a unified feature representation for each modality.

[0008] S2. Based on the vector set of semantic feature units for each modality, perform intra-modal association modeling and inter-modal association modeling to obtain intra-modal semantic association graphs and inter-modal association matrices; through a semantic logic consistency enhancement algorithm, introduce an external knowledge graph to optimize the inter-modal association matrix to obtain an optimized inter-modal association matrix;

[0009] S3. Based on the intramodal semantic association graph and the optimized intermodal association matrix, a dynamic modal contribution allocation algorithm is introduced to calculate the dynamic contribution of the modality. Based on the dynamic contribution of the modality, the unified feature representation of each modality is integrated to generate a comprehensive feature. Based on the comprehensive feature, the category label of the long text is predicted.

[0010] Preferably, S1 specifically includes:

[0011] By using mean pooling, the feature vectors of all semantic feature units of each modality are integrated to generate a global feature vector for each modality.

[0012] Preferably, S1 specifically includes:

[0013] By introducing modality-specific projection matrices and bias vectors, the global feature vectors of different modalities are mapped from their respective dimensions to a unified dimension, resulting in a unified feature representation for each modality.

[0014] Preferably, S2 specifically includes:

[0015] Based on the vector set of semantic feature units for each modality, the semantic similarity between semantic feature units is calculated. Then, using semantic feature units as nodes and the semantic similarity between semantic feature units as edge weights, intramodal association modeling is performed to construct an intramodal semantic association graph.

[0016] Preferably, S2 specifically includes:

[0017] Iterate through all modal pairs, calculate the semantic similarity of each feature vector of the semantic feature unit of each modal pair, and calculate the average of the semantic similarity of all semantic feature unit pairs between the two modalities to obtain the intermodal association strength; construct the intermodal association matrix based on the intermodal association strength.

[0018] Preferably, S2 specifically includes:

[0019] The external knowledge graph consists of concept nodes and logical relationship edges. In the implementation of the semantic logic consistency enhancement algorithm, all logical relationship edges of the external knowledge graph are traversed. The semantic similarity between the unified feature representation of any two modalities and the semantic representation vector of the concept node is calculated, and the knowledge graph correction term is constructed by combining the edge weights in the external knowledge graph.

[0020] Preferably, S2 specifically includes:

[0021] Based on the knowledge graph correction term and by introducing the first adjustment parameter, the intermodal association matrix is ​​optimized to obtain the optimized intermodal association strength, and the optimized intermodal association matrix is ​​constructed.

[0022] Preferably, S3 specifically includes:

[0023] In the implementation of the dynamic modal contribution allocation algorithm, the dynamic contribution of the modality is obtained based on the optimized intermodal association matrix and the edge weights of the intramodal semantic association graph, and by introducing a second adjustment parameter.

[0024] Preferably, S3 specifically includes:

[0025] The dynamic contribution of each modality is normalized to obtain the normalized dynamic contribution. Based on the normalized dynamic contribution, the unified feature representation of each modality is weighted and summed to generate a comprehensive feature. The comprehensive feature is then input into a classifier to predict the category label of the long text.

[0026] The beneficial effects of the technical solution of the present invention are:

[0027] 1. By using data preprocessing tools, multiple modal data such as text, images, and audio are separated from the long text to be classified, ensuring the integrity and independence of each modal data. Based on the pre-trained model, feature vectors of semantic feature units are extracted, capturing the semantic information of each modality in a fine-grained manner. This provides high-quality basic data for subsequent semantic integration and cross-modal association, significantly improving the processing efficiency of multimodal data and the accuracy of semantic representation.

[0028] 2. By integrating the feature vectors of semantic feature units of each modality through mean pooling, a global feature vector is generated. The global feature vector is then mapped to a unified semantic space using modality-specific projection matrices and bias vectors, achieving semantic alignment of features from different modalities. This enables heterogeneous data such as text, images, and audio to be compared in a unified dimension, laying the foundation for cross-modal semantic association and fusion, and significantly improving the ability of multimodal data collaborative analysis and the robustness of classification tasks.

[0029] 3. By constructing intramodal semantic association graphs and intermodal association matrices, the semantic structure within modalities and the semantic correlation between modalities are fully captured, making up for the shortcomings of traditional methods in understanding the semantics of complex long texts. This improves the semantic mining depth and classification accuracy of multimodal data, and provides strong support for semantic analysis in complex scenarios.

[0030] 4. By introducing an external knowledge graph and optimizing the intermodal association matrix through a semantic logic consistency enhancement algorithm, we ensure that the semantic association between modalities conforms to the logical constraints of domain knowledge. By combining the logical relationships and weights of the knowledge graph, we correct the intermodal association strength, which significantly improves the semantic consistency of the intermodal association matrix and the accuracy of classification tasks, and enhances its applicability and robustness in domain-specific scenarios.

[0031] 5. By using a dynamic modality contribution allocation algorithm, the semantic association graph within modalities and the optimized intermodal association matrix are comprehensively analyzed to calculate the dynamic contribution of each modality. Based on this, the unified feature representation of each modality is weighted and summed to generate comprehensive features. This dynamically adjusts the importance of each modality in the classification task, fully explores the synergistic effect of multimodal data, and the generated comprehensive features more comprehensively reflect the overall semantics of long texts, thereby significantly improving the prediction accuracy and generalization ability of the classifier. Attached Figure Description

[0032] Figure 1 This is a flowchart of a long text classification method based on multimodal knowledge enhancement as described in this invention. Detailed Implementation

[0033] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] The following description, in conjunction with the accompanying drawings, details a specific scheme for a long text classification method based on multimodal knowledge enhancement provided by this invention.

[0036] See attached document Figure 1 The diagram illustrates a flowchart of a long text classification method based on multimodal knowledge enhancement, provided by an embodiment of the present invention. The method includes the following steps:

[0037] S1. Extract multimodal data from the long text to be classified, divide each modality into semantic feature units and extract features to obtain a vector set of semantic feature units for each modality; integrate the feature vectors of all semantic feature units of each modality to generate a global feature vector for each modality; project the global feature vector of each modality onto a unified semantic space to generate a unified feature representation for each modality.

[0038] Extracting multimodal data from long texts to be classified involves using data preprocessing tools, such as text segmenters, image segmenters, and audio frame segmenters, to separate different modal data from the long texts to be classified, ensuring the integrity and independence of each modal data.

[0039] Each modality of data is divided into multiple semantic feature units (such as sentences in text, regions in images, rows or columns in tables, frames in audio, and keyframes in video), and feature vectors of each semantic feature unit are extracted through a pre-trained model to obtain a set of feature vectors for the semantic feature units of each modality; the pre-trained models, such as BERT and ResNet-50, are well-known techniques to those skilled in the art and will not be described in detail here.

[0040] Furthermore, the feature vectors of all semantic feature units of each modality are integrated through mean pooling to generate a global feature vector for each modality. , Indicates the first The feature vector dimension of the semantic feature unit of a modality;

[0041] To achieve cross-modal semantic alignment, the global feature vectors of each modality are... Projecting onto a unified semantic space generates a unified feature representation for each modality. , This represents the dimension of the unified feature representation of a modality. Specifically, by introducing a modality-specific projection matrix and bias vector, the global feature vectors of different modalities are mapped from their respective dimensions to a unified dimension, as expressed in the following formula:

[0042] ,

[0043] in, Indicates the first A unified feature representation for all modes; Indicates the first Global feature vectors of various modalities; Indicates the first A modality-specific projection matrix is ​​used to map global feature vectors from modality-specific dimensions to a unified dimension. This transformation of the feature dimension is achieved through matrix multiplication, aligning the representation spaces of different modalities while preserving semantic information. The modality-specific projection matrix is ​​learned through neural network training. Specifically, it is first randomly initialized using a Gaussian distribution, and then adjusted using optimization algorithms such as gradient descent. The value range of elements in the modality-specific projection matrix is... ; Indicates the first A modality-specific bias vector is used to enhance the flexibility of projection transformation, allowing feature representations to be translated and adjusted within a unified semantic space to capture modality-specific semantic biases. This modality-specific bias vector is learned through neural network training. Specifically, it is first initialized as a zero vector or a minimal random vector, and then optimized using algorithms such as gradient descent. The value range is... The aforementioned neural network training and gradient descent methods are well-known techniques to those skilled in the art and will not be elaborated upon here.

[0044] By acquiring multimodal data and performing feature extraction, integration, and projection based on the multimodal data, we can ensure that the features of different modalities (such as text, images, audio, etc.) are semantically comparable, thus laying the foundation for cross-modal semantic association and fusion.

[0045] S2. Based on the vector set of semantic feature units for each modality, perform intra-modal association modeling and inter-modal association modeling to obtain intra-modal semantic association graphs and inter-modal association matrices; through a semantic logic consistency enhancement algorithm, introduce an external knowledge graph to optimize the inter-modal association matrix to obtain an optimized inter-modal association matrix;

[0046] To uncover the semantic relationships within each modality, based on the vector set of semantic feature units for each modality, the semantic similarity between semantic feature units is calculated. Then, using semantic feature units as nodes and the semantic similarity between semantic feature units as edge weights, intramodal association modeling is performed to construct an intramodal semantic association graph.

[0047] The feature vectors of semantic feature units in different modalities may have different dimensions, but cosine similarity is independent of dimension, thus ensuring the universality of the calculation;

[0048] The edge weights can reflect the semantic similarity between two semantic feature units within a modality. The higher the edge weight, the stronger the association between the semantic feature units, and the lower the edge weight, the weaker the association between the semantic feature units.

[0049] The formula for calculating the edge weights of a semantic association graph is:

[0050] ,

[0051] in, Indicates the first Within a certain modality The semantic feature unit and the first The edge weights between semantic feature units reflect the semantic similarity between two semantic feature units. They are calculated using cosine similarity and range from [value range missing]. ; Indicates the first Within a certain modality Feature vectors of semantic feature units; Indicates the first Within a certain modality Feature vectors of semantic feature units; Indicates the first Within a certain modality The L2 norm of the feature vector of each semantic feature unit is used to normalize the length of the feature vector of the semantic feature unit. Indicates the first Within a certain modality The L2 norm of the feature vectors of each semantic feature unit;

[0052] To quantify the semantic correlation between different modalities, intermodal correlation modeling is performed based on the vector set of semantic feature units of different modalities, and an intermodal correlation matrix is ​​constructed. The elements of the intermodal correlation matrix represent the correlation strength between two modalities.

[0053] Specifically, iterate through all modal pairs, calculate the semantic similarity of the feature vectors of the semantic feature units of each modal pair, and further calculate the average of the semantic similarity of all semantic feature unit pairs between the two modalities, i.e. the intermodal association strength, and use the intermodal association strength as the element of the intermodal association matrix.

[0054] The formula for calculating the intermodal correlation strength is:

[0055] ,

[0056] in, Indicates the first Type and first mode The correlation strength between different modalities is used to reflect the semantic relevance between them. Indicates the first The number of semantic feature units for each modality; Indicates the first The number of semantic feature units for each modality; This represents the normalization factor, which is used to sum and average the cosine similarities of all semantic feature unit pairs between the two modalities to ensure that the overall correlation can be reflected. The double summation symbol is used to iterate through the nth iteration. Modal The semantic feature unit and the first Modal Each semantic feature unit is represented by a number of semantic feature units, and the cosine similarity of all semantic feature unit pairs is summed. Indicates the first Within a certain modality Feature vectors of semantic feature units; Indicates the first Within a certain modality The L2 norm of the feature vectors of each semantic feature unit;

[0057] By using intramodal association modeling and intermodal association modeling, the semantic structure of multimodal data is fully captured, making up for the shortcomings of traditional methods in understanding the semantics of complex long texts. The intermodal association matrix quantifies the semantic correlation between modalities, promotes collaborative analysis of multimodal data, and improves the accuracy and robustness of classification.

[0058] To ensure that the semantic associations between modalities conform to the logical constraints of domain knowledge, an external knowledge graph is introduced through a semantic logic consistency enhancement algorithm to optimize the intermodal association matrix, thereby improving the accuracy and robustness of long text classification.

[0059] The external knowledge graph consists of concept nodes and logical relationship edges. Concept nodes represent domain-related entities or concepts, and edges represent logical relationships between concept nodes. Each edge has a weight to reflect the strength of the logical relationship.

[0060] The semantic logic consistency enhancement algorithm calculates the semantic similarity between the unified feature representation of each modality and the semantic representation vector of the concept node in the external knowledge graph by using cosine similarity calculation, thereby quantifying the degree of closeness between the unified feature representation of each modality and the semantic representation vector of the concept node in the external knowledge graph in the semantic space.

[0061] Furthermore, by traversing all logical relationship edges in the external knowledge graph, and based on the semantic similarity between the unified feature representation of the modality and the semantic representation vector of the concept node, combined with the edge weights in the external knowledge graph, a knowledge graph correction term is constructed.

[0062] The knowledge graph correction term is added to the corresponding elements of the inter-modal association matrix, and a first adjustment parameter is introduced to control the strength of the knowledge graph constraints. This balances the original inter-modal association strength with the contribution of the knowledge graph correction term, enhancing optimization flexibility and generating the optimized inter-modal association strength. The formula is expressed as follows:

[0063] ,

[0064] in, Indicates the optimized first... Type and first mode The correlation strength between different modes; This represents the first adjustment parameter, used to control the strength of external knowledge graph constraints. It is determined using cross-validation and its value ranges from [value range missing]. The cross-validation method is a well-known technique in the art and will not be described in detail here. Represents the set of edges to the external knowledge graph. All edges Summation is used to synthesize the effects of logical relationships. Indicates the first The unified feature representation of the first modality and the first in the external knowledge graph The semantic similarity of the semantic representation vectors of each concept node is calculated based on cosine similarity, and the formula is expressed as follows: It is used to measure the semantic consistency between modal features and concept nodes in a knowledge graph. For the first in the external knowledge graph The semantic representation vector of each concept node. and They represent the first Unified feature representation of various modes and the first in external knowledge graphs Semantic representation vector of each concept node The L2 norm; Indicates the first The unified feature representation of the first modality and the first in the external knowledge graph The semantic similarity of the semantic representation vectors of each concept node is calculated based on cosine similarity, and the formula is expressed as follows: , For the first in the external knowledge graph The semantic representation vector of each concept node. Represents the first in the external knowledge graph Semantic representation vector of each concept node The L2 norm; Representing edges in an external knowledge graph The weights are used to quantify the strength of logical connections between concept nodes in the external knowledge graph, thereby enhancing the logicality of knowledge graph correction terms. They originate from the external knowledge graph construction process, are provided by the knowledge base, and their values ​​range from [value range missing]. The knowledge base mentioned is a shared resource open to the public, such as Wikipedia and Baidu Encyclopedia. This represents a correction term in the knowledge graph.

[0065] S3. Based on the intramodal semantic association graph and the optimized intermodal association matrix, a dynamic modal contribution allocation algorithm is introduced to calculate the dynamic contribution of the modality. Based on the dynamic contribution of the modality, the unified feature representation of each modality is integrated to generate a comprehensive feature. Based on the comprehensive feature, the category label of the long text is predicted.

[0066] Furthermore, a dynamic modality contribution allocation algorithm is introduced. By comprehensively analyzing the optimized inter-modal correlation matrix and intra-modal semantic correlation graph, the dynamic contribution of each modality is obtained to reflect the relative importance of different modalities in the classification task.

[0067] The dynamic modal contribution allocation algorithm first calculates the sum of optimized intermodal association strengths; then, for each modality, based on the intramodal semantic association graph of each modality, it calculates the mean of the edge weights of all intramodal semantic association graphs to reflect the density of semantic feature units within the modality; finally, based on the sum of optimized intermodal association strengths and the mean of the edge weights of the intramodal semantic association graphs, it calculates the dynamic contribution of the modality.

[0068] The formula for calculating the dynamic contribution of a mode is:

[0069] ,

[0070] in, Indicates the first The dynamic contribution of each modality is used to reflect the relative importance of the modality to the classification task; This represents the sum of the correlation strengths between the optimized modes; Indicates the number of modes; This represents the second adjustment parameter, used to balance the contributions of inter-modal and intra-modal correlations. It is determined using cross-validation and its value ranges from [value missing]. ; Indicates the first Normalization factor for semantic feature unit pairs within a modality; Indicates the first The sum of edge weights of all semantic feature unit pairs within a modality reflects the overall association strength of semantic feature units within the modality; the larger the value, the closer the semantics within the modality.

[0071] Furthermore, the dynamic contribution of each modality is normalized (i.e., the dynamic contribution of each modality is divided by the sum of the dynamic contributions of all modalities) to obtain the normalized dynamic contribution. Based on the normalized dynamic contribution, the unified feature representation of each modality is weighted and summed to generate a comprehensive feature. The comprehensive feature integrates the semantic information of each modality and can reflect the overall semantics of the long text. The comprehensive feature is input into a classifier to predict the category label of the long text. The classifier uses a fully connected layer combined with a Softmax function to map the comprehensive feature to the category probability distribution and output the final classification result. This is a well-known technique and will not be elaborated here.

[0072] In summary, a long text classification method based on multimodal knowledge enhancement has been developed.

[0073] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0074] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A long text classification method based on multimodal knowledge enhancement, characterized in that, Includes the following steps: S1. Extract multimodal data from the long text to be classified, divide each modality into semantic feature units and extract features to obtain a vector set of semantic feature units for each modality; integrate the feature vectors of all semantic feature units of each modality through mean pooling to generate a global feature vector for each modality; introduce modality-specific projection matrix and bias vector to project the global feature vector of each modality onto a unified semantic space to generate a unified feature representation for each modality. S2. Based on the vector set of semantic feature units for each modality, perform intra-modal association modeling and inter-modal association modeling to obtain an intra-modal semantic association graph and an inter-modal association matrix, where the elements of the inter-modal association matrix represent the association strength between two modalities; an external knowledge graph is composed of concept nodes and logical relation edges. Through a semantic logic consistency enhancement algorithm, all logical relation edges of the external knowledge graph are traversed. By calculating the semantic similarity between the unified feature representation of any two modalities and the semantic representation vector of the concept node, and combining the edge weights in the external knowledge graph, a knowledge graph correction term is constructed; based on the knowledge graph correction term and introducing a first adjustment parameter, the inter-modal association matrix is ​​optimized to obtain the optimized inter-modal association strength, and the optimized inter-modal association matrix is ​​constructed. S3. Based on the intramodal semantic association graph and the optimized intermodal association matrix, a dynamic modal contribution allocation algorithm is introduced to calculate the dynamic contribution of the modality; based on the dynamic contribution of the modality, a comprehensive feature is generated by integrating the unified feature representation of each modality. Based on comprehensive features, predict the category label of long text.

2. The long text classification method based on multimodal knowledge enhancement according to claim 1, characterized in that, S2 specifically includes: Based on the vector set of semantic feature units for each modality, the semantic similarity between semantic feature units is calculated. Then, using semantic feature units as nodes and the semantic similarity between semantic feature units as edge weights, intramodal association modeling is performed to construct an intramodal semantic association graph.

3. The long text classification method based on multimodal knowledge enhancement according to claim 2, characterized in that, S2 specifically includes: Iterate through all modal pairs, calculate the semantic similarity of each feature vector of the semantic feature unit of each modal pair, and calculate the average of the semantic similarity of all semantic feature unit pairs between the two modalities to obtain the intermodal association strength; construct the intermodal association matrix based on the intermodal association strength.

4. The long text classification method based on multimodal knowledge enhancement according to claim 1, characterized in that, S3 specifically includes: In the implementation of the dynamic modal contribution allocation algorithm, the dynamic contribution of the modality is obtained based on the optimized intermodal association matrix and the edge weights of the intramodal semantic association graph, and by introducing a second adjustment parameter.

5. The long text classification method based on multimodal knowledge enhancement according to claim 4, characterized in that, S3 specifically includes: The dynamic contribution of each modality is normalized to obtain the normalized dynamic contribution. Based on the normalized dynamic contribution, the unified feature representation of each modality is weighted and summed to generate a comprehensive feature. The comprehensive feature is then input into a classifier to predict the category label of the long text.

Images