A multimodal data classification method, system, device, and medium

Abstract

This application relates to a multimodal data classification method, system, device, and medium. The method introduces a cross-modal semantic topological alignment mechanism to generate alignment vectors that characterize the deep semantic fit between corresponding nodes in two graph structures. These alignment vectors are then used to determine the credibility weights corresponding to the image and text modalities. Finally, based on a first and a second credibility weight, image and text features are fused to achieve the classification result for the target ebook. This method innovatively dynamically evaluates modal credibility and adaptively weights and fuses data based on alignment vectors by deeply analyzing the semantic graph structures of images and text and performing topological alignment. This achieves semantic understanding beyond the pixel level of images and the vocabulary level of text, improving the accuracy of classifying target ebooks.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of data classification, in particular to a data classification method, system, device and medium based on multi-modal. BACKGROUND

[0002] With the rapid growth of the number of e-books on various reading platforms, how to accurately classify them becomes crucial.

[0003] Traditional classification schemes for e-books include text or image-based classification methods, such as relying on paragraph meaning, abstract or full-text content, using TF-IDF, Word2Vec or BERT models to extract features for classification, but they cannot utilize the rich information of other modalities, and are not effective for e-books with poor text abstract quality.

[0004] Currently, research directions are mostly multi-modal fusion methods, i.e., trying to combine picture (such as illustrations and cover images) and text information, for example: extracting image and text feature vectors respectively, then performing feature concatenation (Concatenation) or using attention mechanism for weighted fusion, but its defects are: only staying at the pixel and word level feature mixing, failing to deeply understand the object relationship (semantic topology) inside the image and the semantic structure of the text, thus leading to low classification accuracy. SUMMARY

[0005] The main purpose of the embodiment of the present disclosure is to propose a data classification method, system, device and medium based on multi-modal, which can improve the accuracy of data classification.

[0006] In a first aspect, the embodiment of the present disclosure proposes a data classification method based on multi-modal, which comprises the following steps:

[0007] In response to a classification signal of a target e-book, extracting e-book text and e-book images from the target e-book; the e-book images include at least one of the cover image, illustrations, and chapter pages of the target e-book;

[0008] Parsing the text semantics in the e-book text to obtain a text semantic graph, and performing feature extraction on the e-book text to obtain text features;

[0009] Parsing the image semantics in the e-book image to obtain an image semantic graph, and performing feature extraction on the e-book image to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters and scenes in the e-book image, and the edges are the association relationships between the nodes;

[0010] A first consistency feature is determined between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and a second consistency feature is determined between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and an alignment vector representing the semantic consistency between text semantics and image semantics is generated based on the first consistency feature and the second consistency feature.

[0011] Based on the alignment vector, a first credibility weight for text semantics and a second credibility weight for image semantics are determined, and based on the first credibility weight and the second credibility weight, the text features and the image features are fused to obtain fused features;

[0012] Based on the fusion features, the category of the target e-book is determined.

[0013] In some embodiments, determining a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of an image node in the image semantic graph, and determining a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph, includes:

[0014] The text semantic graph and the image semantic graph are encoded using graph neural networks respectively to obtain the first graph embedding features of the text nodes in the text semantic graph and the second graph embedding features of the image nodes in the image semantic graph;

[0015] Based on the first graph embedding features and the second graph embedding features, determine the cross-modal attention weights of the text node and the image node;

[0016] Based on the cross-modal attention weights of the text node and the image node, a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node are generated.

[0017] In some embodiments, determining the cross-modal attention weights of the text node and the image node based on the first graph embedding feature and the second graph embedding feature includes:

[0018] A ij =σ(h) i T *W*h j );

[0019] Among them, h ih is the first graph embedding feature of the i-th text node. j h is the second graph embedding feature of the j-th image node. i T for h i The transpose of , W is the learnable parameter matrix, σ is the activation function, and A ij Let be the cross-modal attention weights for the i-th text node and the j-th image node;

[0020] The step of generating a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node, and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node, based on the cross-modal attention weights of the text node and the image node respectively, includes:

[0021] A cross-modal attention weight matrix is ​​generated based on the cross-modal attention weights of the image nodes and the text nodes;

[0022] The cross-modal attention weight matrix is ​​normalized by rows and columns to generate a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node, and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node.

[0023] In some embodiments, generating an alignment vector representing the semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature includes:

[0024] The first weighted summation feature between the text node and all the image nodes is calculated according to the following formula:

[0025]

[0026] Among them, H i For the first weighted summation feature corresponding to the i-th text node, N image A represents the total number of image nodes. text→image The first consistency feature is the first graph embedding feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node;

[0027] The second weighted summation feature between the image node and all the text nodes is calculated according to the following formula:

[0028]

[0029] Among them, H j For the second weighted summation feature corresponding to the j-th image node, N text A represents the total number of text nodes.image→text The second consistency feature is the second graph embedding feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node;

[0030] The first image embedding feature and the first weighted summation feature are concatenated to obtain the first concatenated feature;

[0031] The second image embedding feature and the second weighted summation feature are concatenated to obtain the second concatenated feature;

[0032] The first image embedding feature, the second image embedding feature, the first concatenation feature, and the second concatenation feature are pooled respectively to obtain the pooled first image embedding feature, the second image embedding feature, the first concatenation feature, and the second concatenation feature;

[0033] Calculate the similarity between every two pairs of features in the pooled first graph embedding feature, second graph embedding feature, first concatenated feature, and second concatenated feature to obtain multiple similarity scores;

[0034] The alignment vector is obtained by concatenating the multiple similarities.

[0035] In some embodiments, determining the first credibility weight of text semantics and the second credibility weight of image semantics based on the alignment vector includes:

[0036] The alignment vector is input into a fully connected layer to obtain the output hidden layer features;

[0037] The hidden layer features are input into another fully connected layer to obtain the output text semantic score vector and the output image semantic score vector.

[0038] The scoring vector of the text semantics is normalized to obtain the first credibility weight of the text semantics;

[0039] The image semantic score vector is normalized to obtain the second credibility weight of the image semantics; wherein the sum of the first credibility weight and the second credibility weight is equal to one.

[0040] In some embodiments, fusing the text features and the image features based on the first confidence weight and the second confidence weight includes:

[0041] The text features and the image features are fused by weighting and summing the second confidence weight with the image features and the first confidence weight with the text features.

[0042] In some embodiments, the step of extracting features from the e-book text to obtain text features includes:

[0043] The text features are obtained by extracting features from the e-book text using BERT.

[0044] The step of extracting features from the e-book image to obtain image features includes:

[0045] The e-book image was feature extracted using ResNet-50 to obtain image features.

[0046] Secondly, embodiments of this disclosure propose a multimodal data classification system, the system comprising:

[0047] An image and text acquisition module is used to extract ebook text and ebook images from the target ebook in response to the classification signal of the target ebook; the ebook images include at least one of the cover image, illustrations, and chapter pages of the target ebook;

[0048] The text semantic graph extraction module is used to parse the text semantics in the e-book text to obtain a text semantic graph, and to extract features from the e-book text to obtain text features;

[0049] The image semantic graph extraction module is used to parse the image semantics in the e-book image to obtain an image semantic graph, and to extract features from the e-book image to obtain image features; wherein, the image nodes in the image semantic graph are at least one of objects, people and scenes in the e-book image, and the edges are the association relationships between the nodes;

[0050] The alignment vector extraction module is used to determine a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and to determine a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and to generate an alignment vector representing the semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature.

[0051] The feature fusion module is used to determine a first confidence weight of text semantics and a second confidence weight of image semantics based on the alignment vector, and to fuse the text features and the image features based on the first confidence weight and the second confidence weight to obtain fused features;

[0052] An e-book classification module is used to determine the category of the target e-book based on the fusion features.

[0053] Thirdly, embodiments of this disclosure provide an electronic device, comprising:

[0054] At least one memory;

[0055] At least one processor;

[0056] At least one computer program;

[0057] The computer program is stored in the memory, and the processor executes the at least one computer program to achieve:

[0058] The above is an example of a multimodal data classification method.

[0059] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform:

[0060] The above is an example of a multimodal data classification method.

[0061] The multimodal data classification method provided in this application has at least the following beneficial effects:

[0062] This method introduces a cross-modal semantic topological alignment mechanism to generate alignment vectors that characterize the deep semantic fit between corresponding nodes in two graph structures. These alignment vectors are then used to determine the credibility weights for the image and text modalities. Finally, based on the first and second credibility weights, image and text features are fused to achieve the classification result for the target ebook. This method innovatively uses deep analysis of the semantic graph structures of images and text for topological alignment, dynamically evaluating modal credibility based on alignment vectors and adaptively weighting the fusion. This achieves semantic understanding beyond the pixel level of images and the vocabulary level of text, improving the accuracy of target ebook classification.

[0063] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating a multimodal data classification method provided in one embodiment of this application;

[0066] Figure 2 This is a schematic diagram of the structure of a multimodal data classification system provided in one embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0071] One embodiment of this application provides a multimodal data classification method, the method comprising the following steps S110 to S160:

[0072] Step S110: In response to the classification signal of the target e-book, extract the e-book text and e-book image from the target e-book.

[0073] In this step, the target ebook can be an ebook on any platform (such as Qidian or Qimao), such as "The Three-Body Problem".

[0074] The ebook text refers to the text content within the target ebook, without any specific limitations; the ebook image includes at least one of the target ebook's cover image, illustrations, and chapter pages.

[0075] Step S120: Parse the text semantics in the e-book text to obtain a text semantic graph, and extract features from the e-book text to obtain text features.

[0076] This step is one of the core steps of this application, namely, extracting text features from the e-book text on the one hand, and parsing the e-book text into a structured semantic graph to represent its key entities and logic on the other hand.

[0077] In this step, the text content is also constructed as a semantic graph (e.g., through knowledge graphs or syntactic dependencies). A graph alignment network is designed to calculate the fit between the text semantic graph of the ebook text and the image semantic graph of the image at the topological level. This fit can represent novel and powerful classification features.

[0078] In this step, syntactic analysis (such as dependency parsing) and semantic role labeling are used to construct a text semantic graph from the text content of key chapters. Nodes in the graph represent entities / keywords, and edges represent their semantic or grammatical relationships (such as subject-verb, verb-object, and modifier).

[0079] Step S130: Parse the image semantics in the e-book image to obtain the image semantic graph, and extract features from the e-book image to obtain image features; wherein, the image nodes in the image semantic graph are at least one of the objects, people and scenes in the e-book image, and the edges are the association relationships between the nodes.

[0080] This step is one of the core steps of this application, namely, extracting image features from ebook images on the one hand, and parsing ebook images (such as covers and illustrations) into structured semantic graphs to capture their internal objects, attributes, and spatial relationships on the other hand.

[0081] In this step, not only are image features extracted, but they are also transformed into a structured semantic relationship network. For example, analyzing the spatial relationship between the subject and background, the emotional distribution of colors, and the hierarchical structure of text and graphics in cover design reveals strong category signals in these topological relationships.

[0082] In this step, an image semantic graph is generated for each ebook image. Nodes in the graph represent detected objects, people, and scenes, while edges represent their spatial relationships (e.g., above, containing), action relationships (e.g., driving, reading), and attributes (e.g., color, style).

[0083] Step S140: Determine the first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and determine the second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and generate an alignment vector representing the semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature.

[0084] In this step, graph neural networks can be used to extract graph embedding features of each image node in the image semantic graph and graph embedding features of each text node in the text semantic graph.

[0085] In this step, the first consistency feature refers to the feature that has cross-modal consistency between the graph embedding feature of a text node and the graph embedding feature of another modality; the second consistency feature refers to the feature that has cross-modal consistency between the graph embedding feature of an image node and the graph embedding feature of another modality.

[0086] In this step, an alignment vector representing the semantic consistency between text and image semantics is generated based on the first and second consistency features. This alignment vector represents the deep semantic fit between corresponding nodes in the two graph structures. This alignment vector is the core basis for achieving dynamic and robust classification decisions in this application.

[0087] Step S150: Based on the alignment vector, determine the first confidence weight of text semantics and the second confidence weight of image semantics, and fuse text features and image features based on the first confidence weight and the second confidence weight to obtain fused features.

[0088] Step S160: Determine the category of the target e-book based on the fusion features.

[0089] In this step, the first credibility weight of the text semantics can be determined by the alignment vector and the MLP network, and the second credibility weight of another semantics can be determined.

[0090] In this step, the higher the alignment degree reflected by the alignment vector, the more balanced the weights of the two modalities are; the lower the alignment degree, the more inclined to trust the modality with higher credibility. The first credibility weight and the second credibility weight are used as the basis for decision-making when classifying e-books. For example, if the output first credibility weight and second credibility weight are (0.2, 0.8) respectively, the decision will depend heavily on the image modality.

[0091] Then, based on the first confidence weight and the second confidence weight, the image features and text features are fused to obtain the fused features. Here, the first confidence weight and the second confidence weight are used as the respective weights (or proportions) of image features and text features during image fusion, thereby obtaining the fused features.

[0092] Then, based on the fusion features input to the fully connected layer, the category of the target e-book is determined.

[0093] The embodiments provided in this application have at least the following beneficial effects:

[0094] This application introduces a cross-modal semantic (image semantic graph and text semantic graph) topological alignment mechanism to generate alignment vectors that characterize the deep semantic fit between corresponding nodes in the two graph structures. These alignment vectors are then used to determine the credibility weights (first credibility weight and second credibility weight) for the image and text modalities. Finally, based on the first and second credibility weights, image and text features are fused to achieve the classification result for the target e-book. This method innovatively uses deep analysis of the semantic graph structures of images and text for topological alignment, dynamically evaluating modal credibility based on alignment vectors and adaptively weighting the fusion. This achieves semantic understanding beyond the pixel level of images and the vocabulary level of text, improving the accuracy of classifying target e-books.

[0095] For ease of understanding, the following example is provided:

[0096] A multimodal data classification method includes the following:

[0097] Step S910: Analyze the target ebook and extract the ebook image and ebook text.

[0098] For example, analyzing a target ebook (such as "The Three-Body Problem") can separate the ebook image and the ebook text.

[0099] In some embodiments, the cover, chapter pages, and large illustrations may be extracted as ebook images.

[0100] Step S920: Perform image semantic topology parsing on the e-book image to generate an image semantic graph;

[0101] Semantic role labeling and parsing are performed on e-book text to generate a text semantic graph.

[0102] The text is parsed into a graph structure based on syntactic or semantic relations to represent its key entities and logic.

[0103] The text is preprocessed, named entity recognition is performed, and dependency parsing is conducted to construct a text semantic graph G. text =(V text E text ), where V text It is an entity / keyword node, E text It is a semantic relation edge.

[0104] The input image is processed using a deep learning-based scene graph generation model, and the output image semantic graph G is generated. image =(V image E image ), where V image It is a set of object nodes, E image It is a set of relation edges.

[0105] Step S930: Calculate the alignment vector between the text semantic graph and the image semantic graph.

[0106] G is encoded using a graph neural network. image and G text For example, first calculate the soft correspondence between nodes (node ​​pairs) in two graph structures, and finally aggregate the information of all node pairs to output a fixed-length alignment vector S.

[0107] Input: The text semantic graph is G. text =(V text E text The image semantic graph is G. image =(V image E image ).

[0108] Output: Alignment vector S, a fixed-length numerical vector S, where each dimension represents the degree of consistency between the two modalities (image modality and text modality) on a specific semantic dimension of the graph structure.

[0109] The specific steps are as follows:

[0110] Step S9310, graph encoding, maps two heterogeneous graphs (text semantic graph and image semantic graph) into the same comparable high-dimensional feature space.

[0111] Two graph neural networks are used respectively to analyze G. text and G image Encode it.

[0112] The encoder of the graph neural network outputs a deep feature representation of each text node, forming a matrix of the first graph embedding features of the text node. Where, N text d is the number of text nodes in the text semantic graph, and d is the embedding dimension.

[0113] The encoder of another graph neural network outputs a deep feature representation for each image node, forming a matrix of second graph embedding features for the image nodes. Where, N image It is the number of image nodes in the image semantic graph.

[0114] Step S9320: Cross-modal node-level interaction, calculate the correlation between each node in the text semantic graph and each node in the image semantic graph, and find potential semantic correspondences.

[0115] First, calculate a cross-modal attention matrix. Where each element A in the matrix ij(i.e., cross-modal attention weight) represents the relevance score between text node i in the text semantic graph and image node j in the image semantic graph. Its calculation formula is: A ij =σ(h) i T *W*h j ).

[0116] Among them, h i h is the first graph embedding feature of text node i. j It is the second graph embedding feature of image node j, W∈R (d*d) It is a learnable parameter matrix used to measure the compatibility of two graph embedding features, and σ is the activation function.

[0117] Finally, the cross-modal attention matrix A is normalized by both rows and columns to obtain the first consistency feature matrix A from text nodes to image nodes. text→image And the second consistency feature matrix A from image nodes to text nodes image→text .

[0118] Step S9330: Node alignment feature generation. Based on the consistency feature matrix from the previous step, a feature representation based on another modal perspective is generated for each node.

[0119] For each text node i, compute its first weighted summation feature based on the image semantic graph:

[0120]

[0121] This feature H i It represents the aggregated information of all image nodes most relevant to text node i.

[0122] Similarly, for each image node j, calculate the second weighted summation feature:

[0123]

[0124] At this point, each node possesses two features: its original embedding and the contextual embedding interpreted from another modality. That is: the first graph embedding feature h... i and the corresponding first weighted summation feature H i The second image embeds the feature h. j and the corresponding second weighted summation feature h j .

[0125] Step S9340: Compare the two graph structures and generate the final alignment vector.

[0126] Pooling operations (e.g., max pooling, average pooling, or more advanced attention pooling) are performed on the node feature sets (including graph embedding features and weighted summation features) of the text semantic graph and the image semantic graph respectively to obtain global feature representations of the two graphs:

[0127] The first image after pooling embeds the feature h i For: POOL(h) i );

[0128] The second graph after pooling embeds the feature h j For: POOL(h) j );

[0129] The first splicing feature h after pooling i ||H i For: POOL(h) i ||H i This refers to text graph features that incorporate the image context;

[0130] The second splicing feature h after pooling j ||H j For: POOL(h) j ||H j This refers to image graph features that integrate textual context;

[0131] Where || represents the mapping function for vector concatenation.

[0132] Similarity calculation: Calculates the similarity between multiple sets of features. For example:

[0133] s1 = sim(POOL(h) i ),POOL(h j ));

[0134] s2 = sim(POOL(h) i ||H i ),POOL(h j ||H j ));

[0135] s3 = sim(POOL(h) i ),POOL(h i ||H i ));

[0136] s4 = sim(POOL(h) j ),POOL(h j ||H j ));

[0137] These similarities are concatenated into a single alignment vector, for example:

[0138] S = [s1, s2, s3, s4, ...];

[0139] Step S940: Extract the text feature vector of the text content and the image feature vector of the key image set.

[0140] Extract text features F from ebook text using a pre-trained language model (such as BERT). text Use a pre-trained image model (such as ResNet-50) to extract image features F from ebook images. image .

[0141] Step S950: Using the alignment vector S as input, generate the first credibility weight λ of the text modality. text The second confidence weight λ of the image modality image .

[0142] Using a lightweight multilayer perceptron (MLP) as input, the alignment vector S is used to output the confidence weights λ of the two modalities. text and λ image .

[0143] Step S950 includes the following steps S9510 to S9530;

[0144] Step S9510, Feature Mapping. The alignment vector S is input into a fully connected layer, which maps the alignment vector S to a higher-dimensional or more abstract feature space to capture its non-linear meaning.

[0145] Calculate: h1 = ReLU(W1·S + b1);

[0146] W1 is the weight matrix, b1 is the bias vector, and h1 is the hidden layer feature.

[0147] Step S9520: Input the h1 obtained in the previous step into another fully connected layer.

[0148] Output two unnormalized scores, representing the credibility evidence for the text modality and the image modality, respectively.

[0149] Calculate: z = W2·h1 + b2;

[0150] Here, z is a two-dimensional vector (fraction), where z[0] is the score for the text modality and z[1] is the score for the image modality.

[0151] Step S9530: Normalize the score z using the Softmax function.

[0152] Transform the two scores into a probability distribution that satisfies λ text +λ image =1.

[0153] If the score of one modality is significantly higher than that of another, its weight will approach 1, while the other will approach 0, thus achieving a gating effect.

[0154] calculate:

[0155] λ text =exp(z[0]) / (exp(z[0])+exp(z[1]));

[0156] λ image =exp(z[1]) / (exp(z[0])+exp(z[1]));

[0157] exp is an exponential function.

[0158] Step S960: Based on the weights, dynamically weight and fuse the text feature vector and the image feature vector to obtain the fused feature vector.

[0159] Based on the alignment vector, the credibility of the two modalities is dynamically evaluated, and the features are adaptively weighted and fused to finally complete the classification decision.

[0160] Perform the operation Z = λ t *F t +λ i *F i The fused feature Z is then fed into a fully connected layer classifier to obtain the final classification result.

[0161] Step S970: Input the fused feature vector into the classifier to obtain the classification result of the e-book.

[0162] It is important to note that this is not trained in isolation, but rather as part of an end-to-end joint training process for a comprehensive multimodal data classification network. In some embodiments, the total loss function may consist of three parts: 1) cross-entropy loss (for classification tasks); 2) contrastive loss (for graph alignment); and 3) regularization loss (for credibility assessment).

[0163] like Figure 2 One embodiment of this application provides a multimodal data classification system, the system comprising:

[0164] The image and text acquisition module 1001 is used to extract ebook text and ebook images from the target ebook in response to the classification signal of the target ebook; the ebook images include at least one of the cover image, illustrations, and chapter pages of the target ebook;

[0165] The text semantic graph extraction module 1002 is used to parse the text semantics in the e-book text, obtain the text semantic graph, and extract features from the e-book text to obtain text features;

[0166] The image semantic graph extraction module 1003 is used to parse the image semantics in the e-book image to obtain the image semantic graph, and to extract features from the e-book image to obtain image features; wherein, the image nodes in the image semantic graph are at least one of the objects, people and scenes in the e-book image, and the edges are the association relationships between the nodes;

[0167] The alignment vector extraction module 1004 is used to determine the first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and to determine the second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and to generate an alignment vector representing the semantic consistency between the text semantics and the image semantics based on the first consistency feature and the second consistency feature.

[0168] The feature fusion module 1005 is used to determine the first confidence weight of text semantics and the second confidence weight of image semantics based on the alignment vector, and to fuse text features and image features based on the first confidence weight and the second confidence weight to obtain fused features;

[0169] The e-book classification module 1006 is used to determine the category of a target e-book based on fusion features.

[0170] It should be noted that the multimodal data classification system provided in this application and the multimodal data classification method described above are based on the same inventive concept. Therefore, the content of the multimodal data classification method described above is also applicable to the content of the multimodal data classification system in this application, and will not be repeated here.

[0171] like Figure 3 This application also provides an electronic device, which includes:

[0172] At least one memory;

[0173] At least one processor;

[0174] At least one program;

[0175] The program is stored in memory, and the processor executes at least one program to implement the multimodal data classification method described above in this disclosure.

[0176] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0177] The following is combined with Figure 3 The electronic device according to embodiments of this application will be described in detail. The electronic device includes:

[0178] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0179] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform a multimodal data classification method according to an embodiment of this disclosure.

[0180] The input / output interface 1800 is used to implement information input and output.

[0181] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0182] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0183] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 communicate with each other within the device via bus 2000.

[0184] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the aforementioned multimodal data classification method.

[0185] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, which can be linked to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0186] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0187] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0190] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0191] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication links shown or discussed may be through some interfaces; the indirect coupling or communication links between apparatuses or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A multi-modal based data classification method, characterized in that, The method comprises the following steps: In response to a classification signal of a target e-book, extracting e-book text and e-book images from the target e-book; the e-book images include at least one of cover images, illustrations, and chapter pages of the target e-book; Parsing the text semantics in the e-book text to obtain a text semantic graph, and performing feature extraction on the e-book text to obtain text features; Parsing the image semantics in the e-book images to obtain an image semantic graph, and performing feature extraction on the e-book images to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters and scenes in the e-book images, and the edges are the association relationships between the image nodes; Determine the first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the corresponding image node in the image semantic graph, and determine the second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the corresponding text node in the text semantic graph; and generate an alignment vector representing the semantic consistency of text semantics and image semantics based on the first consistency feature and the second consistency feature; According to the alignment vector, determine the first confidence weight of the text semantics and the second confidence weight of the image semantics, and fuse the text features and the image features according to the first confidence weight and the second confidence weight to obtain the fusion features; Based on the fusion features, determine the category of the target e-book.

2. The method of claim 1, wherein, The determination of the first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the corresponding image node in the image semantic graph, and the determination of the second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the corresponding text node in the text semantic graph, comprises: Use a graph neural network to encode the text semantic graph and the image semantic graph respectively to obtain the first graph embedding feature of the text node in the text semantic graph and the second graph embedding feature of the image node in the image semantic graph; According to the first graph embedding feature and the second graph embedding feature, determine the cross-modal attention weight of the text node and the image node; According to the cross-modal attention weight of the text node and the image node respectively, generate the first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the corresponding image node, and the second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the corresponding text node.

3. The method of claim 2, wherein, The determination of the cross-modal attention weight of the text node and the image node according to the first graph embedding feature and the second graph embedding feature comprises: ； in, For the first The first graph embedding features of each text node. For the first The second graph embedding features of each image node for transpose, For a learnable parameter matrix, For activation function, For the first The text node and the first Cross-modal attention weights for each image node; The first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the corresponding image node and the second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the corresponding text node are generated according to the cross-modal attention weight of the text node and the image node respectively, including: Based on the cross-modal attention weight of the image node and the text node, a cross-modal attention weight matrix is generated; The cross-modal attention weight matrix is normalized by row and column to generate the first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the corresponding image node and the second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the corresponding text node.

4. The method of claim 3, wherein, The alignment vector representing the semantic consistency of text semantics and image semantics is generated based on the first consistency feature and the second consistency feature, including: The first weighted sum feature between the text node and all the image nodes is calculated according to the following formula: ； wherein, is a first weighted sum feature corresponding to the first text node, is a second weighted sum feature corresponding to the second text node, is a total number of image nodes, is a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node; The second weighted sum feature between the image node and all the text nodes is calculated according to the following formula: ； wherein, is a second weighted sum feature corresponding to the th image node, is a total number of text nodes, is a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node; The first graph embedding feature and the first weighted sum feature are spliced to obtain the first spliced feature; The second graph embedding feature and the second weighted sum feature are spliced to obtain the second spliced feature; The first graph embedding feature, the second graph embedding feature, the first spliced feature and the second spliced feature are respectively pooled to obtain the first graph embedding feature, the second graph embedding feature, the first spliced feature and the second spliced feature after pooling; The similarity between each two groups of features in the first graph embedding feature, the second graph embedding feature, the first spliced feature and the second spliced feature after pooling is calculated to obtain a plurality of similarities; The alignment vector is obtained by splicing according to the plurality of similarities.

5. The method of claim 4, wherein, The first credibility weight of text semantics and the second credibility weight of image semantics are determined according to the alignment vector, including: The alignment vector is input into a fully connected layer to obtain the output hidden layer feature; The hidden layer feature is input into another fully connected layer to obtain the output score vector of text semantics and the output score vector of image semantics; The score vector of text semantics is normalized to obtain the first credibility weight of text semantics; The score vector of image semantics is normalized to obtain the second credibility weight of image semantics; wherein the sum of the first credibility weight and the second credibility weight is equal to one.

6. The method of claim 5, wherein, The text feature and the image feature are fused according to the first credibility weight and the second credibility weight, including: The text feature and the image feature are weighted and summed according to the second credibility weight and the image feature, and the first credibility weight and the text feature to fuse the text feature and the image feature.

7. The method of claim 6, wherein, The text feature is obtained by feature extraction on the electronic book text, including: The BERT is used for feature extraction on the e-book text to obtain text features; The feature extraction on the e-book image to obtain image features comprises: The ResNet-50 is used for feature extraction on the e-book image to obtain image features.

8. A multi-modal based data classification system, characterized in that, The system comprises: An image and text acquisition module is configured to extract e-book text and e-book images from a target e-book in response to a classification signal of the target e-book, wherein the e-book images comprise at least one of a cover image, an illustration, and a chapter page of the target e-book; A text semantic graph extraction module is configured to parse text semantics in the e-book text to obtain a text semantic graph, and to perform feature extraction on the e-book text to obtain text features; An image semantic graph extraction module is configured to parse image semantics in the e-book image to obtain an image semantic graph, and to perform feature extraction on the e-book image to obtain image features, wherein image nodes in the image semantic graph are at least one of objects, characters, and scenes in the e-book image, and edges are association relationships between the image nodes; An alignment vector extraction module is configured to determine first consistency features between graph embedding features of each text node in the text semantic graph and graph embedding features of image nodes in the image semantic graph, and to determine second consistency features between graph embedding features of each image node in the image semantic graph and graph embedding features of text nodes in the text semantic graph, and to generate an alignment vector representing semantic consistency of text semantics and image semantics based on the first consistency features and the second consistency features; A feature fusion module is configured to determine a first credibility weight of text semantics and a second credibility weight of image semantics based on the alignment vector, and to fuse the text features and the image features based on the first credibility weight and the second credibility weight to obtain fused features; An e-book classification module is configured to determine a category of the target e-book based on the fused features.

9. An electronic device, comprising: comprises: at least one memory; at least one processor; at least one computer program; The computer program is stored in the memory, and the processor executes the at least one computer program to implement: The method for classifying data based on multiple modes according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to execute: The method for classifying data based on multiple modes according to any one of claims 1 to 7.

Images