Data classification method, system and equipment based on multiple modes and medium

Abstract

The embodiment of the invention relates to a multi-modal-based data classification method, system and device and a medium, and the method comprises the steps: generating an alignment vector which can represent the deep semantic integrating degree between corresponding nodes of two graph structures through introducing a cross-modal semantic topology alignment mechanism; the method comprises the steps of obtaining an alignment vector, determining credibility weights corresponding to image modalities and text modalities by using the alignment vector, finally fusing image features and text features according to a first credibility weight and a second credibility weight, and realizing a classification result corresponding to a target electronic book by using the fused features. According to the method, the semantic graph structures of the image and the text are deeply analyzed, topological alignment is carried out, the modal credibility is innovatively and dynamically evaluated according to the alignment vector, and adaptive weighted fusion is carried out, so that semantic understanding beyond the pixel level of the image and the vocabulary level of the text is realized, and the accuracy of classifying the target electronic book is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data classification, and particularly relate to a data classification method and system based on multi-modal, a device and a medium. BACKGROUND

[0002] With the rapid growth of the number of e-books on various reading platforms, how to accurately classify them becomes crucial.

[0003] Traditional classification schemes for e-books include text or image-based classification methods, such as relying on paragraph meaning, abstract or full-text content, using TF-IDF, Word2Vec or BERT models to extract features for classification, but they cannot utilize the rich information of other modalities, and are not effective for e-books with poor text abstract quality.

[0004] Currently, research directions are mostly multi-modal fusion methods, i.e., trying to combine picture (such as illustrations and cover images) and text information, for example: extracting image and text feature vectors respectively, then performing feature concatenation or using attention mechanism for weighted fusion, but its defects are: only staying at the pixel and word level feature mixing, failing to deeply understand the object relationship (semantic topology) inside the image and the semantic structure of the text, thus leading to low classification accuracy. SUMMARY

[0005] The main purpose of the embodiments of the present disclosure is to provide a data classification method, system, device and medium based on multi-modal, which can improve the accuracy of data classification.

[0006] In a first aspect, the embodiments of the present disclosure provide a data classification method based on multi-modal, which comprises the following steps: In response to a classification signal of a target e-book, extracting e-book text and e-book images from the target e-book; the e-book images include at least one of cover images, illustrations, and chapter pages of the target e-book; Parsing the text semantics in the e-book text to obtain a text semantic graph, and performing feature extraction on the e-book text to obtain text features; Parsing the image semantics in the e-book images to obtain an image semantic graph, and performing feature extraction on the e-book images to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters and scenes in the e-book images, and the edges are the association relationships between the nodes; Determining a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and determining a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and generating an alignment vector representing the semantic consistency of text semantics and image semantics based on the first consistency feature and the second consistency feature; Determining a first credibility weight of text semantics and a second credibility weight of image semantics based on the alignment vector, and fusing the text feature and the image feature based on the first credibility weight and the second credibility weight to obtain a fused feature; Based on the fusion features, the category of the target e-book is determined.

[0007] In some embodiments, determining a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and determining a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph, includes: Encoding the text semantic graph and the image semantic graph using a graph neural network respectively to obtain first graph embedding features of text nodes in the text semantic graph and second graph embedding features of image nodes in the image semantic graph; Determining cross-modal attention weights of the text node and the image node according to the first graph embedding feature and the second graph embedding feature; According to the respective cross-modal attention weights of the text node and the image node, a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node are generated.

[0008] In some embodiments, determining the cross-modal attention weights of the text node and the image node based on the first graph embedding feature and the second graph embedding feature includes: ; in, For the The first graph embedding features of text nodes, For the The second graph embedding features of image nodes, for The transpose of is the learnable parameter matrix, is the activation function, For the text nodes and Cross-modal attention weights for image nodes; The step of generating, according to respective cross-modal attention weights of the text node and the image node, a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node, and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node, comprises: Generate a cross-modal attention weight matrix based on the cross-modal attention weights of the image node and the text node; The cross-modal attention weight matrix is ​​normalized by rows and columns to generate a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node and a second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node.

[0009] In some embodiments, generating an alignment vector representing semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature includes: The first weighted sum feature between the text node and all the image nodes is calculated according to the following formula: ; in, For the The first weighted sum feature corresponding to the text nodes, is the total number of image nodes, is a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node; The second weighted sum feature between the image node and all the text nodes is calculated according to the following formula: ; in, For the The second weighted sum feature corresponding to the image nodes, is the total number of text nodes, A second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node; concatenating the first graph embedding feature and the first weighted sum feature to obtain a first concatenated feature; Concatenating the second graph embedding feature and the second weighted sum feature to obtain a second concatenated feature; pooling the first graph embedding feature, the second graph embedding feature, the first concatenation feature and the second concatenation feature respectively to obtain the first graph embedding feature, the second graph embedding feature, the first concatenation feature and the second concatenation feature after pooling; calculating the similarity between each two groups of features in the first graph embedding feature, the second graph embedding feature, the first concatenation feature and the second concatenation feature after pooling to obtain a plurality of similarities; concatenating according to the plurality of similarities to obtain an alignment vector.

[0010] In some embodiments, the determining the first confidence weight of the text semantics and the second confidence weight of the image semantics according to the alignment vector comprises: inputting the alignment vector into a fully connected layer to obtain an output hidden layer feature; inputting the hidden layer feature into another fully connected layer to obtain an output score vector of the text semantics and an output score vector of the image semantics; normalizing the score vector of the text semantics to obtain the first confidence weight of the text semantics; normalizing the score vector of the image semantics to obtain the second confidence weight of the image semantics; wherein the sum of the first confidence weight and the second confidence weight is equal to one.

[0011] In some embodiments, the fusing the text feature and the image feature according to the first confidence weight and the second confidence weight comprises: weighting and summing according to the second confidence weight and the image feature, and the first confidence weight and the text feature to fuse the text feature and the image feature.

[0012] In some embodiments, the feature extraction of the e-book text to obtain a text feature comprises: performing feature extraction on the e-book text by BERT to obtain a text feature; the feature extraction of the e-book image to obtain an image feature comprises: performing feature extraction on the e-book image by ResNet-50 to obtain an image feature.

[0013] In a second aspect, the embodiments of the present disclosure provide a multi-modal based data classification system, which comprises: an image and text acquisition module, configured to extract e-book text and e-book image from a target e-book in response to a classification signal of the target e-book; the e-book image comprises at least one of cover image, illustration and chapter page of the target e-book; a text semantic graph extraction module configured to parse text semantics in the e-book text to obtain a text semantic graph, and perform feature extraction on the e-book text to obtain text features; an image semantic graph extraction module configured to parse image semantics in the e-book image to obtain an image semantic graph, and perform feature extraction on the e-book image to obtain image features; wherein an image node in the image semantic graph is at least one of an object, a person and a scene in the e-book image, and an edge is an association relationship between the nodes; an alignment vector extraction module configured to determine a first consistency feature between a graph embedding feature of each text node in the text semantic graph and a graph embedding feature of an image node in the image semantic graph, and determine a second consistency feature between a graph embedding feature of each image node in the image semantic graph and a graph embedding feature of a text node in the text semantic graph; and generate an alignment vector representing semantic consistency of text semantics and image semantics based on the first consistency feature and the second consistency feature; a feature fusion module configured to determine a first credibility weight of text semantics and a second credibility weight of image semantics according to the alignment vector, and fuse the text features and the image features according to the first credibility weight and the second credibility weight to obtain fused features; an e-book classification module configured to determine a category of the target e-book based on the fused features.

[0014] In a third aspect, an electronic device is provided, comprising: at least one memory; at least one processor; at least one computer program; the computer program is stored in the memory, and the processor executes the at least one computer program to implement: the above-mentioned data classification method based on multiple modalities.

[0015] In a fourth aspect, a computer readable storage medium is provided, which stores computer executable instructions for causing a computer to execute: the above-mentioned data classification method based on multiple modalities.

[0016] The data classification method based on multiple modalities provided by the embodiments has at least the following beneficial effects: The method generates an alignment vector capable of representing the deep semantic fit degree between the corresponding nodes of the two graph structures by introducing a cross-modal semantic topology alignment mechanism, and then determines the credibility weight corresponding to the image mode and the text mode by using the alignment vector. Finally, according to the first credibility weight and the second credibility weight, the image features and the text features are fused, and the fusion features are used to realize the classification result corresponding to the target electronic book. The method analyzes the semantic graph structure of the image and the text in depth and performs topology alignment, dynamically evaluates the modal credibility according to the alignment vector, and adaptively weights and fuses, realizes the semantic understanding beyond the pixel level of the image and the word level of the text, and improves the accuracy of classifying the target electronic book.

[0017] It can be understood that the beneficial effects of the second aspect to the fourth aspect compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. For details, please refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or related technical descriptions will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flowchart of a data classification method based on multi-modal provided by an embodiment of the present application; Figure 2 is a structural schematic diagram of a data classification system based on multi-modal provided by an embodiment of the present application; Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0021] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing only embodiments of this application and is not intended to be limiting of this application.

[0023] One embodiment of the present application provides a multi-modal based data classification method, the method comprising the following steps S110 to S160: Step S110, in response to a classification signal of a target e-book, extracting e-book text and e-book image from the target e-book.

[0024] In this step, the target e-book can be an e-book on any platform (such as Qidian or Qicat), for example, Three-Body.

[0025] The e-book text is the text content in the target e-book, which is not specifically limited here; the e-book image includes at least one of the cover image, the illustration, and the chapter page of the target e-book.

[0026] Step S120, parsing the text semantics in the e-book text to obtain a text semantic graph, and performing feature extraction on the e-book text to obtain text features.

[0027] This step is one of the core steps of the present application, that is, on the one hand, text features are extracted from the e-book text. On the other hand, the e-book text is parsed into a structured semantic graph to represent its key entities and logic.

[0028] In this step, the text content is also constructed into a semantic graph (for example, through a knowledge graph or a syntactic dependency relationship), and a graph alignment network is designed to calculate the degree of fit between the text semantic graph of the e-book text and the image semantic graph of the image on the topological structure level. This degree of fit can represent a new and powerful classification feature.

[0029] In this step, the text content of the key chapter is constructed into a text semantic graph using syntactic analysis (such as dependency parsing) and semantic role labeling. The nodes in the graph are entities / keywords, and the edges are their semantic or grammatical relationships (such as subject-predicate, verb-object, and modification).

[0030] Step S130, parsing the image semantics in the e-book image to obtain an image semantic graph, and performing feature extraction on the e-book image to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters and scenes in the e-book image, and the edges are the association relationships between the nodes.

[0031] This step is one of the core steps of the present application, that is, on the one hand, image features are extracted from the electronic book image. On the other hand, the electronic book image (such as the cover, the illustration) is parsed into a structured semantic graph, capturing its internal objects, attributes and spatial relationships.

[0032] In this step, not only the image features are extracted, but also they are converted into a structured semantic relationship network. For example, the spatial relationship between the main body and the background in the cover design, the emotional distribution of the color, the hierarchical structure of the text and the graphics, and these topological relationships contain strong class signals.

[0033] In this step, for each electronic book image, an image semantic graph is generated. The nodes in the graph are the detected objects, characters, scenes, the edges are the spatial relationships (for example, above, contains), action relationships (for example, driving, reading) and attributes (for example, color, style).

[0034] In step S140, a first consistency feature between the graph embedding features of each text node in the text semantic graph and the graph embedding features of the image nodes in the image semantic graph is determined, and a second consistency feature between the graph embedding features of each image node in the image semantic graph and the graph embedding features of the text nodes in the text semantic graph is determined; and an alignment vector representing the semantic consistency of the text semantics and the image semantics is generated based on the first consistency feature and the second consistency feature.

[0035] In this step, the graph neural network can be used to extract the graph embedding features of each image node in the image semantic graph and the graph embedding features of each text node in the text semantic graph.

[0036] In this step, the first consistency feature refers to the cross-modal consistency between the graph embedding features of the text nodes and the graph embedding features of another modality; and the second consistency feature refers to the cross-modal consistency between the graph embedding features of the image nodes and the graph embedding features of another modality.

[0037] In this step, based on the first consistency feature and the second consistency feature, an alignment vector representing the semantic consistency of the text semantics and the image semantics is generated. This alignment vector can represent the vector of the deep semantic fit degree between the corresponding nodes of the two graph structures. This alignment vector is the core basis for the dynamic and robust classification decision of the present application.

[0038] In step S150, according to the alignment vector, a first credibility weight of the text semantics and a second credibility weight of the image semantics are determined, and the text features and the image features are fused according to the first credibility weight and the second credibility weight to obtain a fusion feature.

[0039] In step S160, the category of the target electronic book is determined based on the fusion feature.

[0040] In this step, a first credibility weight of a text semantic can be determined by aligning the vector and the MLP network, and a second credibility weight of another semantic can be determined.

[0041] In this step, the higher the degree of alignment reflected by the alignment vector, the more balanced the weights of the two modalities are; the lower the degree of alignment, the more trustworthy the modality is. The first and second credibility weights here serve as the basis for decision-making when classifying e-books. For example, if the output first and second credibility weights are (0.2, 0.8) respectively, the decision is highly dependent on the image modality.

[0042] Then, based on the first credibility weight and the second credibility weight, the image features and the text features are fused to obtain a fused feature. Here, the first credibility weight and the second credibility weight are used as the respective weights (or proportions) of the image features and the text features when the image is fused, thereby obtaining the fused feature.

[0043] Then, the fused features are input to the fully connected layer to determine the category of the target e-book.

[0044] The embodiments provided in this application have at least the following beneficial effects: This application introduces a cross-modal semantic (image semantic graph and text semantic graph) topological alignment mechanism to generate an alignment vector that characterizes the deep semantic fit between corresponding nodes in the two graph structures. This alignment vector is then used to determine the corresponding credibility weights (first credibility weight and second credibility weight) for the image and text modalities. Finally, based on the first and second credibility weights, image and text features are fused, and the fused features are used to achieve the corresponding classification result for the target e-book. This method deeply analyzes the semantic graph structures of images and text and performs topological alignment. It innovatively dynamically evaluates modal credibility based on the alignment vector and adaptively weights the fusion, achieving semantic understanding beyond the pixel level of images and the word level of text, thereby improving the accuracy of target e-book classification.

[0045] For ease of understanding, the following examples are provided: A multimodal data classification method includes the following: Step S910: Parse the target e-book to extract the e-book image and e-book text.

[0046] For example, parse the target e-book (such as "The Three-Body Problem") and separate the e-book image and e-book text.

[0047] In some embodiments, covers, chapter pages, and large illustrations may be preferentially extracted as e-book images.

[0048] Step S920, image semantic topology analysis is performed on the electronic book image to generate an image semantic graph. The electronic book text is subjected to semantic role labeling and analysis to generate a text semantic graph.

[0049] The text is parsed into a graph structure based on syntactic or semantic relationship to represent key entities and logic thereof.

[0050] The text is preprocessed, named entity recognition and dependency syntax analysis are performed, and a text semantic graph is constructed wherein, is an entity / key word node, is a semantic relationship edge.

[0051] A deep learning scene graph generation model is used to process the input image to output an image semantic graph wherein, is a set of object nodes, is a set of relationship edges.

[0052] Step S930, an alignment vector between the text semantic graph and the image semantic graph is calculated.

[0053] The graph neural networks are used to encode and respectively. For example, the soft correspondence between the nodes (node pairs) of the two graph structures is calculated first, and finally the information of all node pairs is aggregated to output an alignment vector of a fixed length .

[0054] Input: the text semantic graph is ; the image semantic graph is .

[0055] Output: alignment vector , a fixed-length numerical vector , each dimension of which represents the consistency degree of the two modalities (image modality, text modality) in a certain semantic dimension of the graph structure.

[0056] The specific process steps are as follows: Step S9310, graph encoding, mapping the two heterogeneous graphs (text semantic graph and image semantic graph) into the same comparable high-dimensional feature space.

[0057] Two graph neural networks are used to encode and respectively.

[0058] The encoder of the graph neural network outputs a deep feature representation of each text node to form a matrix of the first graph embedding features of the text nodes wherein, is the number of text nodes in the text semantic graph, is the embedding dimension.

[0059] The encoder of the other graph neural network outputs a deep feature representation for each image node, forming a matrix of second graph embedding features for the image nodes where, is the number of image nodes in the image semantic graph.

[0060] Step S9320, cross-modal node-level interaction, calculates the relevance of each node in the text semantic graph to each node in the image semantic graph, and finds the potential semantic correspondence.

[0061] First, a cross-modal attention matrix is calculated. Each element in the matrix (i.e., the cross-modal attention weight) represents the relevance score between the text node in the text semantic graph and the image node in the image semantic graph. Its calculation formula is: .

[0062] where, is the first graph embedding feature of the text node , is the second graph embedding feature of the image node , is a learnable parameter matrix used to measure the compatibility of the two graph embedding features, is an activation function.

[0063] Finally, the first consistency feature matrix of the text node to the image node and the cross-modal attention matrix are normalized by row and column, respectively, to obtain the second consistency feature matrix of the image node to the text node .

[0064] Step S9330, node alignment feature generation, based on the consistency feature matrix of the previous step, generates a feature representation based on the other modal perspective for each node.

[0065] For each text node , calculate its first weighted sum feature based on the image semantic graph: ; This feature represents the aggregated information of all image nodes most relevant to the text node .

[0066] Similarly, for each image node , calculate the second weighted sum feature: ; At this time, each node has two features, namely its own original embedding and the context embedding interpreted from another modal perspective. That is: the first graph embedding feature and the corresponding first weighted sum feature , the second graph embedding feature and the corresponding second weighted sum feature .

[0067] Step S9340, compare the two graph structures and generate the final alignment vector.

[0068] The node feature sets (including graph embedding features and weighted sum features) of the text semantic graph and the image semantic graph are respectively pooled (for example, using maximum pooling, average pooling or more advanced attention pooling) to obtain the global feature representations of the two graphs: The pooled first graph embedding feature is: ; The pooled second graph embedding feature is: ; The pooled first spliced feature is: , that is, the text graph feature fused with the image context; The pooled second spliced feature is: , that is, the image graph feature fused with the text context; wherein, represents the mapping function of the vector splicing operation.

[0069] Similarity calculation: calculate the similarity between multiple groups of features. For example: ; ; ; ; Splice these similarities into a comprehensive alignment vector, for example: ; Step S940, extract the text feature vector of the text content and the image feature vector of the key image set.

[0070] Use a pre-trained language model (such as BERT) to extract the text features of the e-book text; use a pre-trained image model (such as ResNet-50) to extract the image features .

[0071] Step S950, taking the alignment vector as input, generates the first confidence weight for the text modality and the second confidence weight for the image modality .

[0072] By a light-weight multi-layer perceptron (MLP), taking the alignment vector as input, outputs the confidence weights for the two modalities and .

[0073] Step S950 includes the following steps S9510 to S9530; Step S9510, feature mapping. The alignment vector is input to a fully connected layer, which maps the alignment vector to a higher-dimensional or more abstract feature space to capture its non-linear implications.

[0074] is computed as ; is a weight matrix, is a bias vector, is the hidden layer feature.

[0075] Step S9520, input the from the previous step to another fully connected layer.

[0076] Outputs two un-normalized scores, representing the confidence evidence for the text modality and the image modality, respectively.

[0077] is computed as ; Here, is a two-dimensional vector (score), where is the score for the text modality, is the score for the image modality.

[0078] Step S9530, normalize the scores using the Softmax function.

[0079] Converts the two scores into a probability distribution that satisfies .

[0080] If the score of one modality is significantly higher than the other, its weight will tend to 1 and the other to 0, thus achieving a gating effect.

[0081] is computed as ; ; is an exponential function.

[0082] Step S960, according to the weight, the text feature vector and the image feature vector are dynamically weighted and fused, and the fused feature vector is obtained.

[0083] According to the alignment vector, the credibility of the two modalities is dynamically evaluated, and the features are adaptively weighted and fused, and finally the classification decision is completed.

[0084] performing operation and sending the fused feature into a fully connected layer classifier to obtain the final classification result.

[0085] Step S970, the fused feature vector is input into the classifier to obtain the classification result of the electronic book.

[0086] It should be noted that here is not trained alone, but as part of the entire multi-modal data classification network is trained end to end. In some embodiments, the total loss function can be composed of three parts: 1) cross entropy loss (for classification task); 2) contrastive loss (for graph alignment); 3) regularization loss (credibility evaluation).

[0087] As Figure 2 , one embodiment of the present application provides a multi-modal based data classification system, the system comprising: The image and text acquisition module 1001 is configured to extract the electronic book text and the electronic book image from the target electronic book in response to a classification signal of the target electronic book; the electronic book image comprises at least one of a cover image, an illustration, and a chapter page of the target electronic book; The text semantic graph extraction module 1002 is configured to parse the text semantics in the electronic book text to obtain a text semantic graph, and to extract features from the electronic book text to obtain text features; The image semantic graph extraction module 1003 is configured to parse the image semantics in the electronic book image to obtain an image semantic graph, and to extract features from the electronic book image to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters and scenes in the electronic book image, and the edges are the association between the nodes; The alignment vector extraction module 1004 is configured to determine first consistency features between the graph embedding features of each text node in the text semantic graph and the graph embedding features of the image nodes in the image semantic graph, and determine second consistency features between the graph embedding features of each image node in the image semantic graph and the graph embedding features of the text nodes in the text semantic graph; and generate an alignment vector representing the semantic consistency between the text semantic and the image semantic based on the first consistency features and the second consistency features. The feature fusion module 1005 is configured to determine a first credibility weight of the text semantic and a second credibility weight of the image semantic according to the alignment vector, and fuse the text features and the image features according to the first credibility weight and the second credibility weight to obtain the fused features. The electronic book classification module 1006 is configured to determine the category of the target electronic book based on the fused features.

[0088] It should be noted that the one kind of multi-modal data classification system provided in the application and the above-mentioned one kind of multi-modal data classification method are based on the same inventive concept, so the content of the above-mentioned one kind of multi-modal data classification method is also applicable to the content of the one kind of multi-modal data classification system of the application embodiment, which will not be described here.

[0089] As Figure 3 The application embodiment further provides an electronic device, and the electronic device comprises: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes the at least one program to implement the one kind of multi-modal data classification method provided in the application.

[0090] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0091] The electronic device of the application embodiment will be described in detail below. Figure 3 The electronic device comprises: The processor 1600 can be implemented in a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute a related program to implement the technical solutions provided in the application embodiment. The memory 1700 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 1700 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1700 and are called and executed by the processor 1600 to implement a multi-modal based data classification method provided by the embodiments of the present disclosure.

[0092] The input / output interface 1800 is configured to realize information input and output. The communication interface 1900 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 2000 is configured to transmit information between various components (for example, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device. The processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are connected to each other through the bus 2000 to realize the communication link between them in the device.

[0093] The present disclosure also provides a storage medium, which is a computer readable storage medium, and stores computer executable instructions for causing a computer to execute the multi-modal based data classification method.

[0094] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be linked to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0095] The embodiments described in the present disclosure are used to more clearly illustrate the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions provided by the present disclosure. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present disclosure are also applicable to similar technical problems.

[0096] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present disclosure, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0097] The apparatus embodiments described above are merely illustrative, and units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0098] Those skilled in the art can understand that all or some steps in the above disclosed method, functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0099] The terms "first", "second", "third", "fourth" and the like used in the description of the specification and the above drawings (if any) are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0100] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0101] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication link between the units shown or discussed can be indirect coupling or communication link through some interfaces, apparatuses or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed to a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0103] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0104] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program storage media.

[0105] The above is a specific description of the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above described embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A multimodal data classification method, characterized in that: The method comprises the following steps: In response to a classification signal of a target e-book, extracting e-book text and an e-book image from the target e-book; the e-book image includes at least one of a cover image, illustrations, and chapter pages of the target e-book; Parsing the text semantics in the e-book text to obtain a text semantic graph, and performing feature extraction on the e-book text to obtain text features; Parsing the image semantics in the e-book image to obtain an image semantic graph, and performing feature extraction on the e-book image to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters, and scenes in the e-book image, and the edges are associations between the image nodes; Determine a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the corresponding image node in the image semantic graph, and determine a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the corresponding text node in the text semantic graph; and generate an alignment vector representing the semantic consistency of text semantics and image semantics based on the first consistency feature and the second consistency feature; Determining a first credibility weight of text semantics and a second credibility weight of image semantics based on the alignment vector, and fusing the text feature and the image feature based on the first credibility weight and the second credibility weight to obtain a fused feature; Based on the fusion features, the category of the target e-book is determined.

2. The multimodal data classification method according to claim 1, characterized in that: The determining of a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the corresponding image node in the image semantic graph, and determining a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the corresponding text node in the text semantic graph, includes: Encoding the text semantic graph and the image semantic graph using a graph neural network respectively to obtain first graph embedding features of text nodes in the text semantic graph and second graph embedding features of image nodes in the image semantic graph; determining cross-modal attention weights of the text node and the image node according to the first graph embedding feature and the second graph embedding feature; According to the respective cross-modal attention weights of the text node and the image node, a first consistency feature between the first graph embedding feature of the text node and the corresponding second graph embedding feature of the image node and a second consistency feature between the second graph embedding feature of the image node and the corresponding first graph embedding feature of the text node are generated.

3. The multimodal data classification method according to claim 2, characterized in that: The determining, according to the first graph embedding feature and the second graph embedding feature, the cross-modal attention weights of the text node and the image node includes: ； in, For the The first graph embedding features of text nodes, For the The second graph embedding features of image nodes, for The transpose of is the learnable parameter matrix, is the activation function, For the text nodes and Cross-modal attention weights for image nodes; The step of generating, according to respective cross-modal attention weights of the text node and the image node, a first consistency feature between the first graph embedding feature of the text node and the corresponding second graph embedding feature of the image node, and a second consistency feature between the second graph embedding feature of the image node and the corresponding first graph embedding feature of the text node, comprises: Generate a cross-modal attention weight matrix based on the cross-modal attention weights of the image node and the text node; The cross-modal attention weight matrix is ​​normalized by rows and columns to generate a first consistency feature between the first graph embedding feature of the text node and the corresponding second graph embedding feature of the image node, and a second consistency feature between the second graph embedding feature of the image node and the corresponding first graph embedding feature of the text node.

4. The multimodal data classification method according to claim 3, characterized in that: The step of generating an alignment vector representing semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature includes: The first weighted sum feature between the text node and all the image nodes is calculated according to the following formula: ； in, For the The first weighted sum feature corresponding to the text nodes, is the total number of image nodes, is a first consistency feature between the first graph embedding feature of the text node and the second graph embedding feature of the image node; The second weighted sum feature between the image node and all the text nodes is calculated according to the following formula: ； in, For the The second weighted sum feature corresponding to the image nodes, is the total number of text nodes, A second consistency feature between the second graph embedding feature of the image node and the first graph embedding feature of the text node; concatenating the first graph embedding feature and the first weighted sum feature to obtain a first concatenated feature; Concatenating the second graph embedding feature and the second weighted sum feature to obtain a second concatenated feature; Pooling the first image embedding feature, the second image embedding feature, the first splicing feature, and the second splicing feature respectively to obtain the pooled first image embedding feature, the second image embedding feature, the first splicing feature, and the second splicing feature; Calculating the similarity between each two groups of features in the pooled first image embedding features, the second image embedding features, the first splicing features, and the second splicing features to obtain multiple similarities; The alignment vector is obtained by concatenating the plurality of similarities.

5. The multimodal data classification method according to claim 4, characterized in that: Determining a first credibility weight of text semantics and a second credibility weight of image semantics based on the alignment vector includes: Input the alignment vector into a fully connected layer to obtain output hidden layer features; Inputting the hidden layer features into another fully connected layer to obtain an output text semantic score vector and an output image semantic score vector; Normalizing the scoring vector of the text semantics to obtain a first credibility weight of the text semantics; The scoring vector of the image semantics is normalized to obtain a second credibility weight of the image semantics; wherein the sum of the first credibility weight and the second credibility weight is equal to one.

6. The multimodal data classification method according to claim 5, characterized in that: The fusing of the text feature and the image feature according to the first credibility weight and the second credibility weight includes: The text features and the image features are fused by performing a weighted summation based on the second credibility weight and the image features, and the first credibility weight and the text features.

7. The multimodal data classification method according to claim 6, characterized in that: The extracting features of the e-book text to obtain text features includes: Perform feature extraction on the e-book text using BERT to obtain text features; The step of extracting features from the electronic book image to obtain image features includes: Feature extraction is performed on the e-book image using ResNet-50 to obtain image features.

8. A multimodal data classification system, characterized in that: The system comprises: an image and text acquisition module, configured to extract e-book text and e-book images from the target e-book in response to a classification signal of the target e-book; the e-book images comprising at least one of a cover image, illustrations, and chapter pages of the target e-book; A text semantic graph extraction module is used to parse the text semantics in the e-book text to obtain a text semantic graph, and perform feature extraction on the e-book text to obtain text features; an image semantic graph extraction module, configured to parse the image semantics in the e-book image to obtain an image semantic graph, and perform feature extraction on the e-book image to obtain image features; wherein the image nodes in the image semantic graph are at least one of objects, characters, and scenes in the e-book image, and the edges are associations between the image nodes; an alignment vector extraction module, configured to determine a first consistency feature between the graph embedding feature of each text node in the text semantic graph and the graph embedding feature of the image node in the image semantic graph, and to determine a second consistency feature between the graph embedding feature of each image node in the image semantic graph and the graph embedding feature of the text node in the text semantic graph; and to generate an alignment vector representing the semantic consistency between text semantics and image semantics based on the first consistency feature and the second consistency feature; a feature fusion module, configured to determine a first credibility weight of text semantics and a second credibility weight of image semantics based on the alignment vector, and fuse the text feature and the image feature based on the first credibility weight and the second credibility weight to obtain a fused feature; The e-book classification module is used to determine the category of the target e-book based on the fusion features.

9. An electronic device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The computer program is stored in the memory, and the processor executes at least one computer program to implement: A multimodal data classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute: A multimodal data classification method according to any one of claims 1 to 7.

Images