Document classification method and device based on multi-modal hypergraph clustering
By extracting features from multimodal documents and performing hypergraph structured modeling, this method solves the problem that traditional intelligent document classification methods cannot cover multimodal document scenarios, and achieves efficient intelligent document classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-08-07
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, information is mainly expressed through text, and the presentation methods are singular. Traditional intelligent document classification methods rely heavily on annotation and cannot cover multimodal document scenarios.
By extracting features from multimodal document data, constructing hypergraph structures for image and text modalities, performing hypergraph structured modeling and fusion, and using modularity for on-graph clustering, the final document classification results are obtained.
It enables intelligent classification of multimodal documents, improving the accuracy and efficiency of classification and reducing reliance on human resources.
Smart Images
Figure CN117216620B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data intelligent classification technology, and in particular to a document classification method and apparatus based on multimodal hypergraph clustering. Background Technology
[0002] In the era of big data, the medium for documents has transitioned from traditional paper-based media to online internet platforms. The massive storage capacity of these platforms has led to the rapid development of online documents. People now access documents not only through traditional media like newspapers and books, but also increasingly through mobile devices such as smartphones and laptops. This significant expansion of online document volume has also brought additional challenges, such as classification. Taking news media documents as an example, in the face of the explosive growth of internet media data, users need to sift through the chaotic news databases on internet news media platforms to find topics of interest. Traditional media classification of news topics is primarily done manually, which, while accurate, requires substantial human resources and is inefficient, especially with today's large-scale news media databases. Therefore, there is a widespread market demand for automated classification of large-scale documents using intelligent methods.
[0003] In related technologies, document summaries, keywords, and classification tags are obtained from classified documents, and the classification results are judged based on the calculated classification efficiency value.
[0004] However, in related technologies, information is mainly expressed through text, and the presentation methods are singular. Traditional intelligent document classification methods rely heavily on annotation and cannot cover multimodal document scenarios, so improvements are urgently needed. Summary of the Invention
[0005] This application provides a document classification method and apparatus based on multimodal hypergraph clustering to solve problems in related technologies, such as information expression mainly being text-based, presentation being singular, traditional document intelligent classification methods being heavily reliant on annotation, and failing to cover multimodal document scenarios.
[0006] The first aspect of this application provides a document classification method based on multimodal hypergraph clustering, comprising the following steps: extracting at least one image modality feature and at least one text modality feature for each document from a multimodal document dataset; performing hypergraph structuring modeling on the multimodal document dataset using the at least one image modality feature and the at least one text modality feature to obtain hypergraph structures of different modalities; and performing multimodal hypergraph fusion using the hypergraph structures of different modalities to obtain a fused hypergraph, and performing graph clustering on the fused hypergraph to obtain clustering results, so as to obtain a final document classification result based on the clustering results.
[0007] Optionally, in one embodiment of this application, the step of using the at least one image modality feature and the at least one text modality feature to perform hypergraph structured modeling on the multimodal document dataset to obtain hypergraph structures for different modalities includes: constructing hyperedges based on the at least one image modality feature using the K nearest neighbors of each sample to generate a structured hypergraph for the image modality; and constructing hyperedges based on the attributes of the at least one text modality feature to generate a structured hypergraph for the text modality.
[0008] Optionally, in one embodiment of this application, when constructing a hyperedge based on the attributes of a feature, samples that share the same attribute belong to the same hyperedge.
[0009] Optionally, in one embodiment of this application, performing graph clustering on the fused hypergraph to obtain clustering results includes: using the modularity of the fused hypergraph to partition the hypergraph internally to obtain the clustering results for all document samples.
[0010] Optionally, in one embodiment of this application, the formula for calculating the modularity is:
[0011]
[0012] Where z is a vector representing node labels, Ω is the affinity function, d is a vector storing node degrees, and p is a defined split vector.
[0013] A second aspect of this application provides a document classification apparatus based on multimodal hypergraph clustering, comprising: an extraction module for extracting at least one image modality feature and at least one text modality feature for each document from a multimodal document dataset; a construction module for performing hypergraph structuring modeling on the multimodal document dataset using the at least one image modality feature and the at least one text modality feature to obtain hypergraph structures of different modalities; and a classification module for performing multimodal hypergraph fusion with the hypergraph structures of different modalities to obtain a fused hypergraph, and performing graph clustering on the fused hypergraph to obtain a clustering result, so as to obtain a final document classification result based on the clustering result.
[0014] Optionally, in one embodiment of this application, the construction module includes: a first generation unit, configured to construct hyperedges based on the at least one image modality feature, using the K nearest neighbors of each sample to generate a structured hypergraph of the image modality; and a second generation unit, configured to construct hyperedges based on the attributes of the at least one text modality feature to generate a structured hypergraph of the text modality.
[0015] Optionally, in one embodiment of this application, when constructing a hyperedge based on the attributes of a feature, samples that share the same attribute belong to the same hyperedge.
[0016] Optionally, in one embodiment of this application, the classification module includes: a partitioning unit, used to partition the hypergraph internally using the modularity of the fused hypergraph to obtain the clustering result of all document samples.
[0017] Optionally, in one embodiment of this application, the formula for calculating the modularity is:
[0018]
[0019] Where z is a vector representing node labels, Ω is the affinity function, d is a vector storing node degrees, and p is a defined split vector.
[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the document classification method based on multimodal hypergraph clustering as described in the above embodiments.
[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the document classification method based on multimodal hypergraph clustering as described above.
[0022] This application's embodiments can intelligently classify documents by extracting features from different modal data of a document, performing hypergraph structured modeling for each modality, fusing the hypergraphs of different modalities, and performing graph clustering on the fused hypergraph. This solves the problems in related technologies, such as information expression being primarily text-based, having a single presentation method, and traditional intelligent document classification methods being heavily reliant on annotations and unable to cover multimodal document scenarios.
[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0025] Figure 1 This is a flowchart illustrating a document classification method based on multimodal hypergraph clustering according to an embodiment of this application;
[0026] Figure 2 This is a schematic diagram illustrating the working principle of the document classification method based on multimodal hypergraph clustering according to an embodiment of this application;
[0027] Figure 3This is a schematic diagram of multimodal document data according to the document classification method based on multimodal hypergraph clustering according to an embodiment of this application;
[0028] Figure 4 This is an example diagram of a document segmentation device based on multimodal hypergraph clustering according to an embodiment of this application;
[0029] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0031] The following description, with reference to the accompanying drawings, illustrates a document classification method and apparatus based on multimodal hypergraph clustering, according to embodiments of this application. Addressing the issues raised in the background section regarding the related technologies where information expression is primarily text-based, presentation methods are limited, traditional intelligent document classification methods rely heavily on annotation, and cannot cover multimodal document scenarios, this application provides a document classification method based on multimodal hypergraph clustering. In this method, features are extracted from different modal data of the document, and a hypergraph structure model is performed for each modality. The hypergraphs of different modalities are then fused, and graph clustering is performed on the fused hypergraph, thereby intelligently classifying the document based on the results of the graph clustering. This solves the problems in the related technologies where information expression is primarily text-based, presentation methods are limited, traditional intelligent document classification methods rely heavily on annotation, and cannot cover multimodal document scenarios.
[0032] Specifically, Figure 1 This is a flowchart illustrating a document classification method based on multimodal hypergraph clustering provided in an embodiment of this application.
[0033] like Figure 1 As shown, this document classification method based on multimodal hypergraph clustering includes the following steps:
[0034] In step S101, at least one image modality feature and at least one text modality feature are extracted from each document in the multimodal document dataset.
[0035] Specifically, embodiments of this application can obtain a document dataset D = {x} i |i=1,,n}, where x i This represents the i-th document, where each document contains data x in two modalities. i ={d i ,ti}, where d i and t i The dataset represents image modal data and text modal data respectively, and modal features are extracted for each document's image modal data. Since the dataset lacks label information and is unsupervised, a pre-trained model is used for image feature extraction. In this embodiment, a ResNet52 pre-trained model based on ImageNet can be used for feature extraction, and the 1024-dimensional vector output from the final fully connected layer is used as the feature d for extracting the document sample image modalities. i ∈R 1024 .
[0036] Furthermore, embodiments of this application can extract modal features from the text modal data of each document. First, the text analysis library jieba is used to extract nouns and filter sparse nouns from all document text data. This allows the extraction of text modal features from all document samples, with each document sample's text modal features consisting of a series of generated nouns.
[0037] In step S102, hypergraph structuring modeling is performed on the multimodal document dataset using at least one image modality feature and at least one text modality feature to obtain hypergraph structures of different modalities.
[0038] The methods for generating structured hypergraphs for image and text modalities will be described in detail below.
[0039] Optionally, in one embodiment of this application, a hypergraph structure model is performed on a multimodal document dataset using at least one image modality feature and at least one text modality feature to obtain hypergraph structures for different modalities, including: constructing hyperedges based on the K nearest neighbors of each sample based on at least one image modality feature to generate a structured hypergraph for the image modality; and constructing hyperedges based on the attributes of at least one text modality feature to generate a structured hypergraph for the text modality.
[0040] Specifically, after obtaining the image features {d} of all document samples i After |i=1,...,n}, the embodiments of this application can perform hypergraph structured modeling based on this feature. A hypergraph is an extension of a regular graph structure. The difference is that, compared to the edges in a regular graph (each edge can only connect 2 nodes), each hyperedge in a hypergraph can connect any number of nodes. That is, each hyperedge in a hypergraph can be regarded as a subset of the node set.
[0041] Hypergraph structured modeling is based on the feature d of each sample. i The nearest neighbors are calculated, and the samples of the top K nearest neighbors are selected and compared with sample d. iThese nodes together form a hyperedge. The nearest neighbor calculation uses a simple Euclidean distance. This allows us to model a hypergraph of document visual modalities with n hyperedges, each containing K+1 nodes. The nearest neighbor matrix of the hypergraph can be represented as H. d ∈R n×n Each row represents a node, and each column represents a hyperedge, where:
[0042]
[0043] Where N K (d j ) represents d j The set of K nearest neighbors.
[0044] Optionally, in one embodiment of this application, when constructing a hyperedge based on the attributes of a feature, samples that share the same attribute belong to the same hyperedge.
[0045] For example, embodiments of this application can analyze the nouns extracted from all samples, select samples with the same noun attributes to construct a hyperedge, that is, construct a hypergraph using the filtered nouns as hyperedges. The nearest neighbor matrix of the hypergraph for the text modality can be represented as H. t ∈R n×m , where m is the total number of nouns generated in the dataset.
[0046] In step S103, multi-hypergraph fusion is performed using hypergraph structures with different modalities to obtain a fused hypergraph, and graph clustering is performed on the fused hypergraph to obtain clustering results, so as to obtain the final document classification results based on the clustering results.
[0047] In actual implementation, the embodiments of this application can process the hypergraph H of the generated image modality. d Hypergraph H of text modality t To achieve fusion, embodiments of this application may employ a parallel-join method, enabling the fused hypergraph to simultaneously retain the structural information of both hypergraphs:
[0048] H = [H] d H t ]
[0049] Where the fusion hypergraph H∈R n×(n+m) .
[0050] Optionally, in one embodiment of this application, graph clustering is performed on the fused hypergraph to obtain clustering results, including: using the modularity of the fused hypergraph to partition the hypergraph internally to obtain clustering results for all document samples.
[0051] Specifically, embodiments of this application can use the modularity of a hypergraph to partition the area within the hypergraph, thereby obtaining clustering results for all document samples. The formula for calculating the modularity will be explained in detail below.
[0052] Where z is a vector representing node labels, Ω is the affinity function used to control the probability of placing a hyperedge in a node set, d is a vector storing the degree of each node (the degree of a node: the number of edges connecting that node), and p is a defined partitioning vector used to store the number of clusters in the node set rather than their specific distribution. To make the algorithm faster, the All-Or-Nothing (AON) affinity function is used:
[0053]
[0054] The AON affinity function only considers whether a given edge is completely contained within a single cluster. Substituting the affinity function into the definition of modularity yields a variant of modularity:
[0055]
[0056] J(ω) represents the part independent of the segmentation z, and k is the size of the hyperedge. The modularity of the hypergraph is derived from a Poisson-corrected hypergraph random block model, which is applied to Louvain-type algorithms to solve the hypergraph segmentation task.
[0057] The Louvain-type algorithm in this embodiment consists of two phases: In the first phase, all nodes in the hypergraph are considered as independent communities, and the number of communities is equal to the number of nodes. For each node i, it is attempted to move it to a community of an adjacent node, retaining the community that maximizes the modularity after the move. If no such move is found, the label of i remains unchanged. This process is repeated until the labels of all nodes remain stable. In the second phase, nodes within each cluster are merged into a supernode. The weight of the supernode is converted to the sum of the weights of all merged nodes, and the weights between communities are converted to the weights of the edges between supernodes. These two phases are then repeated until the labels of the nodes no longer change.
[0058] After obtaining the label of each node, in order to obtain the semantics of each label, that is, the category name to which all documents under the label belong, this embodiment of the application can analyze the key text noun features of all samples under each label, determine the semantics of the label by majority voting on all its nouns, and classify all documents under the label into that category.
[0059] Optionally, in one embodiment of this application, the formula for calculating modularity is:
[0060]
[0061] Where z is a vector representing node labels, Ω is the affinity function, d is a vector storing node degrees, and p is a defined split vector.
[0062] Combination Figure 2 and Figure 3 As shown, an embodiment is used to illustrate in detail the working principle of the document classification method based on multimodal hypergraph clustering of this application.
[0063] like Figure 2 As shown, this document classification method based on multimodal hypergraph clustering may include the following steps:
[0064] Step S201: Obtain the document dataset D = {x} i |i=1,,n}, where x i This represents the i-th document, where each document contains data x in two modalities. i ={d i ,ti}, where d i and t i These represent image modal data and text modal data, respectively.
[0065] Step S202: Extract key features from the image modality data of each document. Since the dataset lacks label information and is an unsupervised dataset, a pre-trained model can be used for image feature extraction. In this embodiment, a ResNet52 pre-trained model based on ImageNet can be used for feature extraction, and the 1024-dimensional vector output from the final fully connected layer is used as the feature d extracted from the document sample image modality. i ∈R 1024 .
[0066] Obtain the image features {d} of all document samples i After |i=1,...,n}, the embodiments of this application can perform hypergraph structured modeling based on this feature. A hypergraph is an extension of a regular graph structure. The difference is that, compared to the edges in a regular graph (each edge can only connect 2 nodes), each hyperedge in a hypergraph can connect any number of nodes. That is, each hyperedge in a hypergraph can be regarded as a subset of the node set.
[0067] Hypergraph structured modeling is based on the feature d of each sample. i The nearest neighbors are calculated, and the samples of the top K nearest neighbors are selected and compared with sample d. i These nodes together form a hyperedge. The nearest neighbor calculation uses a simple Euclidean distance. This allows us to model a hypergraph of document visual modalities with n hyperedges, each containing K+1 nodes. The nearest neighbor matrix of the hypergraph can be represented as H. d ∈R n×nEach row represents a node, and each column represents a hyperedge, where:
[0068]
[0069] Where N K (d j ) represents d j The set of K nearest neighbors.
[0070] Furthermore, embodiments of this application can extract key features from the text modality data of each document. Embodiments of this application first use the text analysis library jieba to extract nouns and filter sparse nouns from all document text data. This allows for the extraction of key text features from all document samples, with each document sample's key text features consisting of a series of generated nouns.
[0071] Furthermore, embodiments of this application can analyze the nouns extracted from all samples, select samples with the same noun attributes to construct a hyperedge, that is, construct a hypergraph using the filtered nouns as hyperedges. The nearest neighbor matrix of the hypergraph for the text modality can be represented as H. t ∈R n×m , where m is the total number of nouns generated in the dataset.
[0072] Step S203: Calculate the hypergraph H of the generated image modalities. d Hypergraph H of text modality t The process involves merging the two hypergraphs. Embodiments of this application may employ a combination-merge method, ensuring that the merged hypergraph retains the structural information of both hypergraphs simultaneously.
[0073] H = [H] d H t ]
[0074] Where the fusion hypergraph H∈R n×(n+m) .
[0075] Step S204: Perform graph clustering on the fused hypergraph. In this embodiment, the modularity of the hypergraph can be used to partition the hypergraph internally to obtain the classification results for all document samples. The modularity of the hypergraph is defined as:
[0076]
[0077] Where z is a vector representing node labels, Ω is the affinity function used to control the probability of placing a hyperedge in a node set, d is a vector storing the degree of each node (the degree of a node: the number of edges connecting that node), and p is a defined partitioning vector used to store the number of clusters in the node set rather than their specific distribution. To make the algorithm faster, the All-Or-Nothing (AON) affinity function is used:
[0078]
[0079] The AON affinity function only considers whether a given edge is completely contained within a single cluster. Substituting the affinity function into the definition of modularity yields a variant of modularity:
[0080]
[0081] J(ω) represents the part independent of the segmentation z, and k is the size of the hyperedge. The modularity of the hypergraph is derived from a Poisson-corrected hypergraph random block model, which is applied to Louvain-type algorithms to solve the hypergraph segmentation task.
[0082] The Louvain-type algorithm in this embodiment consists of two phases: In the first phase, all nodes in the hypergraph are considered as independent communities, and the number of communities is equal to the number of nodes. For each node i, it is attempted to move it to a community of an adjacent node, retaining the community that maximizes the modularity after the move. If no such move is found, the label of i remains unchanged. This process is repeated until the labels of all nodes remain stable. In the second phase, nodes within each cluster are merged into a supernode. The weight of the supernode is converted to the sum of the weights of all merged nodes, and the weights between communities are converted to the weights of the edges between supernodes. These two phases are then repeated until the labels of the nodes no longer change.
[0083] After obtaining the label of each node, in order to obtain the semantics of each label, that is, the category name to which all documents under the label belong, the key text noun features of all samples under each label are analyzed, and the semantics of the label are determined by majority voting of all its nouns. At the same time, all documents under the label are classified into that category.
[0084] in, Figure 3 This is a schematic diagram of multimodal document data.
[0085] The document classification method based on multimodal hypergraph clustering proposed in this application can intelligently classify documents by extracting features from different modal data of the document, performing hypergraph structured modeling for each modality, fusing the hypergraphs of different modalities, and performing graph clustering on the fused hypergraph. This solves the problems in related technologies, such as the information expression being primarily text-based, the presentation being monotonous, and the traditional intelligent document classification methods relying heavily on annotations and failing to cover multimodal document scenarios.
[0086] Next, referring to the accompanying drawings, a document classification apparatus based on multimodal hypergraph clustering proposed according to an embodiment of this application is described.
[0087] Figure 4This is a block diagram of a document classification device based on multimodal hypergraph clustering according to an embodiment of this application.
[0088] like Figure 4 As shown, the document classification device 10 based on multimodal hypergraph clustering includes: an extraction module 100, a construction module 200, and a classification module 300.
[0089] The extraction module 100 is used to extract at least one image modality feature and at least one text modality feature from each document in the multimodal document dataset.
[0090] Module 200 is used to perform hypergraph structuring modeling on a multimodal document dataset using at least one image modality feature and at least one text modality feature to obtain hypergraph structures of different modalities.
[0091] The classification module 300 is used to perform multi-hypergraph fusion with different modal hypergraph structures to obtain a fused hypergraph, and to perform graph clustering on the fused hypergraph to obtain clustering results, so as to obtain the final document classification result based on the clustering results.
[0092] Optionally, in one embodiment of this application, the construction module 200 includes: a first generation unit and a second generation unit.
[0093] The first generation unit is used to construct a hyperedge based on the K nearest neighbors of each sample, based on at least one image modality feature, to generate a structured hypergraph of the image modality.
[0094] The second generation unit is used to construct hyperedges based on the attributes of at least one text modality feature, thereby generating a structured hypergraph of the text modality.
[0095] Optionally, in one embodiment of this application, when constructing a hyperedge based on the attributes of a feature, samples that share the same attribute belong to the same hyperedge.
[0096] Optionally, in one embodiment of this application, the classification module 300 includes: a partitioning unit.
[0097] The partitioning unit is used to partition the hypergraph within the hypergraph using the modularity of the fused hypergraph, thereby obtaining the clustering results of all document samples.
[0098] Optionally, in one embodiment of this application, the formula for calculating modularity is:
[0099]
[0100] Where z is a vector representing node labels, Ω is the affinity function, d is a vector storing node degrees, and p is a defined split vector.
[0101] It should be noted that the foregoing explanation of the document classification method based on multimodal hypergraph clustering also applies to the document classification device based on multimodal hypergraph clustering in this embodiment, and will not be repeated here.
[0102] The document classification device based on multimodal hypergraph clustering proposed in this application can intelligently classify documents by extracting features from different modal data of documents, performing hypergraph structured modeling for each modality, fusing the hypergraphs of different modalities, and performing graph clustering on the fused hypergraph. This solves the problems in related technologies, such as information expression being primarily text-based, having a single presentation method, and traditional intelligent document classification methods relying heavily on annotations and failing to cover multimodal document scenarios.
[0103] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:
[0104] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0105] When the processor 502 executes the program, it implements the document classification method based on multimodal hypergraph clustering provided in the above embodiments.
[0106] Furthermore, electronic devices also include:
[0107] Communication interface 503 is used for communication between memory 501 and processor 502.
[0108] The memory 501 is used to store computer programs that can run on the processor 502.
[0109] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0110] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0111] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0112] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the document classification method based on multimodal hypergraph clustering as described above.
[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0116] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0118] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0121] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A document classification method based on multimodal hypergraph clustering, characterized in that, Includes the following steps: Extract at least one image modal feature and at least one text modal feature from each document in the multimodal document dataset; Hypergraph structuring modeling is performed on the multimodal document dataset using at least one image modality feature and at least one text modality feature to obtain hypergraph structures of different modalities; as well as Multi-hypergraph fusion is performed using the hypergraph structures of different modalities to obtain a fused hypergraph, and graph clustering is performed on the fused hypergraph to obtain clustering results, so as to obtain the final document classification results based on the clustering results; The process of using at least one image modality feature and at least one text modality feature to perform hypergraph structuring modeling on the multimodal document dataset to obtain hypergraph structures of different modalities includes: Based on the at least one image modality feature, a hyperedge is constructed using the K nearest neighbors of each sample to generate a structured hypergraph of the image modality; Based on the at least one text modality feature, hyperedges are constructed using the attributes of the features to generate a structured hypergraph of the text modality; The step of performing graph clustering on the fused hypergraph to obtain clustering results includes: The modularity of the fused hypergraph is used to partition the hypergraph, resulting in the clustering results for all document samples. The formula for calculating the modularity is: ， in, A vector representing the node label. For affinity function, A vector for storing node degrees. The defined segmentation vector.
2. The method according to claim 1, characterized in that, When constructing hyperedges based on the attributes of features, samples that share the same attributes belong to the same hyperedge.
3. A document classification device based on multimodal hypergraph clustering, characterized in that, The document classification device based on multimodal hypergraph clustering is used to implement the document classification method based on multimodal hypergraph clustering as described in any one of claims 1-2, wherein the document classification device based on multimodal hypergraph clustering comprises: The extraction module is used to extract at least one image modal feature and at least one text modal feature for each document from the multimodal document dataset; A construction module is used to perform hypergraph structuring modeling on the multimodal document dataset using at least one image modality feature and at least one text modality feature to obtain hypergraph structures for different modalities; and The classification module is used to perform multi-hypergraph fusion with the hypergraph structures of different modalities to obtain a fused hypergraph, and to perform graph clustering on the fused hypergraph to obtain clustering results, so as to obtain the final document classification result based on the clustering results; The building module includes: The first generation unit is used to construct a hyperedge with the K nearest neighbors of each sample based on the at least one image modality feature, and generate a structured hypergraph of the image modality. The second generation unit is used to construct hyperedges based on the attributes of the at least one text modality feature, and generate a structured hypergraph of the text modality.
4. The apparatus according to claim 3, characterized in that, When constructing hyperedges based on the attributes of features, samples that share the same attributes belong to the same hyperedge.
5. An electronic device, characterized in that, include: The document classification method based on multimodal hypergraph clustering as described in any one of claims 1-2 includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the document classification method based on multimodal hypergraph clustering as described in any one of claims 1-2.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the document classification method based on multimodal hypergraph clustering as described in any one of claims 1-2.