Multi-modal classification method fusing graph convolutional neural network and capsule graph neural network

Abstract

The invention discloses a multi-modal classification method fusing a graph convolutional neural network and a capsule graph neural network, and solves the problem of how to apply an artificial intelligence technology to an appearance patent. The method comprises the following steps: coding a patent text sequence by utilizing BERT, segmenting semantic graph convolution by utilizing PCNN, constructing a dependency tree by utilizing grammar graph convolution, carrying out interactive attention and relationship classification, and finally carrying out image feature extraction through a capsule network. According to the method, the appearance patent classification technology fusing heterogeneous multi-modal information is researched, the feature extraction and fusion problems of multi-modal information such as characters, images and networks are solved, a heterogeneous multi-modal appearance design patent classification model is constructed, and the classification accuracy is greatly improved relative to pure text content. And meanwhile, an appearance design patent AI standard data set is constructed on the basis of the intellectual property hypergraph network, so that the blank of patent types in a traditional Lankyo classification system is filled, and the bottleneck of a traditional appearance patent classification method is broken through.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal information analysis and modeling such as text and images, and in particular to a multimodal information analysis and modeling classification method. Background Art

[0002] The classification level of existing design patent documents is simple and broad, which reduces the accuracy of searching by category. The huge patent document database needs a clear classification system to manage. The existing design classification system has the following defects: 1. The classification level is relatively simple, and the product range included in each lowest level is too broad; 2. The number of products included in the classification table is small, and many product types are not included in the classification table, resulting in different and non-similar products being classified into the same subcategory; 3. Some subcategories in the classification table are not detailed enough, and cover too many product types; 4. The distribution of design patent applications in categories is extremely uneven. The above defects make it impossible to distinguish them by classification numbers, which easily causes confusion between products and reduces the accuracy of classifying designs by category.

[0003] With the advancement of artificial intelligence technology, semantic understanding technology in natural language processing technology has emerged. This technology can deeply understand and analyze semantic information in texts, including entity recognition, relationship extraction, sentiment analysis, etc. How to apply artificial intelligence technology to appearance patents and build an AI standard data set for design patents based on the intellectual property hypergraph network to fill the gaps in patent types in the traditional Locarno classification system, thereby breaking through the bottleneck of traditional appearance patent classification methods, has become a problem to be solved. Summary of the invention

[0004] This application provides a multimodal classification method that integrates graph convolutional neural networks and capsule graph neural networks, aiming to apply artificial intelligence technology to appearance patents. At the same time, it constructs an AI standard dataset for design patents based on the intellectual property hypergraph network to fill the gaps in patent types in the traditional Locarno classification system, thereby breaking through the bottleneck of traditional appearance patent classification methods.

[0005] In the first aspect, a multimodal classification method integrating a graph convolutional neural network and a capsule graph neural network includes the following steps 1 to 5.

[0006] Step 1: Represent the patent text as a character sequence S = C1,...,C e , where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and incorporating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain a vector representation of each character.

[0007] Step 2: Use PCNN to segment each paragraph or sentence in the text to extract the local features of each paragraph or sentence, which reflects the detailed information inside the text. Then, use semantic graph convolution to model and integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation.

[0008] A segmented convolutional neural network (PCNN) with a gated attention mechanism is used to capture the relative position information between characters in a text sequence, understand the contextual relationship, and obtain the long-term dependency features of the sentence.

[0009] Step 3: Parse the sentence grammatical structure in the patent text, build a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information. During the grammatical graph convolution process, the nodes on the dependency tree will fuse information with other nodes that are grammatically related to the node, so that the representation of each node not only contains the node's own information, but also fuses the information of other nodes that are grammatically related to the node.

[0010] Among them, the pruning rule for pruning the grammatical dependency tree is: prune the original dependency tree, only retain the dependency relationship directly connected to the two entities, and construct a local grammatical dependency tree.

[0011] Step 4: Set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of the specific entity, use the interactive attention mechanism to discover the potential relationship between different elements in the text, extract specific entity features from the context information, concatenate the feature vectors after the entity features are fused, and use it as the final representation of the specific entity.

[0012] Step 5: Preliminarily extract patent image features through the multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, use the multi-dimensional vector neurons in the capsule network to represent more features of the target, capture the part-to-whole relationship in the image, and guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

[0013] In the above scheme, optionally, step 1 includes the following steps 101-104.

[0014] Step 101: Encode the entire input patent text to obtain the corresponding vector representation. The text encoding formula is as follows:

[0015] H s =CharEmbedding(S)

[0016] in, represents the word embedding representation of sentence S after BERT encoding, d c Represents the dimensions of a character vector.

[0017] Step 102: Use the relative distance between the character and the entity to represent the position feature, reduce the distance between the character and the entity, and obtain richer information representation. The relative position expression formula of the head entity and each character is as follows:

[0018]

[0019] in, Indicates the distance between the head entity and each character, b h Represents the head entity e head The index position of the first character, e h Represents the head entity e head The index position of the last character. Similarly, the relative position of the tail entity can be obtained. t represents the relative distance from the current character to the head entity and the tail entity.

[0020] Step 103: Then, the relative position information is encoded by word vectors to obtain a vector representation of the relative position information. The word vector encoding expression formula is as follows:

[0021]

[0022] H tail =PositionEmbedding tail (P t tail )

[0023] in, Represents the relative position vector of the head entity and each character, Represents the tail entity and the position vector of each character, d p Represents the dimension of the relative position vector, P t head Indicates the distance between the head entity and each character, P t tail Indicates the distance between the tail entity and each character.

[0024] Step 104: Embed the word after BERT encoding into H s , the vector representation of the head and tail entities and the vector representation of the relative position of the head and tail entities are concatenated to obtain the final representation H of the sequence encoding module c , the splicing expression formula is as follows:

[0025] H c =[H s ;H entl ;Hent2 ;H head ;H tail ]

[0026] H entl The word representing the head entity embedding, H ent2 Word representing the tail entity embedding.

[0027] In the above scheme, optionally, the step 2 includes the following steps 201-203.

[0028] Step 201: The gated linear unit GLU and attention are integrated into a unified layer through the gated attention mechanism GAU, which maximizes the sharing of calculation parameters between the two, effectively improves the calculation efficiency of the attention score, and captures more advanced semantic representations. The integrated expression formula is as follows:

[0029]

[0030] Among them, H att represents the output of the gated attention mechanism GAU, w att represents the weight matrix, U represents the output feature of the gated linear unit, V represents the feature representation of the hidden state input to the fully connected layer, and A sem represents the attention weight matrix, and ⊙ represents the XOR operation.

[0031] Step 202: GAU output H att And initialize the attention weight A sem,0 Input into GCN, GCN updates the node representation of the next layer according to the hidden representation of the neighboring nodes. The update process is expressed as follows:

[0032]

[0033] Among them, the attention weight matrix Represented by the matrix After standardization, ReLU represents a nonlinear activation function, and the input It represents the first l-1 layers of GCN training generated by node i, represents the linear transformation weight, Represents the linear transformation bias term.

[0034] Step 203: In order to better fuse the global and local semantic features of the sentence, the features are adaptively fused through a gating mechanism. The gating mechanism consists of a gated Tanh unit GTU and a gated linear unit GLU, which are used to retain valid features and filter out useless information during the fusion process. The fusion expression formula is as follows:

[0035] g = sigmoid(w gl tanh(wg2 H sem +b g2 )+b gl )

[0036] R sem = g·H p

[0037] Among them, sigmoid and tanh represent nonlinear activation functions, w gl and w g2 represents the weight matrix, b gl and b g2 Represents the bias, and finally obtains the local global semantic representation R through the gating mechanism sem , H p represents the final output of the segmented convolutional network, and g represents the feature output of the adaptive fusion of the gating mechanism.

[0038] In the above scheme, optionally, the step 3 includes the following steps 301-303.

[0039] Step 301: Use the dependency parser to obtain a grammatical dependency tree, and construct an adjacency matrix A of the sentence based on the grammatical dependency tree. syn , in order to construct the graph neural network structure in the relationship extraction task, element A syn,ij The calculation formula is as follows:

[0040]

[0041] Among them, the adjacency matrix A syn The main diagonal elements of are set to 1 to retain the information of their own nodes, constructing an undirected graph, c i ,c j Represents the text sequence of i and its neighboring nodes in the dependency tree.

[0042] Step 302: The adjacency matrix A constructed above syn Input into the graph convolutional network, and update the node representation of the next layer according to the hidden representation of the neighboring nodes. The hidden representation calculation formula is as follows:

[0043]

[0044] Among them, ReLU represents a nonlinear activation function, i represents the current node, j represents the neighboring node of node i, represents the degree of the i-th node in the dependency tree, represents the linear transformation weight, represents the linear transformation bias term, represents the hidden representation of the previous layer of GCN of node i, ΣA represents the total value of the adjacency matrix, and h l-1 Indicates the GCN training result of the previous layer of the node.

[0045] Step 303: Through the operation of L-layer grammatical graph convolution, each node continuously aggregates the information of its neighboring nodes and updates its own representation accordingly, and finally obtains the grammatical representation. The aggregation formula is as follows:

[0046]

[0047] represents the operation result of the L-layer grammar graph convolution, ... represents the many nodes of the L-layer, The hidden representation of the GCN representing node n.

[0048] In the above scheme, optionally, step 4 includes the following steps 401-403.

[0049] Step 401: embedding the word after BERT encoding H s Representation of specific entities Interact and get the grammatical interaction feature vector R syn , the specific formula is as follows:

[0050]

[0051] β syn represents the specific entity features extracted from the context information using interactive attention, α i Indicates H s the results of interacting with a particular entity;

[0052] Step 402: The soft-max function is used to perform final relationship classification on the fused semantic features and the generated results of the grammatical interaction module. The classification formula is as follows:

[0053] r=[R sem ; R syn ]

[0054]

[0055] Among them, “;” represents the concatenation of vectors, w p and b p represents the trainable parameters, Represents the result of relation classification after grammatical interaction, R syn represents the grammatical interaction feature vector, R sem It indicates that the local global semantics is obtained through the gating mechanism, and r indicates the result of the fusion of semantic features and grammatical interaction modules;

[0056] Step 403: Use cross entropy to minimize the target loss function. The calculation formula is as follows:

[0057]

[0058] Among them, s represents all training data, y i represents the true label, Represents the predicted label, λ represents the L2 regularization term coefficient, and θ represents the trainable parameter.

[0059] In the above scheme, optionally, the step 5 includes the following steps 501-504.

[0060] Step 501: Perform convolution operation on the input image data through a multi-scale convolution kernel to generate a feature map. The feature map is nonlinearized by an activation function to generate output features and input to the next layer. The convolution operation formula is as follows:

[0061]

[0062] Among them, * indicates the dot product operation between the convolution kernel and the local area. represents the output feature of the jth neuron in feature map i in layer l+1, and The weight and bias of the i-th convolution kernel in layer l are respectively l (j) represents the feature of the jth neuron in the feature map of the lth layer;

[0063] Step 502: Use capsule vector neurons as the input and output of the model. The structure of the capsule network consists of a common convolution layer, a main capsule layer, and a digital capsule layer. First, use the common convolution layer to perform a convolution operation on the input data to extract primary features. The operation formula of the common convolution layer is as follows:

[0064]

[0065] in, represents the i-th kernel of the l-th layer, and w represents the size of the convolution kernel; r j represents the jth receptive field, Z l(i.j) represents the convolution result of the lth layer, x l() Represents the eigenvalue of the feature map in layer l.

[0066] Step 503: Use the main capsule layer to further extract the primary features extracted by the convolution layer into high-level features, and form a main capsule layer after encapsulation. The main capsule is mainly composed of the output of the convolution layer and the squeezing function. Its calculation formula is as follows:

[0067]

[0068] Among them, u l(i,j) represents the main capsule, represents the output of the convolutional layer, fs () represents the squeezing function.

[0069] Step 504: Use the linear transformation of the digital capsule layer to transfer between digital capsule layers. The calculation formula is as follows:

[0070]

[0071] Among them, u i represents the i-th input capsule; W ij represents the weight matrix; Represents the prediction vector.

[0072] In the above scheme, further optionally, before step 502, it also includes: using a rectified linear unit RELU activation function to allow the network to nonlinearly map the input signal to enhance the expression ability of the network. The activation function expression formula is as follows:

[0073]

[0074] in, represents the activation value after the activation function, represents the output feature of the jth neuron in the feature map i in the l+1th layer, f a () represents the convolution function, and max{} represents the maximum value selection function.

[0075] In the above scheme, optionally, before step 503, it also includes: using batch normalization to adjust the input distribution of data between layers within the network, and using the distribution of input data of each layer of the network to standardize to accelerate the convergence of the network. The batch normalization formula is as follows:

[0076]

[0077] Among them, y (i) represents the output of the neuron response, x (i) represents the eigenvalue of the input data point, γ (i) represents the scale factor, Represents x (i) The variance, E[x (i) ] represents x (i) The mathematical expectation of (i) Represents the translation factor.

[0078] In a second aspect, a multimodal classification device integrating a graph convolutional neural network and a capsule graph neural network, the device comprising:

[0079] BERT patent text sequence encoding module, used to represent patent text as a character sequence S = C1,...,C e, where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and incorporating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain a vector representation of each character.

[0080] The PCNN segmented semantic graph convolution module is used to segment each paragraph or sentence in the text through PCNN to extract the local features of each paragraph or sentence. These local features reflect the detailed information inside the text. Subsequently, the semantic graph convolution is used to model and integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation. In order to further capture the relative position relationship between characters in the text sequence, a segmented convolutional neural network PCNN with a gated attention mechanism is used to understand the contextual relationship and obtain the long-term dependency features of the sentence.

[0081] The grammatical graph convolution constructs a dependency tree module, which is used to parse the grammatical structure of sentences in patent texts, construct a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information; during the grammatical graph convolution process, the nodes on the dependency tree will fuse information with their grammatically related nodes, so that the representation of each node not only contains its own information, but also fuses the information of its grammatically related nodes; among them, the pruning rules for pruning the grammatical dependency tree are: prune the original dependency tree, only retain the dependency relationship directly connected to the two entities, and construct a local grammatical dependency tree.

[0082] The interactive attention and relationship classification module is used to set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of a specific entity, and use the interactive attention mechanism to discover the potential relationship between different elements in the text and extract the features of the specific entity from the context information. The fused semantic features and the feature vector generated by the grammatical interaction module are concatenated and the fused feature vector is used as the final representation of the specific entity.

[0083] The capsule network image feature extraction module is used to preliminarily extract patent image features through a multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, the multi-dimensional vector neurons in the capsule network are used to represent more features of the target, capture the part-to-whole relationship in the image, guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

[0084] In a third aspect, a computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0085] In a fourth aspect, a computer program product comprises a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0086] Compared with the prior art, this application has at least the following beneficial effects:

[0087] The present invention is based on the intellectual property hypergraph network, and adopts graph convolutional neural network and capsule graph neural network models to realize a multimodal and cross-domain appearance design patent classification algorithm for text, image and network, which can improve the accuracy of traditional pure text content retrieval and effectively classify appearance design patents at all levels in all fields; at the same time, the present invention will combine theoretical models such as knowledge graphs, deep learning, and breadth learning, integrate cross-domain, multimodal and heterogeneous features, establish an AI standard data set for appearance design patents, fill the gaps in patent types in the traditional Locarno classification system, and improve the accuracy of the traditional Locarno classification method, thereby breaking through the bottleneck of the traditional appearance patent classification method. This application can be used to serve large-scale, field-specific, and multimodal appearance design patent retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A flowchart of a multimodal classification method that integrates a graph convolutional neural network and a capsule graph neural network provided in one embodiment of the present application;

[0089] Figure 2 A vector neuron structure diagram of a multimodal classification method that integrates a graph convolutional neural network and a capsule graph neural network provided in one embodiment of the present application;

[0090] Figure 3 A capsule network layer structure diagram of a multimodal classification method that integrates a graph convolutional neural network and a capsule graph neural network provided in one embodiment of the present application;

[0091] Figure 4 A structural diagram of a multimodal classification method that integrates a graph convolutional neural network and a capsule graph neural network provided in one embodiment of the present application;

[0092] Figure 5 A block diagram of the module architecture of a multimodal classification device that integrates a graph convolutional neural network and a capsule graph neural network provided in one embodiment of the present application;

[0093] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0095] In the description of the present application: unless otherwise specified, expressions such as “include”, “comprising”, “having”, etc. also mean “not limited to” (certain units, components, materials, steps, etc.).

[0096] In one embodiment, Figure 1-Figure 4 As shown, a multimodal classification method integrating a graph convolutional neural network and a capsule graph neural network is provided, and the method includes the following steps 1 to 5.

[0097] Step 1: Represent the patent text as a character sequence S = C1,...,C e , where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and incorporating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain a vector representation of each character.

[0098] Step 2: Use PCNN to segment each paragraph or sentence in the text to extract the local features of each paragraph or sentence. The local features reflect the detailed information inside the text. Then, use semantic graph convolution to model and integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation.

[0099] A segmented convolutional neural network (PCNN) with a gated attention mechanism is used to capture the relative position information between characters in a text sequence, understand the contextual relationship, and obtain the long-term dependency features of the sentence.

[0100] Step 3: Parse the sentence grammatical structure in the patent text, build a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information; during the grammatical graph convolution process, the nodes on the dependency tree will fuse information with other grammatically related nodes, so that the representation of each node not only contains the node's own information, but also fuses the information of other grammatically related nodes.

[0101] Among them, the pruning rule for pruning the grammatical dependency tree is: prune the original dependency tree, only retain the dependency relationship directly connected to the two entities, and construct a local grammatical dependency tree.

[0102] Step 4: Set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of the specific entity, use the interactive attention mechanism to discover the potential relationship between different elements in the text, extract specific entity features from the context information, concatenate the feature vectors after the entity features are fused, and use it as the final representation of the specific entity.

[0103] Step 5: Preliminarily extract patent image features through the multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, use the multi-dimensional vector neurons in the capsule network to represent more features of the target, capture the part-to-whole relationship in the image, and guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

[0104] This embodiment studies the appearance patent classification technology that integrates heterogeneous multimodal information, solves the problem of feature extraction and fusion of multimodal information such as text, images, and networks, and constructs a heterogeneous multimodal appearance design patent classification model to achieve a significant improvement in classification accuracy compared to pure text content. At the same time, based on the intellectual property hypergraph network, an appearance design patent AI standard data set is constructed to fill the gap in patent types in the traditional Locarno classification system, thereby breaking through the bottleneck of traditional appearance patent classification methods.

[0105] In one embodiment, step 1 converts each word or token in the patent text into a high-dimensional vector, retains the basic information of the text, and incorporates the semantic relationship between words. These vectors serve as input for subsequent processing steps. The sentence is encoded through the BERT pre-trained model to obtain a vector representation of each character. The processing process also includes the following steps 101-104.

[0106] Step 101: Encode the entire input patent text to obtain the corresponding vector representation. The text encoding formula is as follows:

[0107] H s =CharEmbedding(S)

[0108] in, represents the word embedding representation of sentence S after BERT encoding, d c Represents the dimensions of a character vector.

[0109] Step 102: Use the relative distance between the character and the entity to represent the position feature, reduce the distance between the character and the entity, and obtain richer information representation. The relative position expression formula of the head entity and each character is as follows:

[0110]

[0111] in, Indicates the distance between the head entity and each character, t indicates the relative distance from the current character to the head entity and the tail entity, b h Represents the head entity e head The index position of the first character, e h Represents the head entity e head The index position of the last character, similarly, the relative position of the last entity can be obtained.

[0112] Step 103: Then, the relative position information is encoded by word vectors to obtain a vector representation of the relative position information. The word vector encoding expression formula is as follows:

[0113] H head =PositionEmbedding head (P t head )

[0114] H tail =PositionEmbedding tail (P t tail )

[0115] in, Represents the relative position vector of the head entity and each character, Represents the tail entity and the position vector of each character, d p Represents the dimension of the relative position vector, P t head Indicates the distance between the head entity and each character, P t tail Indicates the distance between the tail entity and each character.

[0116] Step 104: Embed the word after BERT encoding into H s , the vector representation of the head and tail entities and the vector representation of the relative position of the head and tail entities are concatenated to obtain the final representation H of the sequence encoding module c , the splicing expression formula is as follows:

[0117] H c =[H s ;H entl ;H ent2 ;H head ;H tail ]

[0118] H entl The word representing the head entity embedding, H ent2 Word representing the tail entity embedding.

[0119] In one embodiment, step 2 is segmented through PCNN to extract local features for each paragraph or sentence. These local features reflect the detailed information inside the text, and then semantic graph convolution is used to integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation. The processing process also includes the following steps 201-203.

[0120] Step 201: The gated linear unit (GLU) and attention are integrated into a unified layer through the gated attention mechanism (GatedAttentionUnit, GAU), which maximizes the sharing of calculation parameters between the two, effectively improves the calculation efficiency of the attention score, and captures more advanced semantic representations. The integrated expression formula is as follows:

[0121]

[0122] Among them, H att represents the output of the gated attention mechanism GAU, w att represents the weight matrix, U represents the output feature of the gated linear unit, V represents the feature representation of the hidden state input to the fully connected layer, and A sem represents the attention weight matrix, and ⊙ represents the XOR operation.

[0123] Step 202: GAU output H att And initialize the attention weight A sem,0 Input into GCN, GCN updates the node representation of the next layer according to the hidden representation of the neighboring nodes. The update process is expressed as follows:

[0124]

[0125] Among them, the attention weight matrix Represented by the matrix After standardization, ReLU represents a nonlinear activation function, and the input It represents the first l-1 layers of GCN training generated by node i, represents the linear transformation weight, Represents the linear transformation bias term.

[0126] Step 203: In order to better fuse the global and local semantic features of the sentence, the features are adaptively fused through a gating mechanism. The gating mechanism consists of a gated Tanh unit (Gated Tanh Units, abbreviated as GTU) and a gated linear unit (Gated Linear Units, abbreviated as GLU). In the fusion process, effective features can be retained and useless information can be filtered out. The fusion expression formula is as follows:

[0127] g = sigmoid(w gl tanh(w g2 H sem +b g2 )+b gl )

[0128] R sem = g·H p

[0129] Among them, sigmoid and tanh represent nonlinear activation functions, w gl and w g2 represents the weight matrix, b gl and b g2 Represents the bias, and finally obtains the local global semantic representation R through the gating mechanism sem , H p represents the final output of the segmented convolutional network, and g represents the feature output of the adaptive fusion of the gating mechanism.

[0130] In one embodiment, step 3 builds a dependency tree by parsing the sentence grammatical structure in the patent text to clearly show the grammatical relationship between words. During the graph convolution process, the nodes (words) on the dependency tree will be fused with their grammatically related nodes, so that the representation of each node not only contains its own information, but also fuses the information of its grammatically related nodes. The processing process also includes the following steps 301-303.

[0131] Step 301: Use the dependency parser to obtain a grammatical dependency tree, and construct an adjacency matrix A of the sentence based on the grammatical dependency tree. syn , in order to construct the graph neural network structure in the relationship extraction task, element A syn,ij The calculation formula is as follows:

[0132]

[0133] Among them, the adjacency matrix A syn The main diagonal elements of are set to 1 to retain the information of their own nodes, constructing an undirected graph, c i ,c j Represents the text sequence of i and its neighboring nodes in the dependency tree.

[0134] Step 302: The adjacency matrix A constructed above syn Input into the graph convolutional network, and update the node representation of the next layer according to the hidden representation of the neighboring nodes. The hidden representation calculation formula is as follows:

[0135]

[0136] Among them, ReLU represents a nonlinear activation function, i represents the current node, j represents the neighboring node of node i, represents the degree of the i-th node in the dependency tree, represents the linear transformation weight, represents the linear transformation bias term, represents the hidden representation of the previous layer of GCN of node i, ∑A represents the total value of the adjacency matrix, and h l-1 Indicates the GCN training result of the previous layer of the node.

[0137] Step 303: Through the operation of L-layer grammatical graph convolution, each node continuously aggregates the information of its neighboring nodes and updates its own representation accordingly, and finally obtains the grammatical representation. The aggregation formula is as follows:

[0138]

[0139] represents the operation result of the L-layer grammar graph convolution, ... represents the many nodes of the L-layer, The hidden representation of the GCN representing node n.

[0140] In one embodiment, step 4 obtains a mask representation of a specific entity through a grammatical graph convolution module, and then uses an interactive attention mechanism to discover the potential relationship between different elements (such as paragraphs, sentences, and words) in the text to obtain the final representation. The processing process also includes the following steps 401-403.

[0141] Step 401: embedding the word after BERT encoding H s Representation of specific entities Interact and get the grammatical interaction feature vector R syn , the specific formula is as follows:

[0142]

[0143] β syn represents the specific entity features extracted from the context information using interactive attention, α i Indicates H s The result of interacting with a specific entity.

[0144] Step 402: The soft-max function is used to perform final relationship classification on the fused semantic features and the generated results of the grammatical interaction module. The classification formula is as follows:

[0145] r=[R sem ; R syn ]

[0146]

[0147] Among them, “;” represents the concatenation of vectors, w p and b p represents the trainable parameters, Represents the result of relation classification after grammatical interaction, R syn represents the grammatical interaction feature vector, R sem It indicates that local global semantics is obtained through the gating mechanism, and r indicates the result of the fusion of semantic features and grammatical interaction modules.

[0148] Step 403: Use cross entropy to minimize the target loss function. The calculation formula is as follows:

[0149]

[0150] Among them, s represents all training data, y i represents the true label, Represents the predicted label, λ represents the L2 regularization term coefficient, and θ represents the trainable parameter.

[0151] In one embodiment, step 5 preliminarily extracts patent image features through a multi-scale convolutional layer, and then uses the capsule network through its unique routing mechanism to capture the part-to-whole relationship in the image, guide the classification network to focus on features that contribute greatly to the classification results, weaken features that contribute little to the classification results, and improve the learning efficiency of the neural network. The processing process also includes the following steps 501-504.

[0152] Step 501: Perform convolution operation on the input image data through a multi-scale convolution kernel to generate a feature map. The feature map is nonlinearized by an activation function to generate output features and input to the next layer. The convolution operation formula is as follows:

[0153]

[0154] Among them, * indicates the dot product operation between the convolution kernel and the local area. represents the output feature of the jth neuron in the feature map i in the l+1th layer, and The weight and bias of the i-th convolution kernel in layer l are respectively l (j) represents the features of the jth neuron in the feature map of the lth layer.

[0155] Step 502: Use capsule vector neurons as the input and output of the model. The structure of the capsule network consists of a common convolution layer, a main capsule layer, and a digital capsule layer. First, use the common convolution layer to perform a convolution operation on the input data to extract primary features. The operation formula of the common convolution layer is as follows:

[0156]

[0157] in, represents the i-th kernel of the l-th layer, and w represents the size of the convolution kernel; r j represents the jth receptive field, Z l(i.j) represents the convolution result of the lth layer, x l() Represents the eigenvalue of the feature map in layer l.

[0158] Step 503: Use the main capsule layer to further extract the primary features extracted by the convolution layer into high-level features, and form a main capsule layer after encapsulation. The main capsule is mainly composed of the output of the convolution layer and the squeezing function. Its calculation formula is as follows:

[0159]

[0160] Among them, u l(i,j) represents the main capsule, represents the output of the convolutional layer, f s () represents the squeezing function.

[0161] Step 504: Use the linear transformation of the digital capsule layer to transfer between digital capsule layers. The calculation formula is as follows:

[0162]

[0163] Among them, u i represents the i-th input capsule; W ij represents the weight matrix; Represents the prediction vector.

[0164] In one embodiment, before step 502, the method further includes: using a rectified linear unit RELU activation function to allow the network to nonlinearly map the input signal to enhance the network's expression capability. The activation function expression formula is as follows:

[0165]

[0166] in, represents the activation value after the activation function, represents the output feature of the jth neuron in the feature map i in the l+1th layer, f a () represents the convolution function, and max{} represents the maximum value selection function.

[0167] In one embodiment, before step 503, the method further includes: using batch normalization to adjust the input distribution of data between layers within the network, and using the distribution of input data of each layer of the network to standardize to accelerate the convergence of the network. The batch normalization formula is as follows:

[0168]

[0169] Among them, y(i) represents the output of the neuron response, x (i) represents the eigenvalue of the input data point, γ (i) represents the scale factor, Represents x (i) The variance, E[x (i) ] represents x (i) The mathematical expectation of (i) Represents the translation factor.

[0170] In one embodiment, Figure 5 As shown, a multimodal classification device that integrates graph convolutional neural networks and capsule graph neural networks is provided, including the following program modules: BERT patent text sequence encoding module, PCNN segmented semantic graph convolution module, grammatical graph convolution construction dependency tree module, interactive attention and relationship classification module and capsule network image feature extraction module.

[0171] BERT patent text sequence encoding module, used to represent patent text as a character sequence S = C1,...,C e , where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and incorporating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain a vector representation of each character.

[0172] The PCNN segmented semantic graph convolution module is used to segment each paragraph or sentence in the text through PCNN to extract the local features of each paragraph or sentence. These local features reflect the detailed information inside the text. Subsequently, the semantic graph convolution is used to model and integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation. In order to further capture the relative position relationship between characters in the text sequence, a segmented convolutional neural network PCNN with a gated attention mechanism is used to understand the contextual relationship and obtain the long-term dependency features of the sentence.

[0173] The grammatical graph convolution constructs a dependency tree module, which is used to parse the grammatical structure of sentences in patent texts, construct a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information; during the grammatical graph convolution process, the nodes on the dependency tree will fuse information with their grammatically related nodes, so that the representation of each node not only contains its own information, but also fuses the information of its grammatically related nodes; among them, the pruning rules for pruning the grammatical dependency tree are: prune the original dependency tree, only retain the dependency relationship directly connected to the two entities, and construct a local grammatical dependency tree.

[0174] The interactive attention and relationship classification module is used to set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of a specific entity, and use the interactive attention mechanism to discover the potential relationship between different elements in the text and extract the features of the specific entity from the context information. The fused semantic features and the feature vector generated by the grammatical interaction module are concatenated and the fused feature vector is used as the final representation of the specific entity.

[0175] The capsule network image feature extraction module is used to preliminarily extract patent image features through a multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, the multi-dimensional vector neurons in the capsule network are used to represent more features of the target, capture the part-to-whole relationship in the image, guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

[0176] The specific implementation content of each module can be found in the above definition of the multimodal classification method integrating graph convolutional neural network and capsule graph neural network, which will not be repeated here.

[0177] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a communication interface, and a human-computer interaction interface (such as a combination of a display, a keyboard, and a mouse, and a touch screen, etc.) connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities, and the communication interface is used to communicate with an external terminal in a wired or wireless manner. The wireless method can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The computer device loads and runs a computer program to implement the above-mentioned multimodal classification method that integrates the graph convolutional neural network and the capsule graph neural network.

[0178] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In one embodiment, a computer program product is also provided, including a computer program / instruction, which implements the steps of the above-mentioned embodiment method when executed by a processor.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multimodal classification method integrating graph convolutional neural network and capsule graph neural network, characterized in that: include: Step 1: Represent the patent text as a character sequence S = C1,...,C e , where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and integrating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain the vector representation of each character; Step 2: Each paragraph or sentence in the text is segmented by PCNN to extract the local features of each paragraph or sentence, which reflects the detailed information inside the text. Then, semantic graph convolution is used to model and integrate the semantic relationships between different paragraphs or sentences to form a more global semantic representation. Use the segmented convolutional neural network (PCNN) with a gated attention mechanism to capture the relative position information between characters in the text sequence, understand the contextual relationship, and obtain the long-term dependency features of the sentence; Step 3: Analyze the sentence grammatical structure in the patent text, build a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information; In the process of grammatical graph convolution, the nodes on the dependency tree will fuse information with other nodes that are grammatically related to the node, so that the representation of each node not only contains the information of the node itself, but also fuses the information of other nodes that are grammatically related to the node; The pruning rules for pruning the grammatical dependency tree are as follows: pruning the original dependency tree, retaining only the dependency relationship directly connected to the two entities, and constructing a local grammatical dependency tree; Step 4: Set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of the specific entity, use the interactive attention mechanism to discover the potential relationship between different elements in the text, extract specific entity features from the context information, concatenate the feature vectors after the entity features are fused, and use it as the final representation of the specific entity; Step 5: Preliminarily extract patent image features through the multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, use the multi-dimensional vector neurons in the capsule network to represent more features of the target, capture the part-to-whole relationship in the image, and guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

2. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 1, characterized in that: The step 1 comprises: Step 101: Encode the entire input patent text to obtain the corresponding vector representation. The text encoding formula is as follows: H s =CharEmbedding(S) in, represents the word embedding representation of sentence S after BERT encoding, d c represents the dimension of a character vector; Step 102: Use the relative distance between the character and the entity to represent the position feature, reduce the distance between the character and the entity, and obtain richer information representation. The relative position expression formula of the head entity and each character is as follows: in, Indicates the distance between the head entity and each character, t indicates the relative distance from the current character to the head entity and the tail entity, b h Represents the head entity e head The index position of the first character, e h Represents the head entity e head The index position of the last character, and the relative position of the last entity can be obtained in the same way; Step 103: Then, the relative position information is encoded by word vectors to obtain a vector representation of the relative position information. The word vector encoding expression formula is as follows: H head =PositionEmbedding head (P t head ) H tail =PositionEmbedding tail (P t tail ) in, Represents the relative position vector of the head entity and each character, Represents the tail entity and the position vector of each character, d p Represents the dimension of the relative position vector, P t head Indicates the distance between the head entity and each character, P t tail Indicates the distance between the tail entity and each character; Step 104: Embed the word after BERT encoding into H s , the vector representation of the head and tail entities and the vector representation of the relative position of the head and tail entities are concatenated to obtain the final representation H of the sequence encoding module c , the splicing expression formula is as follows: H c =[H s ;H entl ;H ent2 ;H head ;H tail ] H entl The word representing the head entity embedding, H ent2 Word representing the tail entity embedding.

3. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 1, characterized in that: The step 2 comprises: Step 201: The gated linear unit GLU and attention are integrated into a unified layer through the gated attention mechanism GAU, which maximizes the sharing of calculation parameters between the two, effectively improves the calculation efficiency of the attention score, and captures more advanced semantic representations. The integrated expression formula is as follows: Among them, H att represents the output of the gated attention mechanism GAU, w att represents the weight matrix, U represents the output feature of the gated linear unit, V represents the feature representation of the hidden state input to the fully connected layer, and A sem represents the attention weight matrix, ⊙ represents the XOR operation; Step 202: GAU output H att And initialize the attention weight A sem,0 Input into GCN, GCN updates the node representation of the next layer according to the hidden representation of the neighboring nodes. The update process is expressed as follows: Among them, the attention weight matrix Represented by the matrix After standardization, ReLU represents a nonlinear activation function, and the input It represents the first l-1 layers of GCN training generated by node i, represents the linear transformation weight, represents the linear transformation bias term; Step 203: In order to better fuse the global and local semantic features of the sentence, the features are adaptively fused through a gating mechanism. The gating mechanism consists of a gated Tanh unit GTU and a gated linear unit GLU, which are used to retain valid features and filter out useless information during the fusion process. The fusion expression formula is as follows: g=sigmoid(w gl tanh(w g2 H sem +b g2 )+b gl ) R sem =g·H p Among them, sigmoid and tanh represent nonlinear activation functions, w gl and w g2 represents the weight matrix, b gl and b g2 Represents the bias, and finally obtains the local global semantic representation R through the gating mechanism sem , H p represents the final output of the segmented convolutional network, and g represents the feature output of the adaptive fusion of the gating mechanism.

4. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 1, characterized in that: The step 3 comprises: Step 301: Use the dependency parser to obtain a grammatical dependency tree, and construct an adjacency matrix A of the sentence based on the grammatical dependency tree. syn , in order to construct the graph neural network structure in the relationship extraction task, element A syn,ij The calculation formula is as follows: Among them, the adjacency matrix A syn The main diagonal elements of are set to 1 to retain the information of their own nodes, constructing an undirected graph, c i ,c j Represents the text sequence of i and its neighboring nodes in the dependency tree; Step 302: The adjacency matrix A constructed above syn Input into the graph convolutional network, and update the node representation of the next layer according to the hidden representation of the neighboring nodes. The hidden representation calculation formula is as follows: Among them, ReLU represents a nonlinear activation function, i represents the current node, j represents the neighboring node of node i, represents the degree of the i-th node in the dependency tree, represents the linear transformation weight, represents the linear transformation bias term, represents the hidden representation of the previous layer of GCN of node i, ΣA represents the total value of the adjacency matrix, and h l-1 Indicates the GCN training result of the previous layer of the node; Step 303: Through the operation of L-layer grammatical graph convolution, each node continuously aggregates the information of its neighboring nodes and updates its own representation accordingly, and finally obtains the grammatical representation. The aggregation formula is as follows: represents the operation result of the L-layer grammar graph convolution, ... represents the many nodes of the L-layer, The hidden representation of the GCN representing node n.

5. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 1, characterized in that: The step 4 comprises: Step 401: embedding the word after BERT encoding H s Representation of specific entities Interact and get the grammatical interaction feature vector R syn , the specific formula is as follows: β syn represents the specific entity features extracted from the context information using interactive attention, α i Indicates H s the results of interacting with a particular entity; Step 402: The soft-max function is used to perform final relationship classification on the fused semantic features and the results generated by the grammatical interaction module. The classification formula is as follows: r=[R sem ;R syn ] Among them, ";" represents the concatenation of vectors, w p and b p represents the trainable parameters, Represents the result of relation classification after grammatical interaction, R syn represents the grammatical interaction feature vector, R sem It indicates that the local global semantics is obtained through the gating mechanism, and r indicates the result of the fusion of semantic features and grammatical interaction modules; Step 403: Use cross entropy to minimize the target loss function. The calculation formula is as follows: Among them, s represents all training data, y i represents the true label, Represents the predicted label, λ represents the L2 regularization term coefficient, and θ represents the trainable parameter.

6. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 1, characterized in that: The step 5 comprises: Step 501: Perform convolution operation on the input image data through a multi-scale convolution kernel to generate a feature map. The feature map is nonlinearized by an activation function to generate output features and input to the next layer. The convolution operation formula is as follows: Among them, * indicates the dot product operation between the convolution kernel and the local area. represents the output feature of the jth neuron in the feature map i in the l+1th layer, and The weight and bias of the i-th convolution kernel in layer l are respectively l (j) represents the feature of the jth neuron in the feature map of the lth layer; Step 502: Use capsule vector neurons as the input and output of the model. The structure of the capsule network consists of a common convolution layer, a main capsule layer, and a digital capsule layer. First, use the common convolution layer to perform a convolution operation on the input data to extract primary features. The operation formula of the common convolution layer is as follows: in, represents the i-th kernel of the l-th layer, and w represents the size of the convolution kernel; r j represents the jth receptive field, Z l(i.j) represents the convolution result of the lth layer, x l() Represents the eigenvalue of the feature map in layer l; Step 503: Use the main capsule layer to further extract the primary features extracted by the convolution layer into high-level features, and form a main capsule layer after encapsulation. The main capsule is mainly composed of the output of the convolution layer and the squeezing function. Its calculation formula is as follows: Among them, u l(i,j) represents the main capsule, represents the output of the convolutional layer, f s () represents the squeezing function; Step 504: Use the linear transformation of the digital capsule layer to transfer between digital capsule layers. The calculation formula is as follows: Among them, u i represents the i-th input capsule; W ij represents the weight matrix; Represents the prediction vector.

7. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 6, characterized in that: Before step 502, the method further includes: using a rectified linear unit RELU activation function to allow the network to nonlinearly map the input signal to enhance the network's expression capability. The activation function expression formula is as follows: in, represents the activation value after the activation function, represents the output feature of the jth neuron in the feature map i in the l+1th layer, f a () represents the convolution function, and max{} represents the maximum value selection function.

8. The multimodal classification method integrating graph convolutional neural network and capsule graph neural network according to claim 6, characterized in that: Before step 503, the method further includes: using batch normalization to adjust the input distribution of data between layers within the network, and using the distribution of input data of each layer of the network to standardize to accelerate the convergence of the network. The batch normalization formula is as follows: Among them, y (i) represents the output of the neuron response, x (i) represents the eigenvalue of the input data point, γ (i) represents the scale factor, Represents x (i) The variance, E[x (i) ] represents x (i) The mathematical expectation, β (i) Represents the translation factor.

9. A multimodal classification device integrating graph convolutional neural network and capsule graph neural network, characterized in that: include: BERT patent text sequence encoding module, used to represent patent text as a character sequence S = C1,...,C e , where e represents the length of the character sequence. The sentence is encoded through the BERT pre-trained model, retaining the basic information of the text and integrating the semantic relationship between words. Each word or token in the patent text is converted into a high-dimensional vector to obtain the vector representation of each character; The PCNN segmented semantic graph convolution module is used to segment each paragraph or sentence in the text through PCNN to extract the local features of each paragraph or sentence. These local features reflect the detailed information inside the text. Subsequently, the semantic graph convolution is used to model and integrate the semantic relationship between different paragraphs or sentences to form a more global semantic representation. In order to further capture the relative position relationship between characters in the text sequence, the segmented convolutional neural network PCNN with a gated attention mechanism is used to understand the contextual relationship and obtain the long-term dependency features of the sentence. The grammatical graph convolution dependency tree module is used to parse the sentence grammatical structure in the patent text, build a grammatical dependency tree that can clearly show the relationship between words, prune the grammatical dependency tree, and use grammatical graph convolution to aggregate the pruned grammatical dependency number and context information and continuously update the representation to capture grammatical dependency information; In the process of grammatical graph convolution, the nodes on the dependency tree will fuse information with their grammatically related nodes, so that the representation of each node not only contains its own information, but also fuses the information of its grammatically related nodes; the pruning rule for pruning the grammatical dependency tree is as follows: prune the original dependency tree, retain only the dependency relationship directly connected to the two entities, and construct a local grammatical dependency tree; The interactive attention and relationship classification module is used to set the non-entity vector mask in the last layer output of the grammatical graph convolution module to 0, obtain the mask representation of a specific entity, use the interactive attention mechanism to discover the potential relationship between different elements in the text, extract specific entity features from the context information, and concatenate the feature vectors after the entity features are fused, and use it as the final representation of the specific entity; The capsule network image feature extraction module is used to preliminarily extract patent image features through a multi-scale convolutional layer, encapsulate the patent image features and input them into the capsule layer. Through the unique routing mechanism of the capsule network, the multi-dimensional vector neurons in the capsule network are used to represent more features of the target, capture the part-to-whole relationship in the image, guide the classification network to focus on the features that contribute most to the classification results, weaken the features that contribute little to the classification results, and improve the learning efficiency of the neural network.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images