Cultural relic pattern recognition method and device based on attention mechanism and graph convolutional network

Abstract

The invention relates to a cultural relic pattern recognition method and device based on an attention mechanism and a graph convolutional network. According to the method, the semantic representation of the image is effectively enhanced by introducing an attention mechanism, so that the model can more accurately capture the key features of the pattern. Meanwhile, in combination with the graph convolutional network, the nonlinear semantic association relationship between the local features of the pattern is further deeply mined, so that finer feature extraction and characterization are realized. In the optimization process of the model, the introduction of an attention mechanism not only improves the accuracy of recognition, but also enhances the distinguishing ability of the model to complex backgrounds and similar patterns. Besides, the spatial structure information in the image can be effectively processed by using the image convolutional network, so that the recognition of the cultural relic pattern has higher robustness and stability.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and device for identifying cultural relic patterns based on an attention mechanism and a graph convolutional network. Background Art

[0002] In the field of cultural heritage image classification, there are some works exploring the classification methods of historical paintings, architectural relics, fabrics, burial objects and other categories of cultural relics, and have achieved good results. For example, in the field of graphic art classification, convolutional neural networks are representative applications. By establishing a thematic database or classification data set, combined with comparative analysis or comparative training, the goal can be achieved better. In addition, graph convolutional networks, thanks to their flexibility in processing nonlinear data and their ability to aggregate local neighborhood information, can accurately capture the complex correlations between the internal structures of target data, and have a wide range of applications in remote sensing image classification, medical image analysis, motion capture and other fields.

[0003] However, due to factors such as the creation period, painting techniques, painting style, social atmosphere, and individual differences of the creators, patterns with the same semantics may have differences in visual form. In addition, patterns with different semantics may also have high similarities. This deviation between vision and semantics is manifested in the dataset as large intra-class differences and small inter-class differences, which poses a huge challenge to the correct classification of the model. Therefore, the existing classification research on ceramic artifacts generally has the problem of accuracy that needs to be improved. Summary of the invention

[0004] Based on this, it is necessary to provide a method and device for cultural relics pattern recognition based on attention mechanism and graph convolutional network to address the above technical problems.

[0005] A method for identifying cultural relics patterns based on semantics and graph convolutional networks, the method comprising:

[0006] The ceramic pattern image to be processed is input into a pre-built feature extraction module to obtain visual features; the feature extraction module is used to perform multi-scale feature privileging using a backbone network based on a residual network, and then a dense feature pyramid network is used to perform feature fusion based on a bottom-up architecture to obtain visual features.

[0007] The visual features are adaptively divided into semantic regions using a pre-built adaptive semantic parsing module; the adaptive semantic parsing module is used to count each feature pixel after convolution processing, define the channel value with the largest statistical value as the semantic region ID, and dynamically generate adaptive semantic regions.

[0008] The visual features after semantic region division are input into a pre-built dynamic graph relationship modeling module based on the attention mechanism to obtain updated visual features. The dynamic graph relationship modeling module based on the attention mechanism is used to use a graph convolutional network to fuse all regional features with different semantics, and use the attention weights of the region-level adjacency relationship to update the image features to obtain updated visual features.

[0009] According to the updated visual features and semantic region information, a classifier is used to perform classification to obtain the final classification result label of the pattern.

[0010] In one of the embodiments, the feature extraction module includes a backbone network and a dense feature pyramid network; the backbone network adopts a ResNet architecture.

[0011] The ceramic pattern image to be processed is input into the pre-built feature extraction module to obtain visual features, including:

[0012] The ceramic pattern image to be processed is input into the backbone network of the pre-built feature extraction module to obtain visual semantic features of four different scales.

[0013] The visual semantic features at four different scales are input into the dense feature pyramid network of the pre-built feature extraction module to obtain the visual features.

[0014] R4=Conv 1×1 (IR4)

[0015]

[0016] Among them, F is the visual feature, IR1, IR2, IR3 and IR4 are the four visual semantic features generated by the four residual blocks of ResNet, R4, R3 and R2 are the three intermediate features; DLSample is a dual-channel linear sampling, For the concatenation operation, Conv k×k It is a convolution operation using a k*k convolution kernel.

[0017] In one embodiment, the visual features are adaptively segmented into semantic regions using a pre-built adaptive semantic parsing module, including:

[0018] The visual features are input into the adaptive semantic parsing module, and the feature vector is obtained through a convolutional layer with a kernel size of 5*5.

[0019] When the number of channels of the feature vector is equal to the preset value, each feature pixel is counted, and the channel value with the largest statistical value is defined as the semantic region ID.

[0020] According to the semantic region ID and visual features of each feature pixel, an adaptive semantic region is dynamically generated.

[0021] In one embodiment, the visual features after semantic region division are input into a pre-built dynamic graph relationship modeling module based on an attention mechanism to obtain updated visual features, including:

[0022] Each semantic region is regarded as a graph node in the graph network.

[0023] Construct an adjacency matrix from all graph nodes.

[0024] Dynamic adjustment of the adjacency matrix by introducing the attention mechanism.

[0025] Each feature pixel of the visual feature is convolved with the graph node update strategy to obtain an updated feature, and the updated feature is passed through the convolution layer to obtain an updated visual feature; the graph node update strategy is:

[0026]

[0027] Among them, f i and f i * are the visual features and the updated features, A' j is the jth row of the dynamically adjusted adjacency matrix, x i is the i-th graph node.

[0028] In one embodiment, the dynamic adjustment of the adjacency matrix by introducing an attention mechanism includes:

[0029] Perform linear transformation on the graph nodes; the expression of linear transformation is:

[0030] W h =XW

[0031] Among them, X is the feature matrix of the graph nodes, W is the trainable weight matrix, and W h is the result of linear transformation of graph nodes.

[0032] The input of the attention mechanism is defined as:

[0033] a ij =a([W h (i)||W h (j)])

[0034] Among them, a ij is the input of the attention mechanism, || represents the feature connection, and a is a trainable weight matrix.

[0035] The attention coefficient is calculated based on the input of the attention mechanism:

[0036]

[0037] Among them, e ij is the attention coefficient of graph node i and graph node j, N i is the set of neighbor nodes of graph node i.

[0038] Activate the attention beam through the softmax function, and get the normalized attention coefficient:

[0039]

[0040] Among them, α ij is the normalized attention coefficient and is the set of neighbor nodes of graph node i.

[0041] Multiply the normalized attention coefficient with the adjacency matrix to obtain the dynamically adjusted adjacency matrix.

[0042] In one embodiment, an adjacency matrix is ​​constructed based on all graph nodes; wherein the element in the i-th row and j-th column of the adjacency matrix is:

[0043]

[0044] Among them, A ij is the element in the i-th row and j-th column of the adjacency matrix, γ is a hyperparameter, and x i and x j are the features of two graph nodes, and D(·) is the Mahalanobis distance metric.

[0045] In one embodiment, a classifier is used to classify the pattern according to the updated visual features and semantic region information to obtain a final classification result label of the pattern, including:

[0046] The updated visual features and semantic region information are input into the classifier, and the updated visual features and semantic region information are weightedly summed to generate a comprehensive feature map. The comprehensive feature map is then globally averaged pooled, and the spatial dimension is compressed into a feature vector for each image to obtain the final classification result label of the pattern.

[0047] A device for identifying cultural relics patterns based on semantics and graph convolutional networks, the device comprising:

[0048] The visual feature extraction module is used to extract visual features from ceramic pattern images using a feature extraction module based on a deep residual network.

[0049] The adaptive semantic region generation module is used to capture the potential semantics in the pattern image by using an adaptive semantic parsing module according to visual features, and dynamically generate adaptive semantic regions.

[0050] The visual feature adaptive update module is used to use a graph convolutional network to fuse all regional features with different semantics, and use the attention weights of the region-level adjacency relationship to update the image features to obtain updated visual features.

[0051] The pattern recognition module is used to generate the final semantic label of the image using the scene decision module according to the updated visual features and semantic region information.

[0052] The above-mentioned method and device for cultural relic pattern recognition based on semantics and graph convolutional networks effectively enhances the semantic representation of the image by introducing an attention mechanism, so that the model can more accurately capture the key features of the pattern. At the same time, combined with the graph convolutional network, the nonlinear semantic association relationship between the local features of the pattern is further explored to achieve more refined feature extraction and characterization. In the process of model optimization, the introduction of the attention mechanism not only improves the recognition accuracy, but also enhances the model's ability to distinguish complex backgrounds and similar patterns. In addition, the use of graph convolutional networks can effectively process the spatial structure information in the image, making the recognition of cultural relic patterns more robust and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of a process of a method for identifying cultural relics patterns based on semantics and graph convolutional networks in one embodiment;

[0054] Figure 2 A framework diagram of a cultural relic pattern recognition model based on semantics and graph convolutional networks in one embodiment;

[0055] Figure 3 is a structural diagram of a feature extraction module in one embodiment;

[0056] Figure 4 It is a structural block diagram of a cultural relic pattern recognition device based on semantic and graph convolutional networks in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, Figure 1 As shown, a method for identifying cultural relics patterns based on semantics and graph convolutional networks is provided, and the method includes the following steps:

[0059] Step 100: Input the ceramic pattern image to be processed into a pre-built feature extraction module to obtain visual features; the feature extraction module is used to use a backbone network based on a residual network to perform multi-scale feature privilege, and then use a dense feature pyramid network to perform feature fusion based on a bottom-up architecture to obtain visual features.

[0060] Specifically, the diversity and semantic complexity of ceramic pattern images pose challenges to feature extraction. In order to capture its significant visual features, this application uses the ResNet architecture as the backbone network of the feature extraction module to extract multi-scale feature vectors. The dense feature pyramid network is used to fuse multi-scale features through a bottom-up architecture, which can enhance the semantic representation of the image.

[0061] Step 102: Adopting a pre-built adaptive semantic parsing module to perform adaptive semantic region division on the visual features; the adaptive semantic parsing module is used to count each feature pixel after convolution processing, define the channel value with the largest statistical value as the semantic region ID, and dynamically generate adaptive semantic regions.

[0062] Specifically, for pattern images, predefined categories outline their macro properties, while fine-grained local semantic regions can provide more relevant information for classifying patterns. In-depth analysis of the nonlinear relationship between these local semantic regions helps to improve the accuracy of classification. Therefore, this application constructs an adaptive semantic parsing module to capture the potential semantics in pattern images and dynamically generate adaptive semantic regions. According to feedback from workshops in cooperation with cultural relics experts, the semantic regions of a pattern image usually do not exceed 10, so the upper limit of the dynamic semantic region is set to 10.

[0063] Step 104: Input the visual features after semantic region division into a pre-built dynamic graph relationship modeling module based on the attention mechanism to obtain updated visual features; the dynamic graph relationship modeling module based on the attention mechanism is used to use a graph convolutional network to fuse all regional features with different semantics, and use the attention weights of the region-level adjacency relationship to update the image features to obtain updated visual features.

[0064] Specifically, each semantic region is taken as a node of the graph network, thereby obtaining a set of graph nodes Xg = {x1, x2, x3, ..., x Nr},in C is the number of channels of the visual feature F. In order to deeply characterize the nonlinear relationship between graph nodes, an adjacency matrix A is constructed; by introducing the attention mechanism to achieve dynamic adjustment of the adjacency matrix A, it can better capture the relationship between nodes.

[0065] The dynamic graph relationship modeling module based on the attention mechanism can use the graph convolutional network to fuse all regional features with different semantics, thereby capturing the global and local information in the porcelain pattern. The introduction of the attention mechanism significantly improves the model's ability to capture important features, allowing the model to pay more attention to the areas that have a greater impact on the prediction results. This method improves the accuracy of feature representation and enhances the robustness of the model under complex backgrounds. Compared with the traditional scheme, which has problems in feature selection (inability to accurately distinguish important features from noise) and information fusion quality (association fusion analysis of multiple semantic features), the attention mechanism can capture feature point information more efficiently by learning point features and regional adjacency matrices without updating regional feature representations, combined with correlation weight calculations between different features, thereby achieving fast and accurate predictions. In addition, this method does not update the regional feature representation Fg, but uses the regional adjacency matrix to learn point features F. The dynamic graph relationship modeling module based on the attention mechanism can approximately learn feature points at a lower computational cost, thereby quickly and accurately predicting the semantic labels of porcelain visual patterns.

[0066] Step 106: Classify using a classifier based on the updated visual features and semantic region information to obtain a final classification result label of the pattern.

[0067] Specifically, the updated visual features contain discriminative information, so that ceramic patterns can be accurately distinguished.

[0068] This application proposes a cultural relic pattern recognition model based on semantics and graph convolutional networks. By exploring multiple local semantics hidden in ceramic patterns and mining the short-range / long-range semantic associations between them, the accurate decision of pattern category is finally obtained. The cultural relic pattern recognition model based on semantics and graph convolutional networks consists of a feature extraction network, an adaptive semantic parsing module, a dynamic graph relationship modeling module based on the attention mechanism, and a classifier. The overall framework of the model is as follows: Figure 2 As shown in the figure, the feature extraction module based on the deep residual network is used to extract representative visual features from the image; the adaptive semantic parsing module is used to capture the potential semantic information of the image by adaptively generating semantic regions; the dynamic graph relationship modeling module based on the attention mechanism is used to update the image features using the attention weights of the region-level adjacency relationship to capture pixel-level knowledge, regions and information and the contextual associations between them; the classification head is used to determine the final semantic label of the image based on the updated visual features and the corresponding semantic regions.

[0069] In the above-mentioned cultural relic pattern recognition method based on semantics and graph convolutional networks, this method effectively enhances the semantic representation of the image by introducing the attention mechanism, so that the model can more accurately capture the key features of the pattern. At the same time, combined with the graph convolutional network, the nonlinear semantic association relationship between the local features of the pattern is further explored, thereby achieving more refined feature extraction and representation. In the process of model optimization, the introduction of the attention mechanism not only improves the recognition accuracy, but also enhances the model's ability to distinguish complex backgrounds and similar patterns. In addition, the use of graph convolutional networks can effectively process the spatial structure information in the image, making the recognition of cultural relic patterns more robust and stable.

[0070] In one embodiment, if Figure 3 As shown, the feature extraction module includes a backbone network and a dense feature pyramid network; the backbone network adopts the ResNet architecture; step 100 includes: inputting the ceramic pattern image to be processed into the backbone network of the pre-built feature extraction module to obtain visual semantic features of four different scales; inputting the visual semantic features of four different scales into the dense feature pyramid network of the pre-built feature extraction module to obtain visual features.

[0071] R4=Conv 1×1 (IR4) (1)

[0072]

[0073] Among them, F is the visual feature, IR1, IR2, IR3 and IR4 are the four visual semantic features generated by the four residual blocks of ResNet, R4, R3 and R2 are the three intermediate features; DLSample is a dual-channel linear sampling, For the concatenation operation, Conv k×k It is a convolution operation using a k×k convolution kernel.

[0074] In one embodiment, step 102 includes: inputting the visual features into an adaptive semantic parsing module, and obtaining a feature vector through a convolution layer with a kernel size of 5×5; when the number of channels of the feature vector is equal to a preset value, each feature pixel is counted, and for each pixel feature, the channel value with the largest statistical value is defined as the semantic region ID; and dynamically generating an adaptive semantic region based on the semantic region ID and visual features of each feature pixel.

[0075] Specifically, for the visual feature F output by the feature extraction module, it first passes through a convolution layer with a kernel size of 5×5 to obtain the output vector F'. When the number of channels of F' is equal to the preset value Nr (i.e., the maximum is 10), for each feature pixel of F', the channel direction with the largest recorded value (i.e., the maximum recorded value on the channel corresponding to each feature pixel) is set as the semantic region ID. Finally, the semantic region of F can be adaptively mapped according to the value of each feature pixel.

[0076] In one embodiment, step 104 includes: treating each semantic region as a graph node of a graph network; constructing an adjacency matrix according to all graph nodes; dynamically adjusting the adjacency matrix by introducing an attention mechanism; performing graph convolution on each feature pixel of the visual feature and the graph node update strategy to obtain an updated feature, and passing the updated feature through a convolution layer to obtain an updated visual feature; wherein the graph node update strategy is:

[0077]

[0078] Among them, f i and f i * are the visual features and the updated features, A' j is the jth row of the dynamically adjusted adjacency matrix, x i is the i-th graph node.

[0079] In one embodiment, the dynamic adjustment of the adjacency matrix by introducing an attention mechanism includes: performing a linear transformation on the graph nodes; wherein the expression of the linear transformation is:

[0080] W h =XW (6)

[0081] Among them, X is the feature matrix of the graph nodes, W is the trainable weight matrix, and W h is the result of linear transformation of graph nodes.

[0082] The input of the attention mechanism is defined as:

[0083] a ij =a([W h (i)||W h (j)]) (7)

[0084] Among them, a ij is the input of the attention mechanism, || represents the feature connection, and a is the trainable weight matrix.

[0085] The attention coefficient is calculated based on the input of the attention mechanism:

[0086]

[0087] Among them, e ij is the attention coefficient of graph node i and graph node j, N i is the set of neighbor nodes of graph node i.

[0088] Activate the attention beam through the softmax function, and get the normalized attention coefficient:

[0089]

[0090] Among them, α ij is the normalized attention coefficient and is the set of neighbor nodes of graph node i.

[0091] Multiply the normalized attention coefficient with the adjacency matrix to obtain the dynamically adjusted adjacency matrix.

[0092] In one embodiment, an adjacency matrix is ​​constructed based on all graph nodes; wherein the element in the i-th row and j-th column of the adjacency matrix is:

[0093]

[0094] Among them, Ai j is the element in the i-th row and j-th column of the adjacency matrix, γ is a hyperparameter, and x i and x j are the features of two graph nodes, and D(·) is the Mahalanobis distance metric.

[0095] Specifically, assuming that the visual feature is divided into N semantic regions, where F = {f1, f2, f3, ..., f n}, And n=W×H.

[0096] In this process, the feature ID of the center of each semantic region is selected to represent the region, thereby obtaining a set of graph nodes Xg = {x1, x2, x3, ..., x Nr},in C is the number of channels of the visual feature F. In order to deeply characterize the nonlinear relationship between graph nodes, an adjacency matrix A is constructed. The expression of the element in the i-th row and j-th column of the adjacency matrix is ​​shown in formula (10).

[0097] In an embodiment, the Mahalanobis distance metric is used to measure the similarity between nodes, which is defined as:

[0098]

[0099] Among them, M is a trainable weight matrix, which can be decomposed into and wd.

[0100] Then, the attention mechanism is introduced to dynamically adjust the adjacency matrix A so that it can better capture the relationship between nodes. Specifically, the node features are first linearly transformed using the method shown in formula (6).

[0101] The input of the attention mechanism is then defined as shown in formula (7), and the attention coefficient is calculated using formula (8), and a softmax function is introduced to obtain the normalized attention coefficient as shown in formula (9).

[0102] Finally, the attention coefficient is applied to the adjacency matrix A:

[0103] A′ ij =α ij A ij (12)

[0104] Among them, A' ij is the adjusted adjacency matrix incorporating the attention mechanism, allowing for better capture of the relationships between nodes.

[0105] Next, a dynamic graph node update strategy is designed. Nr}, the region-level adjacency relationship determined by the graph convolution is used to transform the visual feature F into the updated visual feature F*. Specifically, for a feature point fi of F, if it belongs to the jth semantic region, its update process is shown in formula (5).

[0106] After all feature points are updated through this update strategy, a convolution layer with a kernel size of 3×3 is applied to the updated features to obtain the updated visual feature F*.

[0107] In this example, the visual feature F contains n (n = W × H) feature pixels and is analyzed into 10 semantics, where n = W × H. Therefore, the feature F can be region-averaged into 10 graph nodes x1 to x 10 , and flatten these n feature pixels into f1 to f n .

[0108] The edges of the graph (i.e., the region adjacency matrix) can be generated by measuring the similarity relationships between different nodes.

[0109] Then, each feature pixel f is updated to f* by performing graph convolution with the proposed graph node update strategy. Finally, after passing through the convolution layer, all updated feature pixels f* constitute the updated visual feature F*.

[0110] In one embodiment, step 106 includes: inputting the updated visual features and semantic region information into the classifier, performing weighted summation on the updated visual features and semantic region information to generate a comprehensive feature map, and then globally average pooling the comprehensive feature map, and compressing the spatial dimension into a feature vector for each image to obtain the final classification result label of the pattern.

[0111] Specifically, the updated visual feature F* contains discriminative information, so that the ceramic patterns can be accurately distinguished. In the classification head, the updated feature map and the semantic region are first weighted summed to generate a comprehensive feature map, which is expressed as:

[0112]

[0113] Among them, index_ex i represents the index mask of the i-th semantic region, input_ i Represents the input feature map, adj_means i represents the adjacency matrix mean feature.

[0114] Then, the comprehensive feature map is globally averaged and pooled, and the spatial dimension is compressed into the feature vector of each image, and finally the classification result label Label is obtained, which is expressed as:

[0115] Label=Mean(features,dim=(2,3)) (14)

[0116] Among them, dim = (2,3) means averaging in height and width dimensions, and Label is the category of the final semantic label. Through the above steps, the classification head can effectively combine the updated features and semantic region information to generate the final semantic label of the image.

[0117] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0118] Through the method of this application, the recognition decision of cultural relic patterns has been significantly optimized, and finally a high-precision semantic-based cultural relic pattern recognition technology has been realized. This application method not only performs well in experiments, but also demonstrates its strong adaptability and practicality in practical applications. The use of this method can greatly improve the semantic recognition accuracy of cultural relic patterns, providing important technical support and theoretical basis for the digital protection and research of cultural relics. At the same time, this application method can also be expanded to other fields, such as image classification, target detection, etc., and has broad application prospects.

[0119] In one embodiment, Figure 4 As shown, a cultural relic pattern recognition device based on semantic and graph convolutional networks is provided, comprising: a visual feature extraction module, an adaptive semantic region generation module, a visual feature adaptive update module and a pattern recognition module, wherein:

[0120] The visual feature extraction module is used to extract visual features from ceramic pattern images using a feature extraction module based on a deep residual network.

[0121] The adaptive semantic region generation module is used to capture the potential semantics in the pattern image by using an adaptive semantic parsing module according to visual features, and dynamically generate adaptive semantic regions.

[0122] The visual feature adaptive update module is used to use a graph convolutional network to fuse all regional features with different semantics, and use the attention weights of the region-level adjacency relationship to update the image features to obtain updated visual features.

[0123] The pattern recognition module is used to generate the final semantic label of the image using the scene decision module according to the updated visual features and semantic region information.

[0124] In one embodiment, the feature extraction module includes a backbone network and a dense feature pyramid network; the backbone network adopts the ResNet architecture; the visual feature extraction module is also used to input the ceramic pattern image to be processed into the backbone network of the pre-built feature extraction module to obtain visual semantic features of four different scales; the visual semantic features of four different scales are input into the dense feature pyramid network of the pre-built feature extraction module to obtain visual features; the visual feature calculation process is shown in formulas (1) to (4).

[0125] In one of the embodiments, the adaptive semantic region generation module is also used to input visual features into the adaptive semantic parsing module, and obtain a feature vector through a convolution layer with a kernel size of 5×5; when the number of channels of the feature vector is equal to a preset value, each feature pixel is counted, and the channel value with the largest statistical value is defined as the semantic region ID; based on the semantic region ID and visual features of each feature pixel, an adaptive semantic region is dynamically generated.

[0126] In one of the embodiments, the visual feature adaptive update module is further used to treat each semantic region as a graph node of a graph network; construct an adjacency matrix based on all graph nodes; dynamically adjust the adjacency matrix by introducing an attention mechanism; perform graph convolution on each feature pixel of the visual feature and the graph node update strategy to obtain an updated feature, and obtain an updated visual feature after passing the updated feature through a convolution layer; wherein the graph node update strategy is shown in formula (5).

[0127] In one of the embodiments, the visual feature adaptive update module is also used to perform linear transformation on the graph nodes; wherein the expression of the linear transformation is shown in formula (6).

[0128] The input of the attention mechanism is defined as shown in formula (7).

[0129] According to the input of the attention mechanism, the attention coefficient is calculated using formula (8).

[0130] The attention beam is activated by the softmax function to obtain the normalized attention coefficient as shown in formula (9).

[0131] Multiply the normalized attention coefficient with the adjacency matrix to obtain the dynamically adjusted adjacency matrix.

[0132] In one of the embodiments, the visual feature adaptive update module is further used to construct an adjacency matrix based on all graph nodes; wherein the elements in the i-th row and j-th column of the adjacency matrix are as shown in formula (10).

[0133] In one of the embodiments, the pattern recognition module is also used to input the updated visual features and semantic area information into the classifier, perform weighted summation on the updated visual features and semantic area information to generate a comprehensive feature map, and then perform global average pooling on the comprehensive feature map, and compress the spatial dimension into a feature vector for each image to obtain the final classification result label of the pattern.

[0134] For the specific definition of the cultural relic pattern recognition device based on semantic and graph convolutional networks, please refer to the definition of the cultural relic pattern recognition method based on semantic and graph convolutional networks mentioned above, which will not be repeated here. Each module in the above-mentioned cultural relic pattern recognition device based on semantic and graph convolutional networks can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying cultural relics patterns based on semantics and graph convolutional networks, characterized in that: The method comprises: The ceramic pattern image to be processed is input into a pre-built feature extraction module to obtain visual features; the feature extraction module is used to perform multi-scale feature extraction using a backbone network based on a residual network, and then perform feature fusion based on a bottom-up architecture using a dense feature pyramid network to obtain visual features; The visual features are divided into adaptive semantic regions using a pre-built adaptive semantic parsing module; the adaptive semantic parsing module is used to count each feature pixel after convolution processing, define the channel value with the largest statistical value as the semantic region ID, and dynamically generate adaptive semantic regions; Inputting the visual features after semantic region division into a pre-built dynamic graph relationship modeling module based on an attention mechanism to obtain updated visual features; the dynamic graph relationship modeling module based on the attention mechanism is used to fuse all regional features with different semantics using a graph convolutional network, and update the image features using the attention weights of the region-level adjacency relationship to obtain updated visual features; Classification is performed using a classifier according to the updated visual features and semantic region information to obtain a final classification result label of the pattern.

2. The method according to claim 1, characterized in that The feature extraction module includes a backbone network and a dense feature pyramid network; the backbone network adopts a ResNet architecture; The ceramic pattern image to be processed is input into the pre-built feature extraction module to obtain visual features, including: The ceramic pattern image to be processed is input into the backbone network of the pre-built feature extraction module to obtain visual semantic features of four different scales; The visual semantic features of four different scales are input into the dense feature pyramid network of the pre-built feature extraction module to obtain the visual features; R4=Conv 1×1 (IR4) Among them, F is the visual feature, IR1, IR2, IR3 and IR4 are the four visual semantic features generated by the four residual blocks of ResNet, R4, R3 and R2 are the three intermediate features; DLSample is a two-channel linear sampling operation, For the concatenation operation, Conv k×k It is a convolution operation using a k×k convolution kernel.

3. The method according to claim 1, characterized in that The visual features are subjected to adaptive semantic region segmentation using a pre-built adaptive semantic parsing module, including: The visual features are input into an adaptive semantic parsing module and a feature vector is obtained through a convolutional layer with a kernel size of 5×5; When the number of channels of the feature vector is equal to a preset value, each feature pixel is counted, and the channel value with the largest statistical value is defined as the semantic region ID; An adaptive semantic region is dynamically generated according to the semantic region ID of each feature pixel and the visual feature.

4. The method according to claim 1, characterized in that: The visual features after semantic region division are input into the pre-built dynamic graph relationship modeling module based on the attention mechanism to obtain updated visual features, including: Treat each semantic region as a graph node in the graph network; Constructing an adjacency matrix based on all of the graph nodes; Dynamic adjustment of the adjacency matrix by introducing the attention mechanism; Each feature pixel of the visual feature is convolved with the graph node update strategy to obtain an updated feature, and the updated feature is passed through the convolution layer to obtain an updated visual feature; the graph node update strategy is: Among them, f i and f i * are the visual features and the updated features, A' j is the jth row of the dynamically adjusted adjacency matrix, x i is the i-th graph node.

5. The method according to claim 4, characterized in that The adjacency matrix is ​​dynamically adjusted by introducing the attention mechanism, including: Perform linear transformation on the graph nodes; the expression of linear transformation is: W h =XW Among them, X is the feature matrix of the graph nodes, W is the trainable weight matrix, and W h is the result of linear transformation of graph nodes; The input of the attention mechanism is defined as: a ij =a([W h (i)||W h (j)]) Among them, a ij is the input of the attention mechanism, || represents the feature connection, and a is a trainable weight matrix; The attention coefficient is calculated based on the input of the attention mechanism: Among them, e ij is the attention coefficient of graph node i and graph node j, N i is the set of neighbor nodes of graph node i; The attention beam is activated by the softmax function, and the normalized attention coefficient is obtained as follows: Among them, α ij is the normalized attention coefficient of the neighbor node set of graph node i; The normalized attention coefficient is multiplied by the adjacency matrix to obtain a dynamically adjusted adjacency matrix.

6. The method according to claim 4, characterized in that An adjacency matrix is ​​constructed based on all the graph nodes; wherein the element in the i-th row and j-th column of the adjacency matrix is: Among them, A ij is the element in the i-th row and j-th column of the adjacency matrix, γ is a hyperparameter, and x i and x j are the features of two graph nodes, and D(·) is the Mahalanobis distance metric.

7. The method according to claim 1, characterized in that According to the updated visual features and semantic region information, a classifier is used to classify the pattern to obtain a final classification result label of the pattern, including: The updated visual features and semantic region information are input into the classifier, and the updated visual features and semantic region information are weightedly summed to generate a comprehensive feature map. The comprehensive feature map is then globally average pooled, and the spatial dimension is compressed into a feature vector for each image to obtain the final classification result label of the pattern.

8. A device for identifying cultural relics patterns based on semantics and graph convolutional networks, characterized in that: The square device comprises: A visual feature extraction module, used to extract visual features from ceramic pattern images using a feature extraction module based on a deep residual network; An adaptive semantic region generation module, used to capture the potential semantics in the pattern image using an adaptive semantic parsing module according to the visual features, and dynamically generate an adaptive semantic region; The visual feature adaptive update module is used to fuse all regional features with different semantics using a graph convolutional network, and update the image features using the attention weights of the region-level adjacency relationship to obtain updated visual features; The pattern recognition module is used to generate a final semantic label of the image using a scene decision module according to the updated visual features and semantic region information.

Images