Product knowledge graph construction method based on image driving

Abstract

The invention discloses a product knowledge graph construction method based on image driving. The method comprises the following steps: constructing a product knowledge graph ontology; according to the ontology class and the ontology relationship of the batik product, constructing a batik product ontology framework, and combining the preprocessed product image data to construct an image-driven batik product knowledge graph, including entity acquisition and relationship acquisition; an entity set E is obtained and represented as E = EIMG U ETEXT, EIMG represents image entities including products, patterns and representative pattern entities, and ETEXT represents text type entities including themes, layouts and pattern type entities; visualizing the product knowledge graph; aiming at the automatic updating of the composition relationship, the pattern is automatically extracted from the batik product image, and the method is realized based on a target detection technology. The method has the characteristic of constructing the knowledge graph by combining with the image data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for constructing a product knowledge graph based on image driving. Background Art

[0002] As an intangible cultural heritage, batik still faces the challenges of inheritance difficulties and the gradual fading of excellent traditional culture. In this context, it is particularly important to use digital technology to record, inherit and spread batik culture. In the protection and research projects of intangible cultural heritage, digital technology has become an important supporting tool. Among these technologies, the construction of ontology and knowledge graph has received extensive attention from researchers. Ontology, as a formal expression method that clearly defines various concepts and their relationships, can be used to create and simulate the knowledge structure of a certain field. Knowledge graphs organize and express knowledge in a graphical way, and effectively integrate scattered information to form an organic and semantically rich knowledge network. Among them, ontology can provide a shared knowledge foundation for knowledge graphs, ensure the consistency and accuracy of information, and thus support the effective work of knowledge graphs.

[0003] In the prior art, with the continuous development of deep learning, the construction of knowledge graphs has expanded from a single text modality to a multimodal form, including multiple data types such as text, images, and videos, so as to more comprehensively capture and express the content of intangible cultural heritage. However, in the face of a colorful cultural treasure house, different intangible cultural heritages are of various types and huge in number. For different cultural heritages, the same knowledge graph construction strategy may not be applicable. Existing research on knowledge graphs mainly relies on huge and rich text data or data sets that have been processed and improved by predecessors, and corresponding improvements are made based on the ontology of ICH. In short, when facing intangible cultural heritage, text-based knowledge graphs mainly conduct basic text introductions and retrievals, and it is difficult to deeply explore the rich connotations of batik product art. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings and propose an image-driven product knowledge graph construction method that combines text and image data.

[0005] The present invention provides a method for constructing a product knowledge graph based on image-driven, wherein: the method comprises the following steps:

[0006] Step 1: Product image data acquisition and preprocessing: Obtain batik product images and preprocess them, and use image annotation tools to annotate them;

[0007] Step 2: Product knowledge graph ontology construction, the specific steps include:

[0008] Step 2.1: Class determination: Based on the analysis of the characteristics of batik product images, determine the ontology class of batik products;

[0009] Step 2.2: Determination of relationships: Based on the analysis of the characteristics of batik product images, determine the ontological relationships of batik products, including composition relationships, expression relationships, layout relationships, evolution relationships, belonging relationships, and having relationships;

[0010] Step 3: Product knowledge graph construction: According to the ontology classes and ontology relationships of the batik products, a batik product ontology framework is constructed. Combined with the pre-processed product image data, an image-driven batik product knowledge graph is constructed, including entity acquisition and relationship acquisition. The specific steps include:

[0011] Step 3.1: Entity acquisition: Get the entity set E, expressed as E = E IMG UE TEXT , where E IMG Represents image entities, including products, patterns, and representative pattern entities. TEXT Represents text-type entities, including subject, layout, and pattern type entities;

[0012] The products, patterns, and representative pattern entities are represented by images or corresponding image addresses in the constructed product knowledge graph;

[0013] Acquisition of the pattern and product entity: directly using the batik product image after shooting and preprocessing as the product entity, and extracting a single pattern from the product according to the image annotation result to obtain the pattern entity;

[0014] The representative pattern acquisition is performed by using image clustering based on the nearest neighbor propagation clustering algorithm. First, the batik product image set After binarization, feature extraction is performed through the feature extraction network, where N is the number of images, C, H, and W represent the number of channels, height, and width of the image, respectively, to obtain the feature representation set of the batik image. Where h represents its feature dimension; then, cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix

[0015] The cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the calculation formula is:

[0016]

[0017] The feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix

[0018] Step 3.2: Relationship acquisition: The relationship set is set to R = {CR, LR, HR, ER, PR, BR}, where CR represents a composition relationship, LR represents a layout relationship, HR represents a has relationship, ER represents an evolution relationship, PR represents a performance relationship, and BR represents a belongs relationship. CR is used to describe the connection between products and patterns. When acquiring product and pattern nodes, the corresponding CR is also obtained; the acquisition of the PR requires the determination of the theme of a work, which is determined according to the established rules; the layout relationship LR is determined and acquired using the description method of relevant literature; the belongs relationship BR and the has relationship HR are obtained through data search; the evolution relationship ER is obtained by using the representative pattern obtained by the clustering algorithm, and the pattern forming the cluster has an evolution relationship with it;

[0019] Step 4: Get results and organize them: collect and organize all the entities and relationships obtained, and use EXCEL as a storage tool;

[0020] Step 5: Visualization of product knowledge graph: Import the sorted EXCEL spreadsheet files in batches through the interactive code between Python and the graph database Neo4j, and visualize the product knowledge graph through Neo4j;

[0021] Step 6: Automatic update of product knowledge graph: For the automatic update of composition relationships, patterns are automatically extracted from batik product images. This is achieved based on target detection technology. The specific steps include:

[0022] Step 6.1: Process the batik product image dataset to obtain a batik target detection dataset, and divide it into training set, validation set and test set;

[0023] Step 6.2: Improve the target detection model: Improve the target detection model RT-DETR, and improve its backbone network: replace the last layer of convolution in the backbone network with snake convolution to adapt to the line features of batik; Improve the feature fusion framework: Add a feature channel gating module LFCG based on low-rank tensor fusion to the feature fusion framework, and the feature fusion process is as follows:

[0024] Step 6.2.1: First, use a convolution module with a step size of 1 and a convolution kernel size of 1 to unify the number of channels of the two different feature maps; then, perform global maximum pooling of the attention channel on the small-sized feature map to obtain the global maximum pooling value Perform global average pooling of the attention channel on the large-size feature map to obtain the global average pooling value Among them C 256 Indicates that the number of channels of the two feature maps is unified to 256 dimensions;

[0025] Step 6.2.2: P max With P avg Splice an additional value of 1 on the channel dimension dim=1 to obtain an intermediate representation for low-rank tensor fusion and It is expressed as:

[0026] Z max =Concat(P max ,1,dim=1)

[0027] Z avg =Concat(P avg ,1,dim=1)

[0028] Step 6.2.3: We will get Z max and Z avg Perform low-rank tensor fusion:

[0029]

[0030] Where r is the rank factor of the original tensor, M is the number of modes, and w m ( i ) is the corresponding weight, z m is the corresponding intermediate feature representation, h is the fusion information value, and ∧ represents the element product of the tensor; finally, the channel gating value ChannelGate is obtained through the Sigmoid function; the output of the feature fusion module Fusion in the feature fusion framework is multiplied by ChannelGate and then added to the output to obtain a new feature representation, which is input into the subsequent module of the target detection model RT-DETR to obtain the final target detection result;

[0031] Step 6.3: Train and improve the target detection model: set the training hyperparameters, including the learning rate and the number of training rounds, and complete the training when the target detection accuracy reaches convergence; train with the input of the training set, test with the test set, and verify with the validation set to obtain a trained batik image target detection model, which is used to update the product knowledge graph.

[0032] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the ontology class of batik products is determined, including patterns, products, representative patterns, pattern types, layouts, themes, and intentional worship.

[0033] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the ontological relationship of determining batik products includes composition relationship, expression relationship, layout relationship, evolution relationship, belonging relationship, and having relationship; the composition relationship describes the connection between batik patterns and products, the expression relationship describes the connection between products and batik themes, the layout relationship describes the connection between products and product layouts, the evolution relationship describes the connection between representative batik patterns and batik patterns, the belonging relationship describes the connection between representative batik patterns and batik pattern types, and the having relationship describes the connection between batik pattern types and batik intention worship.

[0034] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the acquisition of the theme, layout, pattern type, and intentional worship entity: through literature and data search and analysis, the corresponding entities can be directly obtained, wherein the theme entities include: flowers and plants, auspicious birds and beasts, worldly affairs, pattern geometry and style combinations; layout entities include: scattered layout, symmetrical layout, partition layout and orderly layout; pattern type entities include: fish patterns, bird patterns, butterfly patterns, dragon patterns, human patterns, pear blossom patterns, pomegranate patterns, bronze drum patterns, spiral patterns.

[0035] The above-mentioned method for constructing a product knowledge graph based on image drive, wherein: the neighbor propagation clustering algorithm: initialize the responsibility value res(i,k) and the availability value ava(i,k) to zero, first update the responsibility value res(i,k), res(i,k) represents the suitability of point k as the cluster center of point i, and the calculation formula is:

[0036]

[0037] Among them, ava(i,k') represents the suitability of point i as a cluster member of point k', D(i,k) is the similarity value between node k and node i, and D(i,k') is the similarity value between node k' and node i;

[0038] Then, the availability value ava(i,k) is updated. ava(i,k) indicates the suitability of point i as a cluster member of point k. The responsibility value and the availability value are continuously updated until both of them no longer change significantly or reach the set iteration threshold. The update formula of the availability value ava(i,k) is:

[0039]

[0040] Finally, the converged responsibility value and availability value of each point are summed to obtain the center score value CenterScore(k), the point with the largest center score value is selected as the cluster center, and the point that makes its center score value the largest is selected as the cluster member to complete the selection of representative pattern entities; the center score value CenterScore(k) is obtained by summing the converged responsibility value and availability value of each point, which is expressed as:

[0041] CenterScore(k)=ews(k,k)+ava(k,k)

[0042] The above-mentioned method for constructing a product knowledge graph based on image drive, wherein: the batik product image collection The image was binarized with a pixel threshold of 128.

[0043] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the acquisition of the PR requires determining the theme of a work, and the determination is made according to the established rules: assuming that a batik product image is IMG = {n1, n2, n3, n4, s1, s2, s3, s4}, wherein n1, n2, n3 and n4 respectively represent the ratio of the number of animal patterns, plant patterns, geometric patterns and human patterns in the batik product image IMG to the total number of patterns in the IMG; s1, s2, s3 and s4 respectively represent the number of animal patterns, plant patterns, geometric patterns and human patterns in the IMG The ratio of the area of ​​object patterns, plant patterns, geometric patterns and human patterns to the total area of ​​the IMG; when the sum of n1 and s1 is the largest, the theme of the IMG is auspicious birds and beasts; when the sum of n2 and s2 is the largest, the theme of the IMG is flowers and plants; when the sum of n3 and s3 is the largest, the theme of the IMG is geometric patterns; when the sum of n4 and s4 is the largest, the theme of the IMG is the worldly world; and when all values ​​are very low, it means that patterns other than those specified appear in the IMG, and it is determined to be a style combination.

[0044] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the batik product image dataset is processed and annotated using the open source annotation tool LabelImg to obtain a batik target detection dataset and perform data enhancement.

[0045] The above-mentioned image-driven product knowledge graph construction method, wherein: the batik target detection dataset is divided into training set, validation set and test set in a ratio of 7:1:2.

[0046] The above-mentioned method for constructing a product knowledge graph based on image-driven, wherein: the EXCEL table is used as a storage tool, each relationship corresponds to an EXCEL table, and the entities with the relationship are divided into two columns for storage, the first column is the head entity, the second column is the tail entity, and finally the table is named with the relationship name.

[0047] Compared with the prior art, the present invention has obvious beneficial effects. It can be seen from the above scheme that the present invention first analyzes the batik product image level, establishes the batik field ontology, and on this basis, constructs the knowledge graph to ensure the comprehensiveness and systematicness of the knowledge. Among them, the clustering idea of ​​representative pictures is borrowed, and an entity acquisition method based on image clustering is proposed. In the same pattern type, the batik image is binarized and then clustered to obtain the evolutionary relationship between specific entities and patterns; the knowledge graph is automatically updated based on the RT-DETR target detection model, and a serpentine convolution module is proposed to highlight the line features in the batik image, and a feature channel gating module (LFCG) based on low-rank tensor fusion is designed to adjust the model's attention to general features and line features, thereby improving the performance of the model. There are mainly the following advantages:

[0048] Image-driven batik domain ontology: From the perspective of pattern composition, an image-driven batik domain ontology is proposed, which defines classes such as patterns, pattern types, products, themes, layouts, intention worship, representative patterns, and composition relationships, performance relationships, having relationships, belonging relationships, layout relationships, and evolutionary relationships, laying the foundation for the construction of batik product knowledge graphs.

[0049] Entity acquisition based on image clustering: Based on the batik domain ontology, a knowledge graph of batik products was constructed, and a representative pattern entity acquisition method based on image clustering was proposed. This method clustered patterns of the same category, reduced the interference of batik image color through image binarization preprocessing technology, and obtained representative pattern entities and the evolutionary relationship between patterns based on the clustering results, revealing the development and evolution of batik patterns.

[0050] · Update of batik product knowledge graph based on target detection: In order to realize the automatic update capability of the image-driven batik product knowledge graph, starting from the composition relationship, the snake convolutional network is integrated into the backbone network ResNet of the target detection model, and a channel gating mechanism based on low-rank tensor fusion is proposed to establish a target detection model for automatic extraction of batik patterns, providing support for the automatic update of the knowledge graph.

[0051] The knowledge graph constructed by the present invention can not only enrich the academic research on batik culture, but also provide valuable guidance and inspiration for artistic creation, and can be applied to the research of other intangible cultural heritage similar to batik art, providing new ideas for the protection and dissemination of cultural heritage. In short, the present invention has the characteristics of constructing a knowledge graph by combining with image data.

[0052] The beneficial effects of the present invention are further illustrated below through specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of the present invention;

[0054] Figure 2 This is a schematic diagram of the batik product body framework of the present invention;

[0055] Figure 3 It is a schematic diagram of the image clustering process of the present invention;

[0056] Figure 4 It is a schematic diagram of the structure of the embedded subject determination process of the present invention;

[0057] Figure 5 It is a schematic diagram of the structure of the improved target detection model of the present invention;

[0058] Figure 6 It is a schematic diagram of the structure of the feature channel gating module based on low-rank tensor fusion of the present invention. DETAILED DESCRIPTION

[0059] In conjunction with the accompanying drawings and preferred embodiments, the specific implementation, features and efficacy of an image detection method based on an improved YOLO algorithm proposed in the present invention are described in detail as follows.

[0060] See also Figure 1 , a method for constructing a product knowledge graph based on image-driven in the present invention, wherein: the method comprises the following steps:

[0061] Step 1: Product image data acquisition and preprocessing: Obtain batik product images, preprocess the images using image processing tools, and annotate them using open source image annotation tools;

[0062] The batik product image is obtained by shooting or searching on the Internet. The image processing tool is PhotoShop. The image annotation tool is LabelImg.

[0063] Step 2: Product knowledge graph ontology construction, the specific steps include:

[0064] Step 2.1: Class determination: Based on the analysis of batik product image characteristics, the ontology of batik products is determined, including patterns, products, representative patterns, pattern types, layout, theme, and intention worship;

[0065] Step 2.2: Determination of relationships: Based on the analysis of the characteristics of batik product images, determine the ontological relationships of batik products, including composition relationships, expression relationships, layout relationships, evolution relationships, belonging relationships, and having relationships; the composition relationship describes the relationship between batik patterns and products, the expression relationship describes the relationship between products and batik themes, the layout relationship describes the relationship between products and product layouts, the evolution relationship describes the relationship between representative batik patterns and batik patterns, the belonging relationship describes the relationship between representative batik patterns and batik pattern types, and the having relationship describes the relationship between batik pattern types and batik intention worship;

[0066] Step 3: Product knowledge graph construction: Construct a batik product ontology framework (such as Figure 2 ), combined with the pre-processed product image data, to construct an image-driven batik product knowledge graph, including entity acquisition and relationship acquisition. The specific steps include:

[0067] Step 3.1: Entity acquisition: Get the entity set E, expressed as E = E IMG UE TEXT , where E IMG Indicates product entity, pattern entity, representative pattern entity, E TEXT Represents textual entities such as theme, layout, and pattern type;

[0068] The product entity, pattern entity, and representative pattern entity are represented by images or corresponding image addresses in the constructed product knowledge graph.

[0069] Acquisition of the pattern and product entity: directly using the batik product image after shooting and preprocessing as the product entity, and extracting a single pattern from the product according to the image annotation result to obtain the pattern entity;

[0070] Acquisition of the theme, layout, pattern type, and intention worship entity: Through literature and data search and analysis, the corresponding entity can be directly obtained, among which the theme entity includes: flowers and plants, auspicious birds and beasts, worldly customs, pattern geometry and style combination; layout entity includes: scattered layout, symmetrical layout, partition layout and orderly layout; pattern type entity includes: fish pattern, bird pattern, butterfly pattern, dragon pattern, human pattern, pear flower pattern, pomegranate pattern, bronze drum pattern, spiral pattern;

[0071] The representative pattern acquisition is performed by using image clustering based on the nearest neighbor propagation clustering algorithm (such as Figure 3), firstly, the batik product image collection After binarization with a pixel threshold of 128, feature extraction is performed through the feature extraction network Resnet50, where N is the number of images, C, H, and W represent the number of channels, height, and width of the image, respectively, to obtain the feature representation set of the batik image. Where h represents its feature dimension; then, cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix

[0072] The cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the calculation formula is:

[0073]

[0074] The feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix

[0075] The neighbor propagation clustering algorithm: Initialize the responsibility value res(i, k) and the availability value ava(i, k) to zero, first update the responsibility value res(i, k), res(i, k) represents the suitability of point k as the cluster center of point i, and the calculation formula is:

[0076]

[0077] Among them, ava(i,k') represents the suitability of point i as a cluster member of point k', D(i,k) is the similarity value between node k and node i, and D(i,k') is the similarity value between node k' and node i;

[0078] Then, the availability value ava(i,k) is updated. ava(i,k) indicates the suitability of point i as a cluster member of point k. The responsibility value and the availability value are continuously updated until both of them no longer change significantly or reach the set iteration threshold. The update formula of the availability value ava(i,k) is:

[0079]

[0080] Finally, the converged responsibility value and availability value of each point are summed to obtain the center score value CenterScore(k), the point with the largest center score value is selected as the cluster center, and the point that makes its center score value the largest is selected as the cluster member to complete the selection of representative pattern entities; the center score value CenterScore(k) is obtained by summing the converged responsibility value and availability value of each point, which is expressed as:

[0081] CenterScore(k)=res(k,k)+ava(k,k)

[0082] Step 3.2: Relationship acquisition: The relationship set is set to R = {CR, LR, HR, ER, PR, BR}, where CR represents a composition relationship, LR represents a layout relationship, HR represents a has relationship, ER represents an evolution relationship, PR represents a performance relationship, and BR represents a belongs relationship. CR is used to describe the connection between products and patterns. When acquiring product and pattern nodes, the corresponding CR is also obtained; the acquisition of the PR requires the determination of the theme of a work, which is determined according to the established rules; the layout relationship LR is determined and acquired using the description method of relevant literature; the belongs relationship BR and the has relationship HR are obtained through data search; the evolution relationship ER is obtained by using the representative pattern obtained by the clustering algorithm, and the pattern forming the cluster has an evolution relationship with it;

[0083] The acquisition of PR requires determining the theme of a work and making the determination according to the established rules (e.g. Figure 4 ): Assume that a batik product image is IMG = {n1, n2, n3, n4, s1, s2, s3, s4}, where n1, n2, n3 and n4 represent the ratio of the number of animal patterns, plant patterns, geometric patterns and human patterns in the batik product image IMG to the total number of patterns in the IMG; s1, s2, s3 and s4 represent the area of ​​animal patterns, plant patterns, geometric patterns and human patterns in the IMG, respectively. The ratio of the total area of ​​IMG to that of G; when the sum of n1 and s1 is the largest, the theme of the IMG is auspicious birds and beasts; when the sum of n2 and s2 is the largest, the theme of the IMG is flowers and plants; when the sum of n3 and s3 is the largest, the theme of the IMG is geometric patterns; when the sum of n4 and s4 is the largest, the theme of the IMG is the mundane world; and when all values ​​are very low, it means that patterns other than those specified appear in the IMG, and it is determined to be a style combination.

[0084] Step 4: Get results and organize them: Collect and organize all the entities and relationships obtained, and use Excel as a storage tool. Each relationship corresponds to an Excel table. The entities with the relationship are divided into two columns for storage. The first column is the head entity, and the second column is the tail entity. Finally, the table is named with the relationship name.

[0085] Step 5: Visualization of product knowledge graph: Import the sorted EXCEL spreadsheet files in batches through the interactive code between Python and the graph database Neo4j, and visualize the product knowledge graph through Neo4j;

[0086] Step 6: Automatic update of product knowledge graph: For the automatic update of composition relationships, patterns are automatically extracted from batik product images. This is achieved based on target detection technology. The specific steps include:

[0087] Step 6.1: Process the batik product image dataset to obtain the batik target detection dataset, and divide it into training set, validation set and test set.

[0088] The batik product image dataset is processed by using an open source labeling tool LabelImg to obtain a batik target detection dataset and perform data enhancement.

[0089] The batik target detection dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0090] Step 6.2: Improve the target detection model: Figure 5 As shown in the figure, the target detection model RT-DETR is improved. The backbone network is improved by replacing the last layer of convolution in the backbone network with snake convolution to adapt to the line features of batik. The feature fusion framework is improved by adding a feature channel gating module LFCG based on low-rank tensor fusion to the feature fusion framework (such as Figure 6 ), the feature fusion process is:

[0091] Step 6.2.1: First, use a convolution module with a step size of 1 and a convolution kernel size of 1 to unify the number of channels of the two different feature maps; then, perform global maximum pooling of the attention channel on the small-sized feature map to obtain the global maximum pooling value Perform global average pooling of the attention channel on the large-size feature map to obtain the global average pooling value Among them C 256 Indicates that the number of channels of the two feature maps is unified to 256 dimensions;

[0092] Step 6.2.2: P max With P avgSplice an additional value of 1 on the channel dimension dim=1 to obtain an intermediate representation for low-rank tensor fusion and It is expressed as:

[0093] Z max =Comcat(P max , 1, dim=1)

[0094] Z avg =Concat(P avg ,1,dim=1)

[0095] Step 6.2.3: We will get Z max and Z avg Perform low-rank tensor fusion:

[0096]

[0097] Where r is the rank factor of the original tensor, M is the number of modes, and w m ( i ) is the corresponding weight, z m is the corresponding intermediate feature representation, h is the fusion information value, and ∧ represents the element product of the tensor; finally, the channel gating value ChannelGate is obtained through the Sigmoid function; the output of the feature fusion module Fusion in the feature fusion framework is multiplied by ChannelGate and then added to the output to obtain a new feature representation, which is input into the subsequent module of the target detection model RT-DETR to obtain the final target detection result;

[0098] Step 6.3: Train and improve the target detection model: set the training hyperparameters, including the learning rate and the number of training rounds, and complete the training when the target detection accuracy reaches convergence; train with the input of the training set, test with the test set, and verify with the validation set to obtain the trained batik image target detection model, which is used to update the product knowledge graph. Performance analysis:

[0099] In terms of target detection data sets, the present invention divides the constructed batik target detection data set into training set, validation set and test set in a ratio of 7:1:2.

[0100] The target detection experiment of the present invention is carried out on a computer using a NIVIDIA GeForce RTX 3090Ti GPU, with 24GB RAM and Windows 11 operating system. The deep learning framework used is pytorch 2.0.1+cu117. The optimizer used is AdamW, and the learning rate is set to 0.0001 and the batch size is set to 4. In order to better explore the effectiveness of the module of the present invention, the present invention conducts comparative experiments of multiple models and ablation experiments to explore the improvement effect. In order to increase the amount of data used for training, the present invention performs online enhancement on the batik target detection dataset, as shown in Table 1.

[0101] Table 1 Items and parameters used for data augmentation

[0102]

[0103] The present invention is first compared with four models in recent years, as shown in Table 2. Since these four methods directly use ResNet-18 as the backbone network, the present invention introduces the corresponding pre-trained weights for experiments. First, the comparison with the convolutional network model Sparse R-CNN and Tood shows that the method of the present invention is superior to the above two models in the three indicators used for comparison. As newer models, Sparse R-CNN and Tood have excellent effects in general scenarios, but in special scenarios such as batik images, certain optimization and improvement are still needed. Compared with the Transformer-based target detection model DINO and Conditional DETR model, the model of the present invention is also leading. These two models also bring certain difficulties and challenges due to batik image data. However, since the method of the present invention introduces snake convolution, it does not use pre-trained weights. Therefore, in order to make a fairer comparison, the present invention also trains an RT-DETR model based on pre-trained weights. It can be found that after the introduction of pre-trained weights, the various indicators of the RT-DETR model are worse than those of the RT-DETR model without the introduction of pre-trained weights, but the difference is not large. This not only shows that the pre-trained weights have little effect on the effect of batik image target detection, but also proves that although the pre-trained weights can cope with special images such as batik, they only use the relatively generalized feature extraction ability of the pre-trained weights. The overall effect is still not as good as the special improvement made by the present invention for batik image data. In order to explore whether the LFCG module proposed in the present invention can be better combined with the snake convolution results, it can be found from the results of the RT-DETR model that replaces the feature fusion framework in the ASF, GoldYolo and HS-FPN models respectively that the method of the present invention is ahead of the above methods. The possible reason is that the above three feature fusion frameworks are either for medical images or for medical images.[46,48] , or it is designed for general images

[47] This also shows that the design of the present invention is more suitable for batik images.

[0104] Table 2 Results of various models and plug-and-play frameworks in batik dataset. “Residual Network-18-Dynamic Snake Convolution” means that the convolution module of the last layer in the backbone network is replaced by the dynamic snake convolution module, “Pre-training” indicates whether the corresponding pre-training weights of the backbone network are used, and the method with “*” indicates that the present invention replaces its feature fusion framework into RT-DETR.

[0105]

[0106] In short, in order to better carry out the inheritance and research of batik art, the present invention, from the perspective of batik product images, explains how to build an image-driven batik knowledge graph based on the various elements in batik products. By establishing a batik product domain ontology, and on this basis, the collected batik product information and image generation knowledge triples are constructed to construct an image-driven batik product knowledge graph, which includes more than 10,000 entities and more than 20,000 relationships, and is visualized using Gephi and Neo4j. This knowledge graph provides additional research tools for batik art enthusiasts and researchers to promote the inheritance and innovation of batik art.

[0107] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A method for constructing a product knowledge graph based on image-driven, characterized in that: The method comprises the following steps: Step 1: Product image data acquisition and preprocessing: Obtain batik product images and preprocess them, and use image annotation tools to annotate them; Step 2: Product knowledge graph ontology construction, the specific steps include: Step 2.1: Class determination: Based on the analysis of the characteristics of batik product images, determine the ontology class of batik products; Step 2.2: Determination of relationships: Based on the analysis of the characteristics of batik product images, determine the ontological relationships of batik products, including composition relationships, expression relationships, layout relationships, evolution relationships, belonging relationships, and having relationships; Step 3: Product knowledge graph construction: According to the ontology classes and ontology relationships of the batik products, a batik product ontology framework is constructed. Combined with the pre-processed product image data, an image-driven batik product knowledge graph is constructed, including entity acquisition and relationship acquisition. The specific steps include: Step 3.1: Entity acquisition: Get the entity set E, expressed as E = E IMG UE TEXT , where E IMG Represents image entities, including products, patterns, and representative pattern entities. TEXT Represents text-type entities, including subject, layout, and pattern type entities; The products, patterns, and representative pattern entities are represented by images or corresponding image addresses in the constructed product knowledge graph; Acquisition of the pattern and product entity: directly using the batik product image after shooting and preprocessing as the product entity, and extracting a single pattern from the product according to the image annotation result to obtain the pattern entity; The representative pattern acquisition is performed by using image clustering based on the nearest neighbor propagation clustering algorithm. First, the batik product image set After binarization, feature extraction is performed through the feature extraction network, where N is the number of images, C, H, and W represent the number of channels, height, and width of the image, respectively, to obtain the feature representation set of the batik image. Where h represents its feature dimension; then, cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix The cosine similarity is used as the clustering metric method, by calculating two n-dimensional vectors and The cosine value of the angle between them is used to evaluate the similarity between the two CosineSimilarity(A,B), and the calculation formula is: The feature representation set F is used as the input of the clustering metric method to calculate the similarity matrix Step 3.2: Relationship acquisition: The relationship set is set to R = {CR, LR, HR, ER, PR, BR}, where CR represents a composition relationship, LR represents a layout relationship, HR represents a has relationship, ER represents an evolution relationship, PR represents a performance relationship, and BR represents a belongs relationship. CR is used to describe the connection between products and patterns. When acquiring product and pattern nodes, the corresponding CR is also obtained; the acquisition of the PR requires the determination of the theme of a work, which is determined according to the established rules; the layout relationship LR is determined and acquired using the description method of relevant literature; the belongs relationship BR and the has relationship HR are obtained through data search; the evolution relationship ER is obtained by using the representative pattern obtained by the clustering algorithm, and the pattern forming the cluster has an evolution relationship with it; Step 4: Get results and organize them: collect and organize all the entities and relationships obtained, and use EXCEL as a storage tool; Step 5: Visualization of product knowledge graph: Import the sorted EXCEL spreadsheet files in batches through the interactive code between Python and the graph database Neo4j, and visualize the product knowledge graph through Neo4j; Step 6: Automatic update of product knowledge graph: For the automatic update of composition relationships, patterns are automatically extracted from batik product images. This is achieved based on target detection technology. The specific steps include: Step 6.1: Process the batik product image dataset to obtain a batik target detection dataset, and divide it into training set, validation set and test set; Step 6.2: Improve the target detection model: Improve the target detection model RT-DETR, and improve its backbone network: replace the last layer of convolution in the backbone network with snake convolution to adapt to the line features of batik; Improve the feature fusion framework: Add a feature channel gating module LFCG based on low-rank tensor fusion to the feature fusion framework, and the feature fusion process is as follows: Step 6.2.1: First, use a convolution module with a step size of 1 and a convolution kernel size of 1 to unify the number of channels of the two different feature maps; then, perform global maximum pooling of the attention channel on the small-sized feature map to obtain the global maximum pooling value Perform global average pooling of the attention channel on the large-size feature map to obtain the global average pooling value Among them C 256 Indicates that the number of channels of the two feature maps is unified to 256 dimensions; Step 6.2.2: P max With P avg Splice an additional value of 1 on the channel dimension dim=1 to obtain an intermediate representation for low-rank tensor fusion It is expressed as: Z max =Concat(P max ,1,dim=1) Z avg =Concat(P avg ,1,dim=1) Step 6.2.3: We will get Z max and Z avg Perform low-rank tensor fusion: Where r is the rank factor of the original tensor, M is the number of modes, and w m (i) is the corresponding weight, z m is the corresponding intermediate feature representation, h is the fusion information value, and ∧ represents the element product of the tensor; finally, the channel gating value ChannelGate is obtained through the Sigmoid function; the output of the feature fusion module Fusion in the feature fusion framework is multiplied by ChannelGate and then added to the output to obtain a new feature representation, which is input into the subsequent module of the target detection model RT-DETR to obtain the final target detection result; Step 6.3: Train and improve the target detection model: set the training hyperparameters, including the learning rate and the number of training rounds, and complete the training when the target detection accuracy reaches convergence; train with the input of the training set, test with the test set, and verify with the validation set to obtain a trained batik image target detection model, which is used to update the product knowledge graph.

2. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 2.1, the ontology of batik products is determined, including patterns, products, representative patterns, pattern types, layouts, themes, and intentional worship.

3. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 2.2, the ontological relationship of batik products is determined, including composition relationship, expression relationship, layout relationship, evolution relationship, belonging relationship, and having relationship; the composition relationship describes the connection between batik patterns and products, the expression relationship describes the connection between products and batik themes, the layout relationship describes the connection between products and product layouts, the evolution relationship describes the connection between representative batik patterns and batik patterns, the belonging relationship describes the connection between representative batik patterns and batik pattern types, and the having relationship describes the connection between batik pattern types and batik intention worship.

4. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 3.1, the acquisition of the theme, layout, pattern type, and intentional worship entities: through literature and data search and analysis, the corresponding entities can be directly obtained, among which the theme entities include: flowers and plants, auspicious birds and beasts, worldly affairs, pattern geometry and style combinations; layout entities include: scattered layout, symmetrical layout, partition layout and orderly layout; pattern type entities include: fish patterns, bird patterns, butterfly patterns, dragon patterns, human patterns, pear blossom patterns, pomegranate patterns, bronze drum patterns, and spiral patterns.

5. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 3.1, the neighbor propagation clustering algorithm: initialize the responsibility value res(i, k) and the availability value ava(i, k) to zero, first update the responsibility value res(i, k), res(i, k) represents the suitability of point k as the cluster center of point i, and the calculation formula is: Among them, ava(i,k') represents the suitability of point i as a cluster member of point k', D(i,k) is the similarity value between node k and node i, and D(i,k') is the similarity value between node k' and node i; Then, the availability value ava(i,k) is updated. ava(i,k) indicates the suitability of point i as a cluster member of point k. The responsibility value and the availability value are continuously updated until both of them no longer change significantly or reach the set iteration threshold. The update formula of the availability value ava(i,k) is: Finally, the converged responsibility value and availability value of each point are summed to obtain the center score value CenterScore(k), the point with the largest center score value is selected as the cluster center, and the point that makes its center score value the largest is selected as the cluster member to complete the selection of representative pattern entities; the center score value CenterScore(k) is obtained by summing the converged responsibility value and availability value of each point, which is expressed as: CenterScore(k)=res(k,k)+ava(k,k).

6. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 3.1, the batik product image collection Binarization was performed with a pixel threshold of 128.

7. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 3.2, the acquisition of the PR requires determining the theme of a work, and the determination is made according to the established rules: Assume that a batik product image is IMG = {n1, n2, n3, n4, s1, s2, s3, s4}, where n1, n2, n3 and n4 respectively represent the ratio of the number of animal patterns, plant patterns, geometric patterns and human patterns in the batik product image IMG to the total number of patterns in the IMG; s1, s2, s3 and s4 respectively represent the number of animal patterns, plant patterns, geometric patterns and human patterns in the IMG. The ratio of the area of ​​patterns, geometric patterns and human patterns to the total area of ​​the IMG; when the sum of n1 and s1 is the largest, the theme of the IMG is auspicious birds and beasts; when the sum of n2 and s2 is the largest, the theme of the IMG is flowers and plants; when the sum of n3 and s3 is the largest, the theme of the IMG is geometric patterns; when the sum of n4 and s4 is the largest, the theme of the IMG is the worldly world; and when all values ​​are very low, it means that patterns other than those specified appear in the IMG, and it is determined to be a style combination.

8. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 6.1, the batik product image dataset is processed and annotated using the open source annotation tool LabelImg to obtain a batik target detection dataset and perform data enhancement.

9. The method for constructing a product knowledge graph based on image-driven operation according to any one of claims 1 and 8, characterized in that: In step 6.1, the batik target detection dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1:

2.

10. The method for constructing a product knowledge graph based on image-driven operation according to claim 1, characterized in that: In step 4, the EXCEL table is used as a storage tool, each relationship corresponds to an EXCEL table, and the entities with the relationship are divided into two columns for storage, the first column is the head entity, the second column is the tail entity, and finally the table is named with the relationship name.

Images