Man-machine fusion driven knowledge graph construction method and system

Abstract

The invention discloses a human-machine fusion driven knowledge graph construction method and system. The system comprises a classification identification module, a knowledge extraction module and a visual analysis module. And the classification identification module is used for establishing a classification system and a data set in combination with knowledge of experts. And the knowledge extraction module is used for performing knowledge extraction on the ancient painting to obtain labels of entities in the ancient painting and relationships among different entities. And the visual analysis module is used for establishing a joint embedded visual view for an expert to modify the predicted semantic features. According to the method, an active learning algorithm and a man-machine intelligent cooperation system are fused, so that the automatic construction efficiency of the ancient painting knowledge graph is improved; according to the method, the knowledge graph thought is introduced, important semantic information can be effectively expressed, domain knowledge of experts is integrated in an efficient mode through an interactive visual interface, and the initially constructed knowledge graph is completed and optimized.

Description

Technical Field

[0001] The present invention belongs to the field of information technology; specifically, it relates to a knowledge graph construction method and system driven by human-machine fusion. Background Art

[0002] Ancient paintings reflect the philosophical thoughts and literary folklore in traditional Chinese culture and have high artistic and educational value. In order to protect them, many museums in my country have converted a large number of ancient painting collections into digital images and established special databases for management, such as CIT, MISS, CTM, etc. However, the current theme-based management method cannot classify ancient paintings with diverse styles well to support high-level semantic retrieval. This has brought certain difficulties to the further analysis and exploration of ancient paintings.

[0003] As a powerful semantic representation, knowledge graphs have been widely used in multiple image tasks such as image classification, retrieval, and analysis. The great success of these tasks has also introduced a new way to manage and analyze ancient paintings, namely the use of ancient painting knowledge graphs (Paint KG). However, despite the emergence of many studies on image knowledge graphs, few works focus on the construction of ancient painting knowledge graphs. Due to the natural differences between ancient paintings and images, most of the artificial intelligence models proposed in these studies (such as SGG) cannot be effectively applied to ancient paintings. In addition, the diverse visual elements in ancient paintings and their complex semantic relationships are difficult to fully extract through the model.

[0004] Based on the above considerations, two major challenges in constructing the ancient painting knowledge graph can be derived. The first is to obtain semantic labels that describe entities (such as cranes, zithers, headscarves, etc.) and relationships (for example, play, hold, look) in Chinese paintings. The different semantic labels between ancient paintings and natural images hinder the use of existing models, so it is necessary to train an effective model for ancient paintings to produce more accurate results. The second is to integrate domain knowledge into the ancient painting knowledge graph. Limited by the performance of the model, the automatically extracted entities and relationships may be wrong and incomplete. This requires domain knowledge to guide the generation of knowledge graphs to improve the quality of the ancient painting knowledge graph.

[0005] To overcome these two challenges, through close collaboration with domain experts, a visual analysis tool, VisTCP, which supports the construction of ancient painting knowledge graphs and human-in-the-loop model optimization, is proposed to facilitate the construction of ancient painting knowledge graphs.

[0006] First, in collaboration with art historians and domain experts, we build a taxonomy and expert annotation dataset for ancient paintings to support transfer learning of knowledge extraction models for ancient paintings, so as to realize the automatic extraction of entities and relationships in ancient paintings. Secondly, we design a visualization view that jointly embeds images and semantic features. Through interactive refinement components and uncertainty visualization components, we allow users to optimize knowledge graph construction and model iterative optimization based on domain knowledge. Finally, we propose VisTCP, a visual analysis tool that supports the construction of ancient painting knowledge graphs and human-in-the-loop model optimization. Through active learning methods, we collect expert feedback, expand annotation samples, and perform iterative model improvements to reduce labor costs and improve the accuracy and reliability of the automated construction of ancient painting atlases. Summary of the invention

[0007] A method for constructing an ancient painting knowledge graph driven by human-machine fusion, characterized in that it specifically includes the following steps: In a first aspect, the present invention provides a method for constructing an ancient painting knowledge graph driven by human-machine fusion, which comprises the following steps: Step 1: Construct a classification system for different entities in ancient painting images, and establish a dataset labeled corresponding to the classification system.

[0008] Step 2: Build a knowledge extraction model and use the data set to train the knowledge extraction model. The knowledge extraction model identifies the target ancient painting and obtains prediction features.

[0009] Step 3: Build a human-in-the-loop model optimization visual analysis tool. The model optimization visual analysis tool fuses the annotation features and the semantic features predicted by the knowledge extraction model with the image features to establish a joint embedding visualization view. The joint embedding visualization view is used to display the difference between the annotation features and the predicted semantic features.

[0010] Step 4: The user modifies the predicted semantic features displayed by the visual analysis tool based on the differences shown in the joint embedding visualization view. The modified semantic features are added to the dataset and the knowledge extraction model is iteratively trained again.

[0011] Step 5: The knowledge extraction model re-identifies the target ancient painting and generates an ancient painting knowledge graph based on the recognition results.

[0012] Preferably, the classification system includes entity types and relationships between different entities. The relationships between different entities include positional relationships and event relationships.

[0013] Preferably, the model optimization visual analysis tool includes an uncertainty visualization component; the uncertainty visualization component uses different visual representations to encode entity information and semantic relationships. A scatter plot is used to display entity information, showing model prediction and annotated entity information. The color of each prediction point encodes the semantic label of the entity; the border style of the prediction point is used to distinguish the annotated features from the predicted features. The border thickness of the prediction point predicts the confidence of the label. The semantic relationship is displayed by a chart that records the orientation relationship and event relationship between different entities.

[0014] Preferably, the model optimization visual analysis tool includes an interactive refinement component. The interactive refinement component selects entities identified by the knowledge extraction model and displays predicted semantic information. The user modifies the semantic features through the interactive refinement component.

[0015] Preferably, the modification of semantic features by the interactive refining component includes: 1) modifying an entity: the user changes the label of an entity by clicking on the corresponding bounding box; 2) adding an entity: the user draws a wireframe and selects a label; 3) deleting an entity: the user deletes the bounding box identified by the knowledge extraction model; 4) modifying a relationship: the user deletes the relationship line between two entities or modifies the relationship label displayed on the relationship line; 5) adding a relationship: the user clicks on entities in turn to add relationship labels.

[0016] Preferably, the model optimization visual analysis tool comprises a structured representation component, wherein the structured representation component comprises a triple table, and the triple table is used to record interaction operations.

[0017] Preferably, the process of establishing a joint embedding visualization view is as follows: by calculating the Euclidean distance of the ancient painting feature matrix and adjusting it to minimize the distance between similar label elements; introducing an uncertainty visualization component to display the distance between embedded vectors and vectors related to semantic labels, and encoding them on a two-dimensional plane, using scatter plots and icons to visually design entities and semantic relationships; and generating a visualization view that displays feature similarity by means of data dimensionality reduction.

[0018] Preferably, the knowledge extraction model includes a target recognition model and a relationship reasoning model. The target recognition model is used to recognize multiple targets in the ancient painting image according to the classification system constructed in step 1; the relationship reasoning model is used to reason about the relationship between different targets.

[0019] In the second aspect, the present invention provides a human-machine fusion driven ancient painting knowledge graph construction system, which is used to execute the aforementioned human-machine fusion driven ancient painting knowledge graph construction method; the ancient painting knowledge graph construction system includes a classification identification module, a knowledge extraction module and a visualization analysis module. The classification identification module is used to establish a classification system and a data set in combination with the knowledge of experts. The knowledge extraction module is used to extract knowledge from ancient paintings, obtain the labels of entities in ancient paintings and the relationships between different entities. The visualization analysis module is used to establish a joint embedded visualization view for experts to modify the predicted semantic features.

[0020] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the method for constructing an ancient painting knowledge graph driven by human-machine fusion as described in any one of claims 1 to 7.

[0021] In a fourth aspect, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the method for constructing an ancient painting knowledge graph driven by human-machine fusion as described in any one of claims 1 to 7.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The present invention integrates active learning algorithms and human-machine intelligent collaborative systems to improve the efficiency of automated construction of ancient painting knowledge graphs; the idea of ​​knowledge graphs is introduced to effectively express important semantic information, and through interactive visualization interfaces, the domain knowledge of experts is integrated in an efficient manner; a model optimization visual analysis system that supports the construction of ancient painting knowledge graphs and human-in-the-loop is designed, which can absorb expert knowledge, complete and optimize the initially constructed knowledge graph. Active learning algorithms are used to train entity and relationship extraction models, and continuous iterative optimization is performed in the process of annotating ancient painting knowledge graphs, significantly reducing the workload of manual annotation. The present invention focuses on providing great convenience for the construction of ancient painting knowledge graphs by combining artificial intelligence technology and domain expert knowledge. The use of this system to construct ancient painting knowledge graphs has greatly improved the work efficiency of experts and provided strong support for the study of art history. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2It is a schematic diagram of the system of the present invention. Part A is an interactive refinement component, which presents the ancient painting knowledge extraction results predicted by the model and provides reference samples for users to interactively optimize the results; Part B is an uncertainty visualization component, which compares the visual and semantic features between the elements marked by experts and the elements predicted by the model, and visualizes the uncertain results for inspection; Part C is a structured representation component, which displays the knowledge graph constructed by the model based on the current ancient painting. DETAILED DESCRIPTION

[0024] The following is a further description of the ancient painting knowledge graph construction method driven by human-machine fusion of the present invention in conjunction with the accompanying drawings: like Figure 1 As shown, a method for constructing an ancient painting knowledge graph driven by human-machine fusion includes the following steps: (1) Constructing a semantic label classification system for ancient paintings. First, an expert pilot study was conducted, in which experts used the tool to annotate 500 ancient paintings of various types, adding entities and relationships in the ancient paintings by creating bounding boxes and establishing relationship links. The types of ancient paintings include figure paintings, landscape paintings, and flower and bird paintings.

[0025] According to the expert annotated data set, semantic labels are integrated to establish a classification system specifically for the semantic labels of ancient paintings. The classification system established in this embodiment is shown in Table 1. This system not only covers common visual content descriptions, but also pays special attention to the semantic representation of ancient Chinese events, which solves the problem that the existing classification system ignores the semantics of ancient Chinese events when describing ancient paintings, and enhances the model's ability to understand and describe the semantics of ancient paintings.

[0026] Table 1 Comparison table of semantic classification systems of ancient paintings Entity Class Entity Name people Men, women, children, servants, monks, Buddha Natural landscape Mountains, trees, river, stones, plants, banana, flowers, leaves, bamboo, branches, fruits, pine trees, waterfalls, petals, grass, landscape, peaks animal Horse, cow, dog, cat, bird, deer, sheep, donkey, bee, butterfly, phoenix crane, dragonfly, duck, chicken, fish, lion, peacock, rabbit Cultural Relics Pots, jars, vases, incense boxes, wine vessels, drinking utensils, baskets, gourds, incense burners, wine glasses, plates, ruyi, fans, whisks, toys, kites, goods, merchandise, lanterns, umbrellas, bonsai, mirrors, canes, flags, candles, chains, candlesticks, railings, steps, roofs, doors, windows, bridges, paths, streets, boats, vehicles, pavilions, houses, courtyards, towers, stages, books, scrolls, seals, chessboards, inkstones, pens, pen holders, calligraphy and painting, robes, shoes, hats, headscarves, belts, couches, tables, stools, chairs, screens, carpets, pipa, ruan, zither, drums, flutes, gongs, tripods Relationship Tags Relationship Name event carry, cover, eat, fly to, hold, read, lie on, draw on, serve, play with, ride, talk to, sit on, stand on, water, walk on, watch, wear, have, hang on, flow into, listen to, drag, follow, pull, carry, travel, beat position along, behind, in, in front of, near, on, under, beside (2) A knowledge extraction model for ancient paintings is constructed based on the SGG model (Scene Graph Generation). The knowledge extraction model includes a target recognition model and a relational reasoning model. The target recognition model is used to recognize multiple targets in ancient painting images; the ancient painting semantic classification system established in step (1) is used as the category label of the target to ensure that the model adapts to the characteristics of ancient paintings. The loss function of the target recognition model selects the cross entropy loss function to supervise the transfer learning process of the network and guide the model training. The relational reasoning model is used to reason about the relationship between different targets to enhance the understanding of the complex semantics in ancient paintings. Through transfer learning, the relational reasoning model obtains information from existing relational reasoning knowledge and effectively applies it to the structured representation of ancient paintings. To ensure the effectiveness of model training, after completing the construction of the ancient painting semantic classification system and the initial expert annotation sample set in step (1), the model training is combined with transfer learning and expert annotation, and the model is continuously optimized to improve the recognition accuracy of ancient painting objects and relationships.

[0027] (2.1) In this embodiment, the object recognition model is built based on the Mask-RCNN algorithm. Mask-RCNN is a state-of-the-art object detection algorithm that is widely used for its ease of use and customization, accurate object detection, and faster processing. The object recognition model uses ResNet-101 as the backbone network architecture and the proposed ancient painting semantic classification system as the category label to adapt to the characteristics of ancient paintings, effectively deal with the gradient vanishing problem in deep networks, and improve training efficiency and detection accuracy.

[0028] In terms of loss function, in order to supervise network transfer learning, cross entropy loss is selected to train the object recognition model of ancient paintings to predict the distance between the probability distribution and the true distribution.

[0029] Among them, L CE is the distance value; N is the number of samples, C is the number of categories, represents the category c ground truth label of sample i (if the sample belongs to category c, the value is 1; otherwise, the value is 0), is the predicted probability of category c for sample i.

[0030] (2.2) In this embodiment, the relational reasoning model is built based on the TDE algorithm (total direct effect). The TDE algorithm is applied to the relationship between reasoning objects to enhance the understanding of complex semantics in ancient paintings, solve the serious bias problem in SGG, and successfully achieve rich relation prediction output. The relational reasoning model uses ResNet-101 as the basic network architecture for predicting entity relationships in ancient paintings. By combining the ancient painting semantic classification system and the initial expert annotation sample set, the model can be adapted to ancient paintings.

[0031] (3) Combine expert knowledge to visualize the uncertainty of knowledge extraction by the knowledge extraction model. Through joint embedding visualization technology, the predicted elements of the knowledge extraction model are intuitively compared with the expert annotation elements. Experts can clearly view the model prediction results and provide feedback, thereby optimizing the structured representation.

[0032] (3.1) The real semantic features and predicted semantic features of each target in the ancient painting are combined with the image features to construct an embedding vector. The embedding vector is projected onto a two-dimensional plane, and different channels encode the embedding vectors representing both image features and semantic labels. The visualization uses a scatter plot to represent entities, color coding to represent semantic labels, and the border style to distinguish between predicted and expert labeled samples, and the border thickness to indicate the confidence of the predicted label.

[0033] (3.2) t-SNE is used to maintain local clustering of embedding vectors, making it easier to compare predictions with expert annotations. The comparison view shows the embedding vectors of the model predictions and expert annotations, allowing users to intuitively understand the extraction results and identify high uncertainty elements for further verification.

[0034] (3.3) Provide expert-annotated samples that are most similar to the selected prediction samples for expert reference and refinement. Through the uncertainty visualization tool, users can evaluate the quality of automatic knowledge extraction of the model, and discover and refine uncertain elements, thereby further improving the reliability of the model and providing important support for optimizing the construction of the ancient painting knowledge graph.

[0035] (4) Adopt an active learning strategy to continuously optimize the model by collecting feedback from domain experts on model uncertainty. First, the model identifies samples with uncertainty in the prediction process. These samples are usually instances where the model prediction results are significantly different from the expert annotation results. Experts use the joint embedding visualization tool shown in step (3) to visually view these differences and provide feedback, including confirmation, correction, or supplementation of the model prediction results. Based on the expert feedback, new annotated samples are added, especially those with uncertain prediction results, for subsequent iterative training of the model. After each iteration, the performance of the model is improved, and the optimization process is repeated until the model achieves satisfactory accuracy. Through this interactive iterative method, not only the workload of manual annotation by experts is reduced, but also the model is helped to better understand domain knowledge, thereby improving prediction accuracy and accelerating model optimization.

[0036] (5) Build a human-in-the-loop model optimization visual analysis tool; this tool supports automatic knowledge extraction of ancient paintings and expert interactive iterative optimization model functions, integrates intelligent models and expert prior knowledge through an intuitive visual interface, and realizes efficient automatic construction of ancient painting knowledge graphs. The system includes the following parts:

[0037] (5.1) Uncertainty Visualization Component ( Figure 2 Part B of the ), which integrates the model panel and comparison view to improve the comprehensibility of knowledge extraction.

[0038] (5.1.1) The Model panel supports automatic knowledge extraction and includes two main functions: entity extraction and relation reasoning. The entity extraction function can detect objects and output corresponding semantic labels and bounding boxes, while relation reasoning predicts semantic relationships between pairs of objects, ultimately integrating preliminary structured representations into the workflow. The comparison view displays the embedding vectors predicted by the model and the embedding vectors of expert-annotated samples, providing a joint visualization space that allows art historians to intuitively understand the extraction results and identify elements with high uncertainty for further verification. To maintain interpretability, this view simultaneously displays three aspects of information: 1) the distance between the embedding vectors obtained by image feature extraction, 2) the embedding vectors associated with each semantic label type, and 3) the embedding vectors associated with model predictions and expert annotations. The embedding vectors are projected onto a two-dimensional plane and different aspects of information are encoded in different channels for easy distinction.

[0039] (5.1.2) When visualizing the joint embedding vector, different visual representation schemes are used to encode entities and semantic relations. For entities, scatter plots are used to show the entity information of model predictions and expert annotations in detail. The color of each point encodes the semantic label of the entity, while the border style of the point is used to distinguish between predictions and expert annotations, with solid borders and dashed borders representing predictions and expert samples, respectively. In addition, the thickness of the predicted point border indicates the confidence of the predicted label. For semantic relations, a diagram is designed that integrates subjects, relations, and objects to distinguish different semantic events, and a similar design as for entities is used to distinguish predicted relations from expert annotations.

[0040] (5.1.3) The arrangement of the embedding vectors is determined by feature similarity. The t-SNE technique is used to maintain local clustering during dimensionality reduction, keep the intra-class distance close, and maximize the inter-class distance, thereby facilitating intuitive comparison of model predictions and expert annotation results, helping users understand the quality of knowledge extraction and indicating potential uncertainties. Through this uncertainty visualization component, the system can effectively integrate the intelligent model with the expert's prior knowledge, providing strong support for the automatic construction of the ancient painting knowledge graph.

[0041] (5.2) Interactive Refining Components ( Figure 2 The interactive refinement component provides a powerful visual coordination environment and extensive interactive functions to simplify the iterative refinement process of knowledge extraction and is guided by the expert's prior knowledge. The interactive refinement component jointly presents an overview of the initial recognition results through the label system view, the ancient painting view, and the expert view, thereby promoting a consistent and efficient refinement process.

[0042] (5.2.1) Label System View: This view provides an overview of the proposed semantic classification system for ancient paintings, with corresponding label icons to clarify the unique characteristics of objects and events related to ancient China. In this overview, users can select entities or relationships and adjust the accuracy, which facilitates filtering and focusing on specific interest categories for refinement.

[0043] (5.2.2) Ancient Painting View: This view shows the original ancient painting image and the recognition results, including the bounding boxes of the objects, the lines of the relationships, and the corresponding predicted labels. In order to improve the efficiency of users in refining entities and relationships, and at the same time reduce the visual clutter caused by numerous entity bounding boxes and relationship lines, the bounding box of each entity will be transformed into a precise point mark after the user confirms its semantic label. For the relationship between entity pairs, these connections are initially represented by dotted lines, and will become solid lines after confirmation, so as to clearly reflect the status of the relationship.

[0044] (5.2.3) Expert View: This view displays expert-annotated samples with the most similar features to the selected prediction sample for refinement reference. Users can transfer the labels of expert-annotated samples to the prediction sample by double-clicking to enhance the accuracy of the annotation. In this view, expert-annotated samples are displayed in depth, including specific entity or event annotations and corresponding illustrations, providing users with additional contextual information.

[0045] (5.2.4) The design of this annotation prototype makes interactive refinement possible, working in conjunction with other views to support six operations: 1) Modify entities: users can change the label of an entity by clicking on the corresponding bounding box; 2) Add entities: users can start this operation by clicking the arrow icon in the upper right corner, then draw a frame and select a label to complete the entity addition; 3) Delete entities: users can delete any active entity bounding box by clicking the delete button; 4) Modify relationships: users can delete, modify, or confirm relationship labels by double-clicking the relationship line between two entities; 5) Add relationships: users can click on the first and last entities in sequence to add relationships; 6) The provision of zoom and pan tools further supports knowledge extraction and exploration of ancient paintings. Through this interactive refinement component, the system can effectively integrate intelligent models with experts' prior knowledge, providing strong support for the automatic construction of ancient painting knowledge graphs.

[0046] (5.3) Structured representation components (see Figure 2The component provides an accurate depiction of the current ancient painting, distinguishing different entity types by node color (blue: human; pink: animal; green: natural landscape; orange: cultural relic), and representing different relationships by line color (grey: location; yellow: event). When editing entities or relationships, the interface will be updated in real time to quickly reflect changes in the state of the graph, ensuring that users can see the effects of their modifications in a timely manner. The component also provides two different layout methods: force-based layout and custom layout to facilitate the exploration and analysis of ancient paintings.

[0047] (5.3.1) The structured representation component contains a triple table, which is used to record various interactive operations, so as to collect user feedback to promote model improvement. Through this table, users can intuitively view the structured information of entities and their relationships, which is convenient for further analysis and optimization. This precise and dynamic structured representation not only improves the user's operating efficiency, but also provides important data support for model optimization, making the construction process of the ancient painting knowledge graph more efficient and reliable. Overall, the structured representation component is the core part of the system, which provides a basis for the effective integration of knowledge extraction and expert feedback.

Claims

1. A method for constructing an ancient painting knowledge graph driven by human-machine fusion, characterized in that: The following steps are involved: Step 1: Construct a classification system for different entities in ancient painting images and establish a dataset labeled corresponding to the classification system; Step 2: Build a knowledge extraction model and use the data set to train the knowledge extraction model; The knowledge extraction model identifies the target ancient painting and obtains prediction features; Step 3: Build a human-in-the-loop model optimization visual analysis tool. The model optimization visual analysis tool fuses the annotation features and the semantic features predicted by the knowledge extraction model with the image features to establish a joint embedding visualization view. The joint embedding visualization view is used to display the difference between the annotation features and the predicted semantic features. Step 4: The user modifies the predicted semantic features displayed by the visual analysis tool according to the differences displayed on the joint embedding visualization view; The modified semantic features are added to the data set, and the knowledge extraction model is iteratively trained again; Step 5: The knowledge extraction model re-identifies the target ancient painting and generates an ancient painting knowledge graph based on the recognition results.

2. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized in that: The classification system includes the types of entities and the relationships between different entities; the relationships between different entities include positional relationships and event relationships.

3. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 2 is characterized in that: The model optimization visual analysis tool includes an uncertainty visualization component; the uncertainty visualization component uses different visual representations to encode entity information and semantic relationships; a scatter plot is used to display entity information, showing model prediction and annotated entity information; the color of each prediction point encodes the semantic label of the entity; the border style of the prediction point is used to distinguish the annotated feature from the predicted feature; The confidence of the predicted label by the bounding box thickness of the predicted point; the semantic relationship is displayed by a graph that records the orientation relationship and event relationship between different entities.

4. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized in that: The model optimization visual analysis tool includes an interactive refining component; the interactive refining component selects entities recognized by the knowledge extraction model and displays predicted semantic information; the user modifies the semantic features through the interactive refining component.

5. The method for constructing ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized by: The interactive refining component modifies semantic features by: 1) modifying entities: the user changes the label of the entity by clicking on the corresponding bounding box; 2) adding entities: the user draws a wireframe and selects a label; 3) deleting entities: the user deletes the bounding box identified by the knowledge extraction model; 4) modifying relationships: the user deletes the relationship line between two entities or modifies the relationship label displayed on the relationship line; 5) adding relationships: the user clicks on entities to add relationship labels.

6. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized in that: The model optimization visual analysis tool includes a structured representation component; the structured representation component contains a triple table; the triple table is used to record interactive operations.

7. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized in that: The process of establishing a joint embedding visualization view is as follows: by calculating the Euclidean distance of the ancient painting feature matrix and adjusting it to minimize the distance between similar label elements; introducing an uncertainty visualization component to display the distance between embedding vectors and vectors related to semantic labels, and encoding them on a two-dimensional plane, using scatter plots and icons to visually design entity and semantic relationships; and generating a visualization view that shows feature similarity by means of data dimensionality reduction.

8. The method for constructing an ancient painting knowledge graph driven by human-machine fusion according to claim 1 is characterized in that: The knowledge extraction model includes a target recognition model and a relationship reasoning model; the target recognition model is used to recognize multiple targets in the ancient painting image according to the classification system constructed in step one; the relationship reasoning model is used to reason about the relationship between different targets.

9. A human-machine fusion driven ancient painting knowledge graph construction system, characterized by: Used to execute the human-machine fusion driven ancient painting knowledge graph construction method as claimed in claim 1; the ancient painting knowledge graph construction system includes a classification identification module, a knowledge extraction module and a visualization analysis module; the classification identification module is used to establish a classification system and a data set in combination with the knowledge of experts; the knowledge extraction module is used to extract knowledge from ancient paintings and obtain the labels of entities in the ancient paintings and the relationships between different entities; The visualization analysis module is used to build a joint embedding visualization view for experts to modify the predicted semantic features.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores computer programs; the processor executes the method for constructing an ancient painting knowledge graph driven by human-machine fusion as described in any one of claims 1-7.

11. A readable storage medium storing a computer program; characterized in that: When the computer program is executed by a processor, it is used to implement the method for constructing an ancient painting knowledge graph driven by human-machine fusion as described in any one of claims 1 to 7.

Images