Image automatic labeling method and system based on landscape element knowledge graph and visual matching

Abstract

A method and system for automated image annotation based on a landscape element knowledge graph and visual matching is disclosed. The method includes: constructing a landscape element knowledge graph; performing structured association modeling of landscape nodes, landscape element nodes, image nodes, and cultural knowledge nodes; extracting relevant landscape nodes from the graph based on user-inputted images and shooting locations to limit the range of elements to be identified; extracting features from the input image using a zero-shot visual matching model and comparing them with image samples of candidate landscape elements to obtain identification results; extracting attributes and associated cultural knowledge content from the graph based on the identified landscape element nodes; semantically fusing the identification results with cultural knowledge according to preset rules to generate structured or natural language annotation text; and outputting the annotation text to a terminal for display. This invention combines geographic location, knowledge graph, and visual matching to achieve accurate, information-rich, and culturally profound automated annotation of landscape images.

Description

Technical Field

[0001] This invention belongs to the fields of digital media technology, spatial information processing and intelligent image understanding, and in particular relates to an automated image annotation method and system based on the fusion of landscape element knowledge graph (SKG) and zero-shot visual matching model, which is applicable to application scenarios such as tourism image understanding, landscape recognition and cultural semantic generation. Background Technology

[0002] With the development of social media, a large number of tourists share travel photos on online platforms. However, due to varying levels of landscape knowledge among tourists, the vast majority of these images lack accurate and structured textual descriptions. Existing visual models perform well on general object recognition tasks, but struggle to identify landscape elements with regional cultural semantics and lack effective alignment mechanisms with landscape knowledge systems. Furthermore, existing landmark recognition systems can only identify typical, famous landscapes, failing to distinguish numerous highly similar landscape elements, such as certain buildings, bridges, or decorations within a scenic spot. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides an automated image annotation method and system based on landscape element knowledge graph and visual matching. It constructs a landscape element knowledge graph (SKG) and uses it as a semantic constraint and matching enhancement mechanism. It integrates a zero-shot visual matching model to improve the ability to recognize and generate images of landscape elements. By combining the knowledge graph and geographic location, it achieves accurate, information-rich and professionally in-depth automated annotation of tourism images.

[0004] The technical solution adopted by this invention to solve its technical problem is: An automated image annotation method based on landscape element knowledge graph and visual matching, the method comprising the following steps: S1, construct a knowledge graph of landscape elements, and perform structured organization and relational modeling of landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes; S2 receives the landscape element image input by the user and obtains the latitude and longitude positioning information of its shooting location; based on the positioning information, it extracts landscape nodes related to the current location from the landscape element knowledge graph and limits the set range of landscape elements to be identified. S3: Input the landscape element images input by the user into the zero-sample visual matching model for extraction and transformation, and compare the features with the landscape element image samples associated with the candidate landscape nodes to obtain the landscape element recognition results. S4. Based on the landscape element nodes obtained by identification and matching, extract their node attribute information and cultural knowledge node content from the knowledge graph. S5, based on preset text generation rules, performs semantic fusion on the matched recognition results and cultural knowledge to generate landscape element annotation text in structured or natural language form; S6 will ultimately output and display the labeled text on the terminal device, realizing the automatic recognition and cultural semantic annotation method of landscape element images.

[0005] An automated image annotation system based on landscape element knowledge graph and visual matching, the system comprising: The landscape element knowledge graph module is used to construct a landscape element knowledge graph, and to structure and model the relationships between landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes. The positioning module is used to obtain the real-time latitude and longitude information of the user device and to attach the latitude and longitude information to the attributes of the captured photo by fusing multi-source positioning data. The knowledge graph query module receives the latitude and longitude positioning information of the photo from the positioning module, sends a query request to the constructed landscape element knowledge graph, obtains the collection information of all landscape elements around the area through a distance algorithm, including the name, type, attributes, image of the element, and spatial relationship data of the landscape element, and passes this information to the landscape element extraction and matching module. The landscape element extraction and matching module receives image data captured by the user. First, it preprocesses the image using a target detection algorithm, segmenting independent and typical visual element regions from the visual subject of the image while ignoring background noise. After obtaining one or more visual element regions, it extracts image features from each region to generate a high-dimensional visual feature vector. Finally, it uses a landscape element knowledge graph module to transmit a list of landscape element images. C P Using the reference set, the generated feature vectors of each visual element region are... Similarity is calculated between the image and the pre-stored feature vectors of landscape elements in the reference library. The calculation determines whether the similarity between the feature vector of a certain visual element region and the feature vector of a certain landscape element image in the reference library exceeds a preset threshold. If the visual element matches the landscape element successfully, and the matching degree of multiple elements in the same image is the highest in a certain landscape reference set, then the specific landscape element in the user's image is identified. The knowledge content acquisition module is used to send a query request to the landscape element knowledge graph again based on the identified landscape element name to obtain the cultural knowledge node corresponding to the landscape element, including basic attribute information, historical background, cultural connotation and tour tips; the acquired knowledge content is structured and organized according to the logical hierarchy of "basic information-in-depth knowledge-practical suggestions" to form a standardized knowledge content set; The text generation module is used to dynamically construct different prompt word templates based on the structured data provided by the knowledge content acquisition module and combined with the user-specified annotation purpose. The prompt words are input into the LLM large language model to generate customized annotated text with different styles and lengths.

[0006] Furthermore, in the landscape element knowledge graph module, Landscape node D records the landscape name, landscape description, and location latitude and longitude. By transmitting latitude and longitude, it can be matched with geographical location nodes to determine whether it is within the spatial content of that landscape location. Landscape element node Y encompasses scenic spot elements with specific visual information. The landscape element node information includes landscape type, appearance, material composition, and functional attribute description. Landscape element image node P is an image node captured with the landscape as the main object, including the image file name, file type, geographical coordinates of the device at the time of image capture, and the time of capture; Cultural knowledge node W expresses the construction techniques, cultural connotations, and philosophical implications behind the appearance of buildings, components, and patterns. It includes types, names, and content, integrates local chronicles, historical legends, and philosophical metaphors to construct cultural contextual knowledge. Its content includes the auspicious meanings behind garden patterns. The landscape element knowledge graph is formally defined using quadruples: SKG = (D, Y, P, W, R), where D, Y, P, and W represent four types of core entity nodes, and R is a set of relations.

[0007] Preferably, the landscape node D = {d1, d2, ..., d...} n} represents a landscape area entity with concrete latitude and longitude coordinates and a clear spatial concept, satisfying Landscape element node Y = {y1, y2, ..., y n} Represents the material entities subordinate to the landscape, including architectural components and decorative physical elements, satisfying Landscape element image node P = {p1, p2, ..., p n The node carries visual information about the appearance of the material and is associated with nodes D and Y through the descriptions relationship; the cultural knowledge node W = {w1, w2, ..., w n Encoding intangible cultural attributes, including historical materials, legends, and semantic knowledge; The relation set R defines the semantic associations between nodes: ; The graph constructs a hierarchical semantic structure using the OWL ontology language and stores knowledge fragments in the form of RDF triples.

[0008] The distance algorithm in the knowledge graph query module follows these steps: 3.1 First, obtain the n landscape nodes closest to the location point from the map. Then, calculate the distance from the location point to each landscape node. The location point loc consists of latitude and longitude. loc = ( lat, lng ),set up dist i = dist(loc, d i ) Locating point from landscape node d i The distance; 3.2 Sort all landscape nodes in ascending order of distance to obtain a sorted list. D sorted， in It is the nearest node; ; satisfy ; 3.3 Initialize the candidate landscape node list C D If the node is empty, start from the nearest node and consider whether to add it to each node in turn. C D ,until C D It contains up to 8 nodes; For each node, the condition is as follows: a. First, the nearest node d 1. Add directly C D ; b. For subsequent nodes d i ,if C D If there are already 8 nodes, then stop. c. Otherwise, calculate d (i) Distance from the positioning point dist i ,as well as C D Each node d j Distance from the positioning point dist j Find the smallest dist j That is, the current C D The closest distance; The conditions are as follows: Condition 1: Current node d idistance dist i and C D The difference in distance to the nearest node is less than 30% of, that is: dist i - <0.3* ; Condition 2: Current node d i The azimuth of the line connecting the positioning point and the positioning point. C D The angle between the azimuth of each existing node and the positioning point is greater than or equal to 45 degrees. Calculated using latitude and longitude: ; in Let be the angle between two azimuth angles, converted to the range [0, 360), satisfying... ; If both condition 1 and condition 2 are met, then... d i join in C D ; definition , , ; for ,when hour; ; The final candidate set is as follows: ; 3.4 Obtain the list of candidate landscape nodes Afterwards, based on Find all relationships with The landscape element nodes associated with the middle node constitute the candidate landscape element list. C Y ,Right now: ; 3.5 Then, through Find all relationships with C Y The landscape element image nodes associated with the central node constitute a candidate landscape element image list. C P .Right now: .

[0009] In the landscape element extraction and matching module, the image feature matching process is as follows: 4.1 Calculate the feature similarity matrix. Input n landscape element image features extracted from user photos: ,in A d-dimensional feature vector; a list of candidate landscape element images: Each image node Pre-extracted features: ; Construct a similarity matrix : ; in, This represents the vector dot product operation. This represents the L2 norm (Euclidean length) of a vector. Let represent the cosine similarity between the i-th user feature and the j-th candidate image feature, with a value range of [-1, 1].

[0010] 4.2 Calculate the landscape node consistency score. Define the landscape node-image node mapping function: ,in This indicates that image node p belongs to landscape node d; For each landscape node The subset of image nodes associated with it is: ; For each landscape node Calculate the consistency score: ; in, Indicates the i-th user feature in the landscape The highest similarity score, The minimum acceptable threshold, This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. The score represents the best matching similarity that each user feature contributes to in the current landscape, but it is only included in the total score when the similarity reaches a threshold. 4.3 Determine the best matching landscape node: Select the landscape node with the highest score: ; The verification conditions are: ; in, This is the threshold for the total landscape matching score. If the threshold is not met, the matching is considered to have failed.

[0011] 4.4 Matching Image Nodes: For each user feature At the best landscape nodes A subset of images Find the best match: Calculate the best match similarity: ; Obtain the corresponding image node: ; Apply individual matching thresholds: like (Recommended value: 0.6) If the match is found, accept the match; otherwise, if there are no matching image nodes for this user feature, output the set of matching images. ; 4.5 Obtaining the corresponding landscape element nodes: Using the `depicts` relationship in the knowledge graph, obtain the landscape element nodes corresponding to the matching images: ; Where depicts is a binary relation in the relation set R, indicating that image node p describes landscape element node y; 4.6 Output the final matching result: Best matching landscape node The set of matching landscape element nodes The set of matched image nodes The similarity of each match .

[0012] The beneficial effects of this invention are mainly reflected in: 1. This invention uses real-time geographic location information from image capture as auxiliary information for rapid localization of landscape element knowledge graphs, and constrains the image recognition process within a spatial range, establishing a correspondence between the recognized object and its actual geographic location. Furthermore, visual matching technology is used to perform refined comparison of local features, improving the accuracy of fine-grained landscape element recognition. By integrating semantic relationships in the landscape element knowledge graph with visual features from existing landscape element image node datasets, automated annotation of atypical landscapes and their constituent elements is achieved, effectively expanding the application scope of landmark recognition and improving the annotation robustness and system scalability in complex scenarios. 2. This invention introduces a knowledge graph of landscape elements into the process of landscape element image recognition and semantic understanding, realizing the integrated processing of tourism landscape recognition and cultural semantic generation. Through the recognition of landscape elements, the images can accurately reflect their regional attributes and cultural differences. 3. This invention integrates a zero-shot visual matching model with a knowledge graph semantic reasoning mechanism. Compared to image annotation methods that rely solely on fixed model classification or manually written rules, it offers greater flexibility and generalization capabilities. The zero-shot visual model can achieve landscape recognition without requiring a large number of labeled samples, significantly reducing the cost of relying on manually trained data. Simultaneously, the knowledge graph can be continuously expanded and updated; new landscape entities, cultural semantic information, and landscape element images can be added to the system at any time without requiring significant modifications to the algorithm structure, thus achieving dynamic expansion of the recognition range and semantic content. Therefore, this invention not only improves the intelligence level of image annotation but also enhances the system's adaptability and scalability in multi-scenario and multi-regional applications. 4. Based on landscape recognition, this invention further realizes the automated generation of cultural semantic annotations, enabling landscape element images to go beyond mere objective recognition and instead combine historical, cultural, and spatial contextual knowledge to form interpretive textual expressions. This enhances the content quality and expressive depth of tourism images, guide systems, and cultural dissemination carriers, demonstrating significant application value and social benefits. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the node structure of the Landscape Element Knowledge Graph (SKG) in this invention; Figure 2 This is a flowchart illustrating the automated image annotation method for landscape elements provided by the present invention; Figure 3 This is a schematic diagram of the structure of the landscape element image automated annotation generation system provided by the present invention; Detailed Implementation

[0014] The present invention will now be further described with reference to the accompanying drawings.

[0015] Reference Figures 1-3 An automated image annotation method based on landscape element knowledge graph and visual matching, the method includes the following steps: S1, construct a knowledge graph of landscape elements, and perform structured organization and relational modeling of landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes; S2 receives the landscape element image input by the user and obtains the latitude and longitude positioning information of its shooting location; based on the positioning information, it extracts landscape nodes related to the current location from the landscape element knowledge graph and limits the set range of landscape elements to be identified. S3: Input the landscape element images input by the user into the zero-sample visual matching model for extraction and transformation, and compare the features with the landscape element image samples associated with the candidate landscape nodes to obtain the landscape element recognition results. S4. Based on the landscape element nodes obtained by identification and matching, extract their node attribute information and cultural knowledge node content from the knowledge graph. S5, based on preset text generation rules, performs semantic fusion on the matched recognition results and cultural knowledge to generate landscape element annotation text in structured or natural language form; S6 will ultimately output and display the labeled text on the terminal device, realizing the automatic recognition and cultural semantic annotation method of landscape element images.

[0016] The automated annotation method for landscape element images provided in this embodiment is executed by an automated annotation generation system for landscape element images. The following explanation will take a mobile app as the client and a cloud system as the server as an example.

[0017] First, perform step S1 to pre-construct a landscape element knowledge graph (SKG) that integrates spatial, visual, and cultural attributes. The process is as follows: 1.1. Landscape description texts are collected from authoritative websites, encyclopedia websites, and literature databases using landscape keywords and converted into landscape nodes. Then, the names of landscape elements and related historical and cultural information are extracted and converted into landscape element nodes and cultural knowledge nodes, and the relationships between the nodes are established. Finally, landscape element images are collected from the Internet using a list of landscape element names and converted into landscape element image nodes, and the relationships between them are established. The latitude and longitude of the landscape nodes are batch constructed using a geographic information interface.

[0018] 1.2 Landscape nodes, landscape element nodes, landscape element image nodes, cultural knowledge nodes, and their relationships are imported into the Neo4j database using Python code to construct a knowledge graph. This paper uses Neo4j version 5.8.0 on iOS and calls py2neo to manipulate Neo4j, thus importing the relevant data into the Neo4j database.

[0019] For example, in Wuzhen Scenic Area, Jiaxing City, Zhejiang Province, a knowledge graph of Wuzhen landscape elements was constructed. The cross-shaped inland waterway system divides the town into four blocks: East Gate, South Gate, West Gate, and North Gate. East Gate contains cultural sites such as Mao Dun's Former Residence and the Jiangnan Hundred Beds Museum; while West Gate features landscapes such as Zhaoming Academy, General Wu's Temple, Caomu Bense Dyehouse, and the landmark White Lotus Pagoda. For example, the spatial attributes of "Zhaoming Academy" are "Wuzhen - Wuzhen West Gate Scenic Area - Historical Building - Zhaoming Academy" and "Latitude and Longitude Information: Longitude: 120.4859, Latitude: 30.7492". Landscape element nodes include "Library, Lecture Hall, Plaque, Head of Prince Xiao Tong of the Southern Liang Dynasty, Stele, Courtyard Garden". Landscape element image nodes are collected through on-site photography and web crawling. Cultural knowledge nodes include textual content in four aspects: historical background, aesthetic craftsmanship, philosophical connection, and symbolic meaning.

[0020] In step S2, the positioning module in the app obtains the latitude and longitude and marks it in the attributes of the captured photo. The process is as follows: When the user takes a photo using the app (such as the indoor scene of Sanwei Bookstore), when the user enables the image capture and annotation function, the system automatically activates the real-time positioning module to obtain the current latitude and longitude coordinates loc=(lat,lng).

[0021] In step S3, the zero-shot visual feature extraction submodule processes the image and submits it to the knowledge graph query module. The process is as follows: The user selects a photo and a content template (e.g., posting to WeChat Moments). The app uses the zero-shot visual feature extraction submodule to extract visual elements from the photo image, segmenting independent and typical visual element regions from the visual subject of the image while ignoring background noise and other information. After obtaining one or more visual element regions (e.g., a photo containing both "Sanwei Study Plaque" and "Sika Deer Painting"), image features are extracted for each region separately (e.g., CLIP-ViT-L / 14). The image feature values ​​of the two visual elements, "Sanwei Study Plaque" and "Sika Deer Painting," along with the latitude and longitude of the photo, are then uploaded to the server system.

[0022] In step S4, the knowledge graph query module uses geographic information and landscape element knowledge graph to obtain a list of candidate landscape element images. The process is as follows: the knowledge graph query module of the server system locates the corresponding "Shaoxing-Lu Xun's Hometown" based on the latitude and longitude of the uploaded photo, and further searches for landscape nodes within a certain range of latitude and longitude (gradually in a circular area centered on the location), filters out the candidate landscape nodes that may match (Sanwei Study, Lu Xun's Former Residence, Lu Xun Memorial Hall), and obtains a list of candidate landscape element images according to "landscape → landscape element → landscape element image".

[0023] In step S5, the image feature matching submodule matches landscape elements in the candidate landscape element image list. If a match is successful, it submits a list of landscape element names to the knowledge content acquisition module. The knowledge content acquisition module then sends a query request to the landscape element knowledge graph and returns the associated cultural knowledge node information to the text generation module. The cloud system uses the image feature matching submodule to perform image feature similarity matching on each uploaded visual element feature value and the image images in the candidate landscape element image list; The image feature matching submodule calculates the cosine similarity between the feature vectors of the "Sanwei Study Plaque" and the "Sika Deer Painting" and the pre-stored image feature vectors of the landscape element images associated with "Sanwei Study," "Lu Xun's Former Residence," and "Lu Xun Memorial Hall" returned by the knowledge graph. Feature value one has a similarity of 1 to the image of the "Sanwei Study Plaque." ( The similarity between feature value two and the image of the "sika deer hanging painting" is . ( The matching results were verified to be consistent, and the landscape elements in the image were finally identified as "Sanwei Study - Sanwei Study Plaque" and "Sanwei Study - Sika Deer Painting".

[0024] If the match is successful, the knowledge content acquisition module submits the landscape element attributes of "Sanwei Study - Sanwei Study Plaque" and "Sanwei Study - Sika Deer Painting" to the landscape element knowledge graph module, which then searches for related cultural knowledge nodes in the knowledge graph and provides them to the text generation module. If the match fails, a failure message is returned to the app.

[0025] In step S6, the text generation module generates labeled content based on the content template selected by the user and the associated cultural knowledge node information. To enable the DeepSeek-V3 model to generate accurate and applicable landscape annotation text, this study develops a multi-purpose text generation rule system and transforms it into a standardized constraint-based cue word framework. This framework integrates multiple types of node attributes from the landscape knowledge graph to ensure that the generated content is both accurate and practical. Users can choose their text annotation purpose through the interactive interface, with three main text styles: social sharing, academic research, and tourist guides. For example, after selecting the "social sharing" annotation purpose, the text generation module inputs structured knowledge and the "social sharing" purpose rules into the DeepSeek-V3 model.

[0026] An automated image annotation system based on landscape element knowledge graph and visual matching, the system comprising: The landscape element knowledge graph module is used to construct a landscape element knowledge graph, and to structure and model the relationships between landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes. A Spatial Knowledge Graph (SKG) is a knowledge representation model that organizes spatial entities and their image relationships using a graph structure. The SKG consists of landscape nodes, landscape element nodes, landscape element image nodes, and cultural knowledge nodes. Landscape nodes (D, representing a landscape area with latitude and longitude coordinates and a clear spatial concept), landscape element nodes (Y, representing material entities belonging to the landscape, such as buildings, landscape features, and decorations), landscape element image nodes (P, images carrying information about the physical appearance of the objects), and cultural knowledge nodes (W, carrying cultural attributes). In the landscape element knowledge graph module, landscape node D records the landscape name, landscape description, and location latitude and longitude. By transmitting latitude and longitude, it can be matched with geographical location nodes to determine whether it is in the spatial content of that landscape location. Landscape element node Y encompasses scenic spot elements with specific visual information. The landscape element node information includes landscape type, appearance, material composition, and functional attribute description. Landscape element image node P is an image node captured with the landscape as the main object, including the image file name, file type, geographical coordinates of the device at the time of image capture, and the time of capture; Cultural knowledge node W expresses the construction techniques, cultural connotations, and philosophical implications behind the appearance of buildings, components, and patterns. It includes types, names, and content, integrates local chronicles, historical legends, and philosophical metaphors to construct cultural contextual knowledge. Its content includes the auspicious meanings behind garden patterns. The landscape element knowledge graph is formally defined using quadruples: SKG = (D, Y, P, W, R), where D, Y, P, and W represent four types of core entity nodes, and R is a set of relations.

[0027] In this embodiment, the landscape node D = {d1, d2, ..., d} n} represents a landscape area entity with concrete latitude and longitude coordinates and a clear spatial concept, such as "Wuzhen Xiuzhen Temple" or "Wuzhen General Wu Temple," which satisfies... Landscape element node Y = {y1, y2, ..., y} n} Represents the material entities subordinate to the landscape, including physical elements such as architectural components and decorations, satisfying Landscape element image nodes P = {p1, p2, ..., p n The cultural knowledge node W carries visual information about the material's appearance and is associated with the D and Y nodes through the `depicts` relation. The cultural knowledge node W = {w1, w2, ..., w...} n Encoding intangible cultural attributes, including semantic knowledge such as historical materials and legends.

[0028] The relation set R defines the semantic associations between nodes: .

[0029] The graph constructs a hierarchical semantic structure using the OWL ontology language and stores knowledge fragments in the form of RDF triples. For example, for the Wuzhen Xiuzhen Temple scene, semantic associations such as (d_Xiuzhen Temple, hasElement, y_Dongyue Hall), (p_Hall Photograph, depictions, y_Dongyue Hall), and (d_Xiuzhen Temple, hasCulturalAttribute, w_Xiuzhen History) are constructed.

[0030] The positioning module is used to obtain the real-time latitude and longitude information of the user device and to attach the latitude and longitude information to the attributes of the captured photo by fusing multi-source positioning data. The knowledge graph query module receives the latitude and longitude positioning information of the photo from the positioning module, sends a query request to the constructed landscape element knowledge graph, obtains the collection information of all landscape elements around the area through a distance algorithm, including the name, type, attributes, image of the element, and spatial relationship data of the landscape element, and passes this information to the landscape element extraction and matching module. The distance algorithm in the knowledge graph query module follows these steps: 3.1 First, obtain the n closest landscape nodes to the location point from the map. Then, calculate the distance from the location point to each landscape node using a spherical distance formula, such as the Haversine formula. dist i = dist(loc, d i ) Locating point from landscape node d i The distance; 3.2 Sort all landscape nodes in ascending order of distance to obtain a sorted list. D sorted， in It is the nearest node; ; satisfy ; 3.3 Initialize the candidate landscape node list C D If the node is empty, start from the nearest node and consider whether to add it to each node in turn. C D ,until C D It contains up to 8 nodes; For each node, the condition is as follows: a. First, the nearest node d 1. Add directly C D ; b. For subsequent nodes d i ,if C D If there are already 8 nodes, then stop. c. Otherwise, calculate d i Distance from the positioning point dist i ,as well as C D Each node d j Distance from the positioning point dist j Find the smallest dist j That is, the current C D The closest distance; The conditions are as follows: Condition 1: Current node d i distance dist i and C D Distance to the nearest node (i.e., the first node) distance The difference is less than 30% of, that is: dist i - <0.3* ; Condition 2: Current node d i The azimuth of the line connecting the positioning point and the positioning point. C D The angle between the azimuth of each existing node and the positioning point is greater than or equal to 45 degrees. Calculated using latitude and longitude: ; in Let be the angle between two azimuth angles, converted to the range [0, 360), satisfying... ; If both condition 1 and condition 2 are met, then... d i join in C D ; definition , , ; for ,when hour; ; The final candidate set is as follows: ; 3.4 Obtain the list of candidate landscape nodes Afterwards, based on Find all relationships with The landscape element nodes associated with the middle node constitute the candidate landscape element list. C Y ,Right now: ; 3.5 Then, through Find all relationships with C Y The landscape element image nodes associated with the central node constitute a candidate landscape element image list. C P .Right now: .

[0031] The landscape element extraction and matching module receives image data captured by the user. First, it preprocesses the image using a target detection algorithm, segmenting independent and typical visual element regions from the visual subject of the image while ignoring background noise. After obtaining one or more visual element regions, it extracts image features from each region to generate a high-dimensional visual feature vector. Finally, it uses a landscape element knowledge graph module to transmit a list of landscape element images. C P Using the reference set, the generated feature vectors of each visual element region are... Similarity is calculated between the image and the pre-stored feature vectors of landscape elements in the reference library. The calculation determines whether the similarity between the feature vector of a certain visual element region and the feature vector of a certain landscape element image in the reference library exceeds a preset threshold. If the visual element matches the landscape element successfully, and the matching degree of multiple elements in the same image is the highest in a certain landscape reference set, then the specific landscape element in the user's image is identified. The landscape element extraction and matching module is one of the core modules of the model. Its design aims to solve the common problems of multi-element extraction and background interference in scenic scenes. It mainly includes a zero-sample visual feature extraction submodule and an image feature matching submodule.

[0032] Zero-shot visual feature extraction submodule: After receiving image data captured by the user, considering the possibility of overlapping landscape elements or background interference in the scenic area images, this submodule first preprocesses the image using an object detection algorithm (such as YOLOv8) to segment independent and typical visual element regions from the visual subject of the image while ignoring background noise and other information. After obtaining one or more visual element regions, image feature extraction (such as CLIP-ViT-L / 14) is performed on each region to generate a high-dimensional visual feature vector. This vector can accurately represent the core visual attributes (such as shape, texture, color distribution, etc.) of each visual element in the image. This submodule can extract as many different visual elements in the image as possible while avoiding mutual interference between the features of different visual elements.

[0033] Image Feature Matching Submodule: Uses the landscape element image list passed from the landscape element knowledge graph module. C P As a reference set, the feature vectors of each visual element region generated by the zero-sample visual feature extraction submodule are used. Similarity is calculated between the images and the pre-stored feature vectors of landscape elements in the reference library. The similarity calculation uses a similarity algorithm. If the similarity between the feature vector of a visual element region and the feature vector of a landscape element image in the reference library exceeds a preset threshold, the similarity is calculated. If the visual element matches the landscape element successfully, and the matching degree of multiple elements in the same image is the highest in a certain landscape reference set, then the specific landscape element in the user's captured image is identified.

[0034] The image feature matching process is as follows: 4.1 Calculate the feature similarity matrix. Input n landscape element image features extracted from user photos: ,in A d-dimensional feature vector; a list of candidate landscape element images: Each image node Pre-extracted features: ; Construct a similarity matrix : ; in, This represents the vector dot product operation. This represents the L2 norm (Euclidean length) of a vector. Let represent the cosine similarity between the i-th user feature and the j-th candidate image feature, with a value range of [-1, 1].

[0035] 4.2 Calculate the landscape node consistency score. Define the landscape node-image node mapping function: ,in This indicates that image node p belongs to landscape node d; For each landscape node The subset of image nodes associated with it is: ; For each landscape node Calculate the consistency score: ; in, Indicates the i-th user feature in the landscape The highest similarity score, The minimum acceptable threshold, This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. The score represents the best matching similarity that each user feature contributes to in the current landscape, but it is only included in the total score when the similarity reaches a threshold. 4.3 Determine the best matching landscape node: Select the landscape node with the highest score: ; The verification conditions are: ; in, This is the threshold for the total landscape matching score. If the threshold is not met, the matching is considered to have failed.

[0036] 4.4 Matching Image Nodes: For each user feature At the best landscape nodes A subset of images Find the best match: Calculate the best match similarity: ; Obtain the corresponding image node: ; Apply individual matching thresholds: like (Recommended value: 0.6) If the match is found, accept the match; otherwise, if there are no matching image nodes for this user feature, output the set of matching images. ; 4.5 Obtaining the corresponding landscape element nodes: Using the `depicts` relationship in the knowledge graph, obtain the landscape element nodes corresponding to the matching images: ; Where depicts is a binary relation in the relation set R, indicating that image node p describes landscape element node y; 4.6 Output the final matching result: Best matching landscape node The set of matching landscape element nodes The set of matched image nodes The similarity of each match .

[0037] The knowledge content acquisition module, after the landscape element extraction and matching module outputs the recognition results, sends a query request to the landscape element knowledge graph based on the identified landscape element name to obtain the corresponding cultural knowledge nodes, including basic attribute information (such as construction date and building materials), historical background (such as related figures and stories, and reasons for construction), cultural connotations (such as the expression of garden artistic conception and symbolic meaning), and visitor tips. The module then structures the acquired knowledge content, organizing it according to a logical hierarchy of "basic information - in-depth knowledge - practical suggestions" to form a standardized knowledge content set, providing well-organized input data for the subsequent text generation module.

[0038] The text generation module is the final step in achieving automated and personalized annotation. Its innovation lies in implementing a multi-purpose, adaptive conditional text generation framework. Based on structured data provided by the knowledge content acquisition module, this framework dynamically constructs different prompt templates according to the user-specified annotation purpose (e.g., "professional explanation," "social sharing," "study notes"). For example, for "professional explanation," the prompts emphasize historical depth and linguistic rigor; while for "social sharing," they focus on visual highlights and emotional expression. These carefully designed prompts are input into an LLM large language model (such as DeepSeek-V3), and by adjusting generation parameters such as the temperature coefficient (e.g., Temperature = 0.7, ensuring a balance between accuracy and diversity in the generated text), customized annotated text with varying styles and lengths is generated while ensuring factual accuracy.

[0039] Specifically, the automated image annotation method for landscape elements provided in this embodiment is executed by an automated image annotation generation system for landscape elements. This system consists of a client and a server. The client can be an app or mini-program installed on a user's mobile phone, tablet, or other smart terminal, while the server can be deployed on a cloud server or cloud computing resources. The client and server communicate bidirectionally via the Internet or communication networks, without specific limitations here.

[0040] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. An automated image annotation method based on landscape element knowledge graph and visual matching, characterized in that, The method includes the following steps: S1, construct a knowledge graph of landscape elements, and perform structured organization and relational modeling of landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes; S2 receives the landscape element image input by the user and obtains the latitude and longitude positioning information of its shooting location; based on the positioning information, it extracts landscape nodes related to the current location from the landscape element knowledge graph and limits the set range of landscape elements to be identified. S3: Input the landscape element images input by the user into the zero-sample visual matching model for extraction and transformation, and compare the features with the landscape element image samples associated with the candidate landscape nodes to obtain the landscape element recognition results. S4. Based on the landscape element nodes obtained by identification and matching, extract their node attribute information and cultural knowledge node content from the knowledge graph. S5, based on preset text generation rules, performs semantic fusion on the matched recognition results and cultural knowledge to generate landscape element annotation text in structured or natural language form; S6 will ultimately output and display the labeled text on the terminal device, realizing the automatic recognition and cultural semantic annotation method of landscape element images.

2. A system for implementing the image automatic annotation method based on landscape element knowledge graph and visual matching as described in claim 1, characterized in that, The system includes: The landscape element knowledge graph module is used to construct a landscape element knowledge graph, and to structure and model the relationships between landscape nodes, landscape element nodes, landscape element image nodes and cultural knowledge nodes. The positioning module is used to obtain the real-time latitude and longitude information of the user device and to attach the latitude and longitude information to the attributes of the captured photo by fusing multi-source positioning data. The knowledge graph query module receives the latitude and longitude positioning information of the photo from the positioning module, sends a query request to the constructed landscape element knowledge graph, obtains the collection information of all landscape elements around the area through a distance algorithm, including the name, type, attributes, image of the element, and spatial relationship data of the landscape element, and passes this information to the landscape element extraction and matching module. The landscape element extraction and matching module receives image data captured by the user. First, it preprocesses the image using a target detection algorithm, segmenting independent and typical visual element regions from the visual subject of the image while ignoring background noise. After obtaining one or more visual element regions, it extracts image features from each region to generate a high-dimensional visual feature vector. Finally, it uses a landscape element knowledge graph module to transmit a list of landscape element images. C P Using the reference set, the generated feature vectors of each visual element region are... Similarity is calculated between the image and the pre-stored feature vectors of landscape elements in the reference library. The calculation determines whether the similarity between the feature vector of a certain visual element region and the feature vector of a certain landscape element image in the reference library exceeds a preset threshold. If the visual element matches the landscape element successfully, and the matching degree of multiple elements in the same image is the highest in a certain landscape reference set, then the specific landscape element in the user's image is identified. The knowledge content acquisition module is used to send a query request to the landscape element knowledge graph again based on the identified landscape element name to obtain the cultural knowledge node corresponding to the landscape element, including basic attribute information, historical background, cultural connotation and tour tips; the acquired knowledge content is structured and organized according to the logical hierarchy of "basic information-in-depth knowledge-practical suggestions" to form a standardized knowledge content set; The text generation module is used to dynamically construct different prompt word templates based on the structured data provided by the knowledge content acquisition module and combined with the user-specified annotation purpose. The prompt words are input into the LLM large language model to generate customized annotated text with different styles and lengths.

3. The system as described in claim 2, characterized in that, In the landscape element knowledge graph module Landscape node D records the landscape name, landscape description, and location latitude and longitude. By transmitting latitude and longitude, it can be matched with geographical location nodes to determine whether it is within the spatial content of that landscape location. Landscape element node Y encompasses scenic spot elements with specific visual information. The landscape element node information includes landscape type, appearance, material composition, and functional attribute description. Landscape element image node P is an image node captured with the landscape as the main object, including the image file name, file type, geographical coordinates of the device at the time of image capture, and the time of capture; Cultural knowledge node W expresses the construction techniques, cultural connotations, and philosophical implications behind the appearance of buildings, components, and patterns. It includes types, names, and content, integrates local chronicles, historical legends, and philosophical metaphors to construct cultural contextual knowledge. Its content includes the auspicious meanings behind garden patterns. The landscape element knowledge graph is formally defined using quadruples: SKG = (D, Y, P, W, R), where D, Y, P, and W represent four types of core entity nodes, and R is a set of relations.

4. The system as described in claim 3, characterized in that, Landscape node D = {d1, d2, ..., d n } represents a landscape area entity with concrete latitude and longitude coordinates and a clear spatial concept, satisfying Landscape element node Y = {y1, y2,..., y n } Represents the material entities subordinate to the landscape, including architectural components and decorative physical elements, satisfying Landscape element image node P = {p1, p2, ..., p n The node carries visual information about the appearance of the material and is associated with nodes D and Y through the descriptions relationship; the cultural knowledge node W = {w1, w2, ..., w n Encoding intangible cultural attributes, including historical materials, legends, and semantic knowledge; The relation set R defines the semantic associations between nodes: ； The graph constructs a hierarchical semantic structure using the OWL ontology language and stores knowledge fragments in the form of RDF triples.

5. The system as described in any one of claims 2 to 4, characterized in that, The distance algorithm in the knowledge graph query module follows these steps: 3.1 First, obtain the n landscape nodes closest to the location point from the map. Then, calculate the distance from the location point to each landscape node. The location point loc consists of latitude and longitude. loc = ( lat, lng ),set up dist i = dist(loc, d i ) Locating point from landscape node d i The distance; 3.2 Sort all landscape nodes in ascending order of distance to obtain a sorted list. D sorted， in It is the nearest node; ； satisfy ; 3.3 Initialize the candidate landscape node list C D If the node is empty, start from the nearest node and consider whether to add it to each node in turn. C D ,until C D It contains up to 8 nodes; For each node, the condition is as follows: a. First, the nearest node d 1. Add directly C D ; b. For subsequent nodes d i ,if C D If there are already 8 nodes, then stop. c. Otherwise, calculate d i Distance from the positioning point dist i ,as well as C D Each node d j Distance from the positioning point dist j Find the smallest dist j That is, the current C D The closest distance; The conditions are as follows: Condition 1: Current node d i distance dist i and C D The difference in distance to the nearest node is less than r min 30% of, that is: dist i - r min <0.3* r min ; Condition 2: Current node d i The azimuth of the line connecting the positioning point and the positioning point. C D The angle between the azimuth of each existing node and the positioning point is greater than or equal to 45 degrees. Calculated using latitude and longitude: ； in Let be the angle between two azimuth angles, converted to the range [0, 360), satisfying... ; If both condition 1 and condition 2 are met, then... d i join in C D ; definition , , ; for ,when hour; ； The final candidate set is as follows: ； 3.4 Obtain the list of candidate landscape nodes Afterwards, based on Find all relationships with The landscape element nodes associated with the middle node constitute the candidate landscape element list. C Y ,Right now: ； 3.5 Then, through Find all relationships with C Y The landscape element image nodes associated with the central node constitute a candidate landscape element image list. C P ,Right now: 。 6. The system as described in any one of claims 2 to 4, characterized in that, In the landscape element extraction and matching module, the image feature matching process is as follows: 4.1 Calculate the feature similarity matrix. Input n landscape element image features extracted from user photos: ,in It is a d-dimensional feature vector; Candidate landscape element image list: Each image node Pre-extracted features: ; Construct a similarity matrix : ； in, This represents the vector dot product operation. Describes the L2 norm of a vector. Let represent the cosine similarity between the i-th user feature and the j-th candidate image feature, with a value range of [-1, 1]. 4.2 Calculate the landscape node consistency score. Define the landscape node-image node mapping function: ,in This indicates that image node p belongs to landscape node d; For each landscape node The subset of image nodes associated with it is: ； For each landscape node Calculate the consistency score: ； in, Indicates the i-th user feature in the landscape The highest similarity score, The minimum acceptable threshold, This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. The score represents the best matching similarity that each user feature contributes to in the current landscape, but it is only included in the total score when the similarity reaches a threshold.

7. The system as described in claim 6, characterized in that, The image feature matching process also includes the following: 4.3 Determine the best matching landscape node: Select the landscape node with the highest score: ； The verification conditions are: ； in, This is the threshold for the total landscape matching score; if it is not met, the matching is considered to have failed. 4.4 Matching Image Nodes: For each user feature At the best landscape nodes A subset of images Find the best match: Calculate the best match similarity: ； Obtain the corresponding image node: ； Apply individual matching thresholds: like If the match is found to be correct, the match is accepted; otherwise, if no matching image node is found for the user feature, the set of matching images is output. .

8. The system as described in claim 7, characterized in that, The image feature matching process also includes the following: 4.5 Obtaining the corresponding landscape element nodes: Using the `depicts` relationship in the knowledge graph, obtain the landscape element nodes corresponding to the matching images: ； Where depicts is a binary relation in the relation set R, indicating that image node p describes landscape element node y; 4.6 Output the final matching result: Best matching landscape node The set of matching landscape element nodes The set of matched image nodes The similarity of each match .

Images