Image corpus automatic annotation and verification method based on semantic link network

Abstract

The invention provides an image corpus automatic annotation and verification method based on a semantic link network. Accurate annotation and efficient verification are achieved. Extracting multi-granularity visual features of the image, and mapping the multi-granularity visual features to a high-dimensional semantic space to generate dynamic semantic nodes; constructing a hierarchical semantic link network, fusing dynamic semantic nodes, external knowledge graph entities and context information, and generating semantic links; through multi-round iterative optimization, cross-modal semantic association is mined, semantic node representation is updated, and a candidate label set is generated; evaluating semantic consistency and context adaptability of candidate labels by utilizing a dynamic game of a generator and a discriminator, and screening high-confidence label results; through a feedback-driven network updating mechanism, a verification result is injected into a network topology structure, link weights are dynamically adjusted, semantic nodes are expanded, collaborative optimization of a network structure and semantic content is achieved, and the accuracy, robustness and efficiency of image corpus tagging are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image annotation and verification, and in particular to an automatic annotation and verification method for image corpus based on a semantic link network. Background Art

[0002] Automatic annotation and verification of image corpora can provide accurate semantic descriptions for image data, making tasks such as image retrieval, classification, and recognition more efficient and accurate. Through automatic annotation, large amounts of image data can be quickly labeled, greatly reducing the cost and time of manual annotation. At the same time, the verification mechanism can ensure the accuracy and reliability of the annotation results, further improving the performance of the image processing system.

[0003] Although the automatic annotation and verification technology of image corpora has made certain progress in recent years, existing methods still have some limitations when processing complex image content. For example, many methods find it difficult to effectively integrate multi-source heterogeneous data when processing image data, resulting in insufficient accuracy and robustness of the annotation results. In addition, existing technologies have deficiencies in dynamic adaptability and adaptive optimization, making it difficult to maintain stable performance in complex and changing scenarios. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides an automatic annotation and verification method for image corpus based on a semantic link network, which solves the problem of how to achieve accurate annotation and efficient verification of image corpus by constructing a semantic link network with dynamic semantic fusion and context awareness.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for automatic annotation and verification of image corpus based on a semantic link network, comprising:

[0008] S1. Extract multi-granular visual features based on image content and map them into a high-dimensional semantic space aligned with the text semantic space to form dynamic semantic nodes. Dynamic semantic nodes serve as basic network units and, through linking with other nodes, form a semantic network structure, providing semantic information for annotation and verification.

[0009] S2. Construct a hierarchical semantic link network. By integrating dynamic semantic nodes, external knowledge base entities, and contextual information, semantic links are generated. Link weights are combined with semantic similarity and association strength to form a dynamically evolving semantic link network.

[0010] S3. Perform multiple rounds of iterative optimization on the semantic link network to mine potential cross-modal semantic associations between nodes, update the semantic representations of dynamic semantic nodes, reconstruct the weights and directions of semantic links, and generate a set of candidate annotations.

[0011] S4. Construct a dynamic game between the generator and the discriminator to evaluate the semantic consistency and contextual adaptability of the candidate annotation set;

[0012] S5. Through the feedback-driven semantic link network update mechanism, the evaluation results are injected back into the semantic link network, the link weight distribution is dynamically adjusted and the semantic nodes are expanded to achieve the coordinated adaptive optimization of the network structure and semantic content.

[0013] Multi-granularity visual features are extracted through multi-level convolutional neural networks, such as ResNet-50, which performs feature extraction on images. The low-level layers extract basic visual information such as edges and textures, while the high-level layers extract more complex semantic features, such as object contours and scene layouts. For example, when processing an image of a natural scene, the low-level layers extract the texture features of leaves, while the high-level layers recognize the concept of trees. The low-level visual features are deeply integrated with the high-level semantic features. A multi-scale feature fusion layer is added to the multi-convolutional neural network to fuse the feature maps of different levels. The feature maps of the low-level and high-level layers are adjusted to the same resolution through upsampling and downsampling operations, and then element-by-element addition is performed to obtain the fused feature maps. For example, the feature maps of the 1st, 3rd and 5th layers of ResNet-50 are fused to form a multi-scale feature representation. In the feature extraction process, the attention mechanism is used to enhance the expressiveness of the features. The feature maps output by the convolutional layer are weighted through the self-attention mechanism to highlight key features and suppress background noise. For example, in an image containing multiple people, the attention mechanism focuses on the facial features of the people, thereby reducing attention to the background buildings; this mechanism dynamically generates a weight matrix by calculating the correlation between each position in the feature map, thereby strengthening important features; through semantic-aware mapping, multi-granular visual features are mapped to a high-dimensional semantic space; the mapping process considers the semantic information within the image and integrates the entity information of the external knowledge graph; for example, for an image containing cats and dogs, semantic-aware mapping associates these visual features with the cat and dog entities in the knowledge graph to generate richer semantic nodes; a fully connected layer is used to flatten the feature map and map it to a high-dimensional vector space, for example, the feature map is flattened into a vector of length 4096, and then mapped to a high-dimensional semantic space of length 1024 through a fully connected layer; in this process, the mapping weights are dynamically adjusted by fusing the internal semantics of the image and the external knowledge graph information to ensure that the generated dynamic semantic nodes accurately reflect the semantic information of the image content.

[0014] When constructing a semantic link network, a hierarchical structure design is adopted to achieve efficient fusion of image semantic information and external knowledge base, forming a dynamically evolving network topology; the low-level network starts from the original image data and extracts low-level visual features of the image, such as edges, textures, etc., through a multi-level convolutional neural network; these features are weighted fused through the attention mechanism to highlight important feature information; for example, by calculating the correlation between each area of the feature map and the target feature, a higher weight is assigned to the key area; the weighted fused low-level features are mapped to a low-dimensional semantic space to form low-level semantic nodes, which mainly reflect the local semantic information of objects in the image; the semantic link weight calculation of the low-level network is based on the static semantic similarity between nodes, such as the semantic association strength between local features and the main body of the graphic, which is measured by methods such as cosine similarity; the middle-level network receives the low-level semantic nodes generated by the low-level network, and obtains the information from the external The entity information related to these nodes is retrieved from the external knowledge base; the low-level semantic nodes are integrated with the entity information of the external knowledge base to generate mid-level semantic nodes, which have more global semantic expressions; the semantic link weight calculation of the mid-level network depends on the static semantic similarity between nodes, so that the semantic links reflect the dynamic evolution of the semantic relationship; for example, the semantic association strength between images is dynamically adjusted according to the context information, so that the semantic links generated by it accurately reflect the complex semantic relationship of the image content; the high-level network receives the mid-level semantic nodes generated by the middle-level network, and uses cosine similarity to calculate the static semantic similarity between nodes to measure the inherent semantic relevance of the nodes; according to the association strength in the context, the two are combined and the final link weight is calculated by weighted summation, where the weight coefficient is used to balance the contribution of static semantic similarity, thereby achieving further optimization of the link weight between high-level semantic nodes.

[0015] The semantic link network uses the attention mechanism to adjust the weights of feature maps to make semantic alignment more accurate. In each iteration, the semantic representation of the dynamic semantic node is updated by aggregating the semantic information of its neighboring nodes and the contextual information. Each node captures high-order semantic dependencies through multi-hop semantic propagation, with 3 hops set for natural scene images and 5 hops set for medical images, and combines entity information in the external knowledge base to generate a more context-consistent set of candidate annotations. In addition, the hierarchical structure of the semantic link network allows semantic information to be transmitted and fused at different levels. The lower layer captures detailed information through local semantic associations, and the higher layer generates a global semantic representation through global semantic integration, thus forming a dynamically evolving network topology.

[0016] The generator takes the semantic representation of its dynamic semantic nodes and the link weights in the semantic link network as basic inputs. Information is transmitted and fused through the hierarchical structure of the semantic link network. The semantic information between nodes interacts through multi-hop propagation, allowing each node to aggregate the semantic information of its neighboring nodes and thus update its own semantic representation. Semantic information is allowed to be transmitted across multiple nodes in the semantic link network, capturing more complex semantic relationships, such as indirect associations between objects in an image or global semantic information in the context. Finally, the generator generates candidate annotations based on the updated semantic representation.

[0017] The discriminator evaluates the candidate annotations generated by the generator to ensure the rationality and accuracy of the annotation results; the discriminator uses the weight distribution information of dynamic semantic links, which reflects the semantic association strength and contextual dependency between nodes in the semantic link network; the discriminator uses this weight distribution information to calculate the semantic matching degree between the candidate annotations and the context; at the same time, the discriminator also combines the entity relationship constraints in the external knowledge graph to evaluate the rationality of the annotation; for example, if the knowledge graph clearly states that a cat is an animal, and the candidate annotation labels a cat as furniture, the discriminator identifies the irrationality of this annotation through the entity relationship constraints in the knowledge graph.

[0018] After the discriminator has screened out reasonable and accurate annotation results, the annotation quality of the entire set is evaluated by comparing the candidate annotation results with other annotation items in the candidate annotation set; if there is a large difference between the annotation items in the candidate annotation set and the candidate annotation results, it indicates that there are erroneous or unreasonable annotation items in the set; then the association between these annotation results and the nodes and links in the semantic link network is analyzed; first, the semantic nodes and link paths on which the reasonable and accurate annotation results depend, as well as the direct links and multi-hop link paths between them, are traced; the link weights on these paths are considered to have important contributions to the annotation results; according to the semantic consistency information of the annotation results, the weights of these key links are dynamically adjusted, and for those links that are highly correlated with reasonable and accurate annotation results, their weights are increased to strengthen the importance of these semantic associations in subsequent annotations; for example, if the link weight between two entities was originally 0.6, it will be increased to 0.8 after enhanced adjustment.

[0019] (3) Beneficial effects

[0020] The present invention provides a method for automatic annotation and verification of image corpus based on a semantic link network, which has the following beneficial effects:

[0021] 1. This paper uses multi-granularity visual feature mapping and dynamic semantic node generation technology to deeply fuse the low-level visual features of an image with high-level semantic features, generating dynamic semantic nodes that can accurately reflect the semantic information of the image content. This not only improves the accuracy of annotation, but also reduces the annotation errors caused by a single feature, significantly improving the overall quality of image corpus annotation.

[0022] 2. Through hierarchical structure design and multiple rounds of iterative optimization, the present invention realizes the dynamic evolution of semantic link networks and the mining of cross-modal semantic associations; it not only improves the interpretability of semantic links, but also reduces the omission of potential semantic associations, significantly enhancing the robustness and adaptability of image corpus annotation.

[0023] 3. The present invention achieves high-confidence screening of annotation results and adaptive optimization of network structure through the dynamic game between the generator and the discriminator and the feedback-driven semantic link network update mechanism; improves the semantic consistency and context adaptability of the annotation results, reduces the need for manual intervention, and significantly improves the efficiency and reliability of automatic annotation and verification of image corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] In the image feature extraction stage, a multi-level convolutional neural network, such as ResNet-50, is used to extract multi-granularity visual features from images. The low-level layers of the ResNet-50 network are responsible for extracting basic visual information such as edges and textures of the image, while the high-level layers focus on extracting more complex semantic features, such as object contours and scene layouts. For example, when processing an image of a natural scene, the low-level layers can accurately extract the texture features of leaves, while the high-level layers can recognize the concept of a tree. To further enhance the expressive power of features, a self-attention mechanism is used in the feature extraction process. By calculating the correlation between positions in the feature map, a weight matrix is generated to enhance key features, thereby highlighting important information and suppressing background noise. For example, in an image containing multiple characters, the attention mechanism can focus on the facial features of the characters, thereby reducing attention to background buildings. Through semantic perception mapping, Multi-granularity visual features are mapped to a high-dimensional semantic space. The mapping process takes into account the semantic information within the image and also integrates entity information from the external knowledge graph. For example, for an image containing a cat and a dog, semantic-aware mapping associates these visual features with the cat and dog entities in the knowledge graph to generate richer semantic nodes. In the specific implementation, a fully connected layer is used to flatten the feature map and map it to a high-dimensional vector space. For example, the feature map is flattened into a vector of length 4096, and then mapped to a high-dimensional semantic space of length 1024 through a fully connected layer. In this process, the mapping weights are dynamically adjusted by integrating the internal semantics of the image and the external knowledge graph information, so that the dynamic semantic nodes generated by it accurately reflect the semantic information of the image content. The multi-granularity visual features of the image are effectively extracted and mapped to the high-dimensional semantic space, providing a basis for the subsequent construction of the semantic link network.

[0027] When constructing a semantic link network, a hierarchical structure design is adopted to achieve efficient fusion of image semantic information and external knowledge base; the low-level network starts from the original image data and extracts low-level visual features of the image, such as edges, textures, etc., through a multi-level convolutional neural network; these features are weightedly fused through the attention mechanism to highlight important feature information; the weighted fused low-level features are mapped to a low-dimensional semantic space to form low-level semantic nodes, which mainly reflect the local semantic information of objects in the image; the semantic link weight calculation of the low-level network is based on the static semantic similarity between nodes, such as the semantic association strength between local features and the main body of the graphic, which is measured by methods such as cosine similarity; the middle-level network receives the low-level semantic nodes generated by the low-level network and retrieves entity information related to these nodes from the external knowledge base; the low-level semantic nodes are fused with the entity information of the external knowledge base to generate middle-level semantic nodes, which are more Global semantic expression; the calculation of semantic link weights in the middle-level network relies on the dynamic adjustment of the static semantic similarity between nodes, so that the semantic links reflect the dynamic evolution of semantic relationships; for example, the semantic association strength between images is dynamically adjusted according to the context information, so that the semantic links generated by them accurately reflect the complex semantic relationships of the image content; the high-level network receives the middle-level semantic nodes generated by the middle-level network, and uses cosine similarity to calculate the static semantic similarity between nodes to measure the inherent semantic relevance of the nodes; according to the association strength in the context, the two are combined and the final link weight is calculated by weighted summation, where the weight coefficient is used to balance the contribution of static semantic similarity, thereby achieving further optimization of the link weights between high-level semantic nodes; through this hierarchical structure design, the semantic information inside and outside the image is effectively integrated to form a dynamically evolving semantic link network, which provides support for the automatic annotation and verification of image corpus.

[0028] After the semantic link network is constructed, it is optimized through multiple rounds of iterative optimization to mine potential cross-modal semantic associations between nodes. In each iteration, the semantic representation of the dynamic semantic node is updated by aggregating the semantic information of its neighboring nodes and the contextual information. Each node captures high-order semantic dependencies through multi-hop semantic propagation and combines entity information in the external knowledge base to generate a more context-consistent set of candidate annotations. Attention weights are generated by calculating the correlation between visual features and textual semantic features, thereby enhancing the mining of cross-modal semantic associations. It also reduces the omission of potential semantic associations and significantly enhances the robustness and adaptability of image corpus annotation.

[0029] In order to evaluate the semantic consistency and contextual adaptability of the candidate annotation set, a dynamic game mechanism between the generator and the discriminator is constructed; the generator generates a list of candidate annotations to be verified through semantic enhancement based on the candidate annotation set; the generator captures high-order semantic dependencies through multi-hop semantic propagation, thereby generating candidate annotations with more contextual consistency; the discriminator combines the constraints of the candidate annotation set with the external knowledge base, calculates the semantic matching between the candidate annotations and the context through the weight distribution information of the dynamic semantic links, and evaluates the rationality and correctness of the annotations based on the entity relationship constraints in the external knowledge base; the discriminator generates an evaluation score by screening the candidate annotation set and the matching degree between the candidate annotations and the entity relationships in the external knowledge base; the generator adjusts the generation strategy according to the evaluation score of the discriminator to generate more reasonable candidate annotations; the semantic consistency and contextual adaptability of the annotation results are improved, the need for manual intervention is reduced, and the efficiency and reliability of automatic annotation and verification of image corpus are significantly improved.

[0030] After the discriminator has screened out reasonable and accurate annotation results, the annotation quality of the entire set is evaluated by comparing the candidate annotation results with other annotation items in the candidate annotation set; if there is a large difference between the annotation items in the candidate annotation set and the candidate annotation results, it indicates that there are erroneous or unreasonable annotation items in the set; then the association between these annotation results and the nodes and links in the semantic link network is analyzed; first, the semantic nodes and link paths on which the reasonable and accurate annotation results depend, as well as the direct links and multi-hop link paths between them, are traced; the link weights on these paths are considered to have important contributions to the annotation results; according to the semantic consistency information of the annotation results, the weights of these key links are dynamically adjusted, and for those links that are highly correlated with reasonable and accurate annotation results, their weights are increased to strengthen the importance of these semantic associations in subsequent annotations; for example, if the link weight between two entities was originally 0.6, it will be increased to 0.8 after enhanced adjustment.

[0031] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic annotation and verification of image corpus based on semantic link network, characterized in that: Construct a semantic link network with dynamic semantic fusion and context-awareness, including: S1. Extract multi-granular visual features based on image content and map them into a high-dimensional semantic space aligned with the text semantic space to form dynamic semantic nodes. Dynamic semantic nodes serve as basic network units and, through linking with other nodes, form a semantic network structure, providing semantic information for annotation and verification. S2. Construct a hierarchical semantic link network. By integrating dynamic semantic nodes, external knowledge base entities, and contextual information, semantic links are generated. Link weights are combined with semantic similarity and association strength to form a dynamically evolving semantic link network. S3. Perform multiple rounds of iterative optimization on the semantic link network to mine potential cross-modal semantic associations between nodes, update the semantic representations of dynamic semantic nodes, reconstruct the weights and directions of semantic links, and generate a set of candidate annotations. S4. Construct a dynamic game between the generator and the discriminator to evaluate the semantic consistency and contextual adaptability of the candidate annotation set; S5. Through the feedback-driven semantic link network update mechanism, the evaluation results are injected back into the semantic link network, the link weight distribution is dynamically adjusted and the semantic nodes are expanded to achieve the coordinated adaptive optimization of the network structure and semantic content.

2. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 1, characterized in that: The multi-granularity visual features refer to visual information of different levels and scales extracted from the image. By constructing a hierarchical semantic perception mechanism, the low-level visual features of the image are deeply integrated with the high-level semantic features to generate dynamic semantic nodes. By combining multi-level convolution with the attention mechanism, multi-level and multi-scale visual features are extracted from the image. The extracted visual features are mapped to a high-dimensional semantic space, and the weights of the feature mapping are dynamically adjusted by fusing the internal semantics of the image with the entity information of the external knowledge base.

3. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 2, characterized in that: The dynamic semantic node serves as the basic unit of the semantic link network and forms a semantic network structure by linking with other nodes. The dynamic semantic node is generated by mapping the extracted multi-granularity visual features to a high-dimensional semantic space, and dynamically adjusting the feature mapping weights by combining the internal semantics of the image with the entity information of the external knowledge base to generate dynamic semantic nodes that reflect the semantic information of the image content. The dynamic semantic node is associated with the entities in the external knowledge base through the link weight, providing semantic information support for the subsequent construction of the semantic link network.

4. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 1, characterized in that: The semantic link network is constructed through the design of a hierarchical structure. Dynamic semantic nodes are mapped through multi-granularity visual features to form node representations in a high-dimensional semantic space. These nodes not only contain semantic information of image content, but are also associated with entities in an external knowledge base through link weights, thereby introducing semantic support of external knowledge; contextual information is captured through the link relationship of dynamic semantic nodes, and semantic links are generated by combining semantic similarity and association strength; the calculation of link weights depends on the static semantic similarity between nodes, so that semantic links reflect the dynamic evolution of semantic relationships.

5. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 1, characterized in that: The multi-round iterative optimization of the semantic link network is based on the hierarchical structure of the semantic link network. It uses the network's multi-layer message passing mechanism to capture local and global semantic associations between nodes. Through the cross-modal attention mechanism, it adjusts the alignment between visual features and textual semantic features, thereby mining the potential semantic associations between image content and external knowledge base entities. In each iteration, the semantic representation of a dynamic semantic node is updated by aggregating the semantic information of its neighboring nodes and contextual information to generate a set of candidate annotations.

6. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 1, characterized in that: The generator generates candidate annotations based on the semantic representation and link weights of dynamic semantic nodes by introducing a context-aware semantic generation strategy. It relies on direct semantic associations between nodes, captures high-order semantic dependencies through multi-hop semantic propagation, and generates candidate annotations that are more context-consistent. The discriminator combines the candidate annotation set with the constraints of the external knowledge base, calculates the semantic matching degree between the candidate annotations and the context through the weight distribution information of dynamic semantic links, and evaluates the rationality and correctness of the annotations in combination with the entity relationship constraints in the external knowledge base.

7. The method for automatic annotation and verification of image corpus based on semantic link network according to claim 1, characterized in that: The feedback-driven semantic link network update mechanism injects the verification results back into the network topology structure; based on the evaluation information of the annotation results, it analyzes the relationship between the annotation results and the nodes and links in the semantic link network, and dynamically adjusts the link weight distribution.

Images