Image region labeling method and system based on visual semantic relation graph

Abstract

The application provides an image region labeling method and system based on a visual semantic relation graph, and relates to the technical field of computer vision and image processing.The application obtains a multi-scale region set of a to-be-labeled image and multi-level node features of each region, constructs an initial semantic relation graph containing nodes and initial connection weights based on the multi-level node features, dynamically updates the connection weights based on current node features in an iteration layer of a graph neural network to obtain a dynamic semantic relation graph, then differentiates and aggregates neighbor node features according to the semantic relation categories between nodes by using corresponding strategies, propagates and fuses the features through a hierarchical mapping relationship between coarse-grained and fine-grained regions, and finally determines the semantic labeling results of each region by using the aggregated features and cross-hierarchical fusion features.The application realizes high-precision image region semantic labeling by constructing and dynamically updating the visual semantic relation graph of multi-scale regions, combining differentiated relation-aware feature aggregation and cross-hierarchical feature propagation and fusion.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to an image region annotation method and system based on visual semantic relationship graphs. Background Technology

[0002] In the field of computer vision, image region labeling aims to assign semantic labels to different regions in an image, and is a core task of scene understanding. Existing technologies, graph model-based methods, by modeling image regions as graph nodes and utilizing graph neural networks for information propagation, leverage the relationships between regions to some extent.

[0003] However, current image region annotation methods typically employ static graph structures based on fixed distance metrics or spatial adjacency relationships. These structures cannot adaptively adjust the connection strength between nodes according to semantic changes in image content, leading to a disconnect between the graph topology and the feature evolution process. Furthermore, existing methods often ignore the differentiated semantic interaction types between nodes and use a uniform aggregation strategy to process all neighbor information, making it difficult to accurately express complex scene semantics.

[0004] Furthermore, traditional image region annotation methods lack an effective cross-level feature propagation mechanism when dealing with multi-scale targets, resulting in decreased annotation accuracy for small target regions due to the lack of contextual information.

[0005] Therefore, it is necessary to provide an image region annotation method and system based on visual semantic relationship graphs to solve the aforementioned problem.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide an image region annotation method and system based on visual semantic relationship graphs to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: An image region annotation method based on visual semantic relationship graphs, comprising the following steps: Step 1: Obtain a multi-scale region set of the image to be labeled, and extract multi-level node features of each image region in the multi-scale region set; Step 2: Construct an initial semantic relationship graph based on the multi-level node features. The initial semantic relationship graph includes nodes and initial connection weights between nodes. Step 3: In the iterative layer of the graph neural network, the connection weights in the initial semantic relationship graph are dynamically updated based on the node features of the current iterative layer to obtain the dynamic semantic relationship graph; Step 4: Based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, use the corresponding aggregation strategy to perform differential aggregation on the features of neighboring nodes to obtain aggregated features; Step 5: Based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, the features of the coarse-grained regions are propagated and fused to the corresponding fine-grained regions to obtain cross-level fused features. Step 6: Based on the aggregated features and the cross-level fusion features, determine the semantic annotation results of each image region in the image to be annotated.

[0009] By dynamically updating connection weights during the iteration of the graph neural network, the graph structure can adaptively adjust to follow the evolution of node features, overcoming the limitations of static graph structures. At the same time, differential aggregation based on semantic relationship categories accurately captures diverse interaction patterns between nodes. Combined with cross-level feature propagation from coarse to fine granular, the contextual semantic information of small target regions is effectively supplemented, thereby significantly improving the overall performance of image region annotation.

[0010] Furthermore, in the iterative layers of the graph neural network, the connection weights in the initial semantic relation graph are dynamically updated based on the node features of the current iteration layer, resulting in a dynamic semantic relation graph, including: In the current iteration layer of the graph neural network, the attention weights between nodes are calculated based on the node features of the current iteration layer; the connection weights of the current iteration layer are updated by combining the connection weights and attention weights of the previous iteration layer. The updated connection weights are used as the connection weights of the dynamic semantic relationship graph.

[0011] By combining the attention weights of the current layer with the connection weights of the previous layer, the temporal continuity of graph structure evolution is reflected, topological abrupt changes caused by independent recalculation of each layer are avoided, and the smoothness and stability of the feature aggregation process are ensured.

[0012] Furthermore, by combining the connection weights and attention weights from the previous iteration layer, the connection weights of the current iteration layer are updated, including: The node features of the first and second nodes in the current iteration layer are concatenated to obtain the concatenated features. The concatenated features are mapped using an attention parameter matrix and then processed by an activation function to obtain attention weights. The attention weights are multiplied by the connection weights between the first and second nodes in the previous iteration layer to obtain the connection weights of the current iteration layer. Updates are performed at the specified iteration layer of the graph neural network, and when the spatial distance between nodes is greater than a preset distance threshold, the corresponding connection weights are set to zero.

[0013] Through specific splicing mapping and product update mechanisms, the dynamic association strength between nodes is quantified. Combined with spatial distance threshold constraints, key semantic connections are preserved while long-distance noise interference is eliminated, balancing the accuracy and computational efficiency of dynamic updates.

[0014] Furthermore, based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, the features of neighboring nodes are differentially aggregated using the corresponding aggregation strategy to obtain aggregated features, including: Determine the semantic relationship categories between nodes in the dynamic semantic relationship graph; for each semantic relationship category, use the corresponding weight matrix to aggregate the features of neighboring nodes with that semantic relationship category to obtain single-relationship aggregated features; We perform weighted fusion of single-relation aggregate features corresponding to multiple semantic relation categories to obtain aggregate features.

[0015] It breaks away from the traditional graph convolutional aggregation method that treats all neighbors equally. By assigning independent weight matrices to different semantic relationships, it achieves decoupled representation of different interaction modes such as inclusion, adjacency, and similarity, thereby enhancing the model's ability to parse the semantic structure of complex scenes.

[0016] Furthermore, the semantic relationship categories between nodes in the dynamic semantic relationship graph are determined, including: The node features of the first node and the second node are concatenated and input into the relationship classifier to predict the probability that the first node and the second node belong to each candidate relationship category. The candidate relationship category with the highest probability is determined as the semantic relationship category between the first node and the second node; We perform weighted fusion of single-relation aggregate features corresponding to multiple semantic relation categories to obtain aggregate features, including: The fusion weights for each semantic relation category are learned through an attention mechanism; The aggregated features are obtained by weighted summation of multiple single-relation aggregated features based on fusion weights.

[0017] By using a relation classifier to explicitly predict the relation types between nodes and by adaptively learning the fusion weights of each relation type through an attention mechanism, the model can automatically focus on the semantic relations that are most discriminative for the labeling task based on the current node features, thereby further improving the targeting of feature aggregation.

[0018] Furthermore, based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, the features of the coarse-grained regions are propagated and fused to the corresponding fine-grained regions to obtain cross-level fused features, including: Based on the overlap between image regions in the multi-scale region set, a parent-child mapping relationship between fine-grained regions and coarse-grained regions is constructed. Based on the parent-child mapping relationship, the features of the coarse-grained region corresponding to the parent node are scale-aligned. The scale-aligned features are propagated and fused into the fine-grained regions corresponding to the child nodes to obtain cross-level fused features.

[0019] By constructing a clear hierarchical topology through overlap, and propagating global semantic information from coarse-grained regions to fine-grained regions, the semantic ambiguity caused by limited receptive fields in small targets or detailed regions is effectively solved, and the consistency of multi-scale features is enhanced.

[0020] Furthermore, a multi-scale region set of the image to be labeled is obtained, including: Feature extraction networks are used to extract features from the image to be labeled, resulting in a multi-scale feature pyramid. Superpixel segmentation is performed on the feature maps of each scale in the multi-scale feature pyramid to generate a set of regions at the corresponding scale. The multi-scale region set includes the set of regions at the corresponding scale. The scale-aligned features are propagated and fused into the fine-grained regions corresponding to the child nodes to obtain cross-level fused features, including: The features of the coarse-grained region corresponding to the parent node are upsampled and weighted according to a preset cross-level fusion coefficient, and then superimposed and fused with the current features of the fine-grained region corresponding to the child node.

[0021] By generating specific paths across multiple scales and performing refined cross-level fusion operations, and through upsampling and weighted overlay, precise spatial and semantic alignment of features at different resolutions is achieved, ensuring the effectiveness of multi-level information interaction.

[0022] Furthermore, an initial semantic relationship graph is constructed based on multi-level node features, including: Based on the spatial adjacency relationship between image regions in the multi-scale region set, the initial connection weights between nodes in the initial semantic relationship graph are determined. Among them, the nodes corresponding to spatially adjacent image regions have an initial connection weight greater than zero, and the nodes corresponding to spatially non-adjacent image regions have an initial connection weight of zero.

[0023] Using spatial adjacency as a priori basis for graph structure provides a reasonable initialization state for subsequent dynamic updates, avoiding computational redundancy caused by fully connected graphs, and also preventing convergence difficulties caused by completely random initialization.

[0024] Furthermore, based on aggregated features and cross-level fusion features, the semantic annotation results of each image region in the image to be annotated are determined, including: The aggregated features and cross-level fusion features are fused to obtain the final feature representation of each image region; The final feature representation is input into the classifier to predict the semantic category label of each image region; based on the semantic category label of each image region, a semantic annotation map of the image to be annotated is generated.

[0025] It completes the closed loop from feature enhancement to semantic decision-making. By fusing dynamic aggregated features and cross-level features, it fully utilizes the dual gains of graph structure optimization and multi-scale context to output high-precision pixel-level or region-level annotation results.

[0026] Compared with the prior art, the beneficial effects of the present invention are: This invention overcomes the shortcomings of static graph structures in traditional methods by dynamically updating the connection weights of semantic relationship graphs during the iteration process of graph neural networks. Its principle lies in using the real-time evolution feedback of node features to correct the graph topology, so that the graph structure can adaptively match the semantic distribution of image content, thereby capturing the real relationship between regions more accurately in complex scenes. By introducing a differentiated aggregation strategy based on semantic relationship categories, the bottleneck of a single aggregation mode in expressing diverse interaction relationships is overcome. The principle is to assign exclusive feature transformation and fusion weights to different semantic interactions, thereby achieving refined transmission of semantic information and significantly improving the labeling consistency of boundary regions.

[0027] Furthermore, this invention solves the technical problem of missing context in small target regions by constructing a cross-level feature propagation path from coarse-grained regions to fine-grained regions. The principle is to use the stable global semantics of large-scale regions to guide the local feature enhancement of small-scale regions, effectively making up for the semantic ambiguity of low-level features and greatly improving the annotation accuracy of small targets and detailed regions. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a schematic diagram of the system module flow of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0030] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] Example 1 Please see Figure 1 An image region annotation method based on visual semantic relationship graphs, the specific steps of which include: Step 1: Obtain a multi-scale region set of the image to be labeled, and extract multi-level node features of each image region in the multi-scale region set.

[0032] Specifically, a multi-scale region set refers to a combination of regions obtained by segmenting the image to be labeled at different granularities. For example, it can include coarse-grained regions reflecting the overall layout and fine-grained regions reflecting detailed textures. This multi-scale segmentation aims to address the difficulty of simultaneously considering global context and local details at a single scale. Multi-level node features are the result of multi-dimensional representation of each image region, encompassing not only low-level visual features such as color and texture but also incorporating high-level semantic embedding information. By extracting multi-level node features, a rich data foundation can be provided for subsequent graph construction and inference, enabling the model to perceive both the appearance attributes of regions and understand their potential semantic connotations, thus laying a feature foundation for accurate annotation.

[0033] It should be noted that, in order to obtain feature maps of the image to be labeled at different resolutions in this embodiment, a method of constructing carriers of different granularities is adopted. Specifically, this includes: using a multi-scale feature pyramid as the physical carrier for segmentation of different granularities, and using a convolutional feature extraction network to propagate forward through the image to be labeled, extracting the output feature maps of different depth stages of the network, forming a feature pyramid of multi-level resolution from high to low; wherein, the shallow high-resolution feature maps (e.g., sampling ratios of 1 / 4 and 1 / 8) correspond to the fine-grained segmentation level, which can carry the segmentation of detailed textures and small target regions; the deep low-resolution feature maps (e.g., sampling ratios of 1 / 16 and 1 / 32) correspond to the coarse-grained segmentation level, which can carry the segmentation of global layout and large-scale semantic regions. The hierarchical division of segmentation granularity is naturally achieved through the difference in the resolution of the feature maps.

[0034] In this embodiment, the scale-differentiated segmentation method used to obtain region combinations at different scale levels specifically includes: for each layer of the feature pyramid feature map, superpixel segmentation is performed independently to obtain a region set of corresponding granularity. Specifically, the SLIC simple linear iterative clustering algorithm can be used: the feature vector of each pixel in the feature map is combined with the spatial coordinates to form a clustering vector, and clustering is performed with feature similarity and spatial distance as constraints to generate several superpixel regions with uniform internal features. By configuring differentiated segmentation parameters for feature maps of different resolutions, the segmentation granularity of each level is precisely controlled: for fine-grained high-resolution feature maps, a larger number of superpixel targets and a smaller maximum region area are set to generate a large number of small-sized fine-grained image regions; for coarse-grained low-resolution feature maps, a smaller number of superpixel targets and a larger maximum region area are set to generate a small number of large-sized coarse-grained image regions.

[0035] Finally, all superpixel regions obtained from segmentation at all scale levels are summarized, and each region retains its spatial coordinate range, scale level, and corresponding feature vector, ultimately forming a multi-scale region set covering coarse, medium, and fine granularities, which serves as the basic unit for subsequent graph node construction.

[0036] Step 2: Construct an initial semantic relationship graph based on multi-level node features. The initial semantic relationship graph includes nodes and the initial connection weights between nodes.

[0037] In this embodiment, the process of constructing the initial semantic relationship graph specifically includes: determining the initial connection weights between nodes in the initial semantic relationship graph based on the spatial adjacency relationships between image regions in the multi-scale region set; wherein, nodes corresponding to spatially adjacent image regions have initial connection weights greater than zero, and nodes corresponding to spatially non-adjacent image regions have initial connection weights of zero. This design introduces an explicit spatial geometric prior. The reason for using spatial adjacency relationships to determine the initial weights is that in natural images, semantic relevance is often closely coupled with spatial location, and adjacent regions are more likely to belong to the same object or have strong semantic associations. Setting the initial connection weights of non-adjacent nodes to zero not only conforms to the assumption of local continuity of the image, but also effectively constructs a sparse graph structure, avoiding the huge computational redundancy brought by fully connected graphs, and also preventing convergence difficulties caused by completely random initialization, providing a reasonable and efficient starting point for subsequent dynamic evolution.

[0038] It should be noted that the adjacency determination rule for image regions of the same scale is as follows: First, for each image region at that scale, a binary spatial mask with the same size as the corresponding feature map is generated. Pixels belonging to this region are marked as 1, and the rest are marked as 0, thereby accurately defining the spatial range and boundary of each region. The binary mask of each region is processed by a morphological edge extraction operator to obtain the set of closed boundary pixels of the region. For any two regions of the same scale, if there is a non-empty intersection of their boundary pixel sets, it is determined that the two regions have boundary contact and are qualified as adjacency candidates. Finally, the length of the shared boundary pixels of the two regions is calculated. When the length of the shared boundary is greater than the preset adjacency threshold, they are finally determined to be spatially adjacent. As a preferred embodiment, the preset adjacency threshold can be set to 2 pixels, or set to 3%-5% of the perimeter of the smaller region in the two regions, to eliminate noisy pseudo-adjacencies with only single-point contact and avoid introducing invalid connections in the initial image.

[0039] The adjacency determination rule for cross-scale image regions is as follows: First, the spatial coordinates of coarse-grained and fine-grained regions are uniformly mapped to the original image coordinate system to eliminate spatial misalignment caused by resolution differences. Then, the proportion of the intersection area of ​​the fine-grained and coarse-grained regions to the total area of ​​the fine-grained region (i.e., the sub-region overlap rate) is calculated to quantify the spatial correlation between the two. Finally, when the sub-region overlap rate is greater than a preset overlap threshold, the coarse-grained and fine-grained regions are determined to be spatially adjacent. As a preferred embodiment, the preset overlap threshold can be set to 0.2, that is, when more than 20% of the area of ​​the fine-grained region falls within the range of the coarse-grained region, an initial connection is established between the two.

[0040] It should be noted that the initial connection weights are assigned according to the following rules: For nodes that are determined to be spatially adjacent, the initial connection weights can be set to a fixed value of 1.0; or normalized assignment can be used: the initial weights of the same scale region are positively correlated with the length of the shared boundary, the initial weights of the cross-scale region are positively correlated with the overlap rate, and the weight values ​​are uniformly normalized to the interval (0,1]. For node pairs that are not determined to be spatially adjacent, the initial connection weight is fixed at 0, thereby constructing a sparse initial adjacency matrix, which avoids computational redundancy in the fully connected graph while preserving spatial geometric priors.

[0041] Step 3: In the iterative layer of the graph neural network, the connection weights in the initial semantic relationship graph are dynamically updated based on the node features of the current iterative layer to obtain the dynamic semantic relationship graph.

[0042] This step is the core difference between this application and traditional static graph methods. During feature propagation in a graph neural network, the feature representation of a node evolves continuously with the increase in network layers, and the semantic information it contains gradually becomes richer. If a fixed graph structure is used, it may lead to a mismatch between the graph topology and deep semantic features, limiting the accurate transmission of information. Therefore, this embodiment utilizes the node features of the current layer in real time to provide feedback and adjust connection weights in at least some iterative layers, enabling the graph structure to adaptively change with the evolution of semantic features. This dynamic update mechanism ensures that the graph topology always remains in sync with the current semantic state, thereby more accurately capturing the true semantic relationships between regions in complex scenarios and improving the effectiveness of feature aggregation.

[0043] Step 4: Based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, use the corresponding aggregation strategy to perform differential aggregation of the features of neighboring nodes to obtain aggregated features.

[0044] In image scenes, the interaction patterns between nodes are diverse, including relationships such as "parts of the same object," "spatially adjacent but semantically different," and "having similar textures." Traditional graph convolutional networks typically apply uniform aggregation weights to all neighboring nodes, ignoring the heterogeneity of these relationships and easily leading to blurred boundaries or semantic confusion.

[0045] This embodiment achieves decoupling processing of different types of interactive information by identifying the semantic relationship categories between nodes and applying corresponding aggregation strategies for each category. For example, feature fusion can be enhanced for "same object" relationships, while feature smoothing can be suppressed for "dissimilar adjacency" relationships. This differentiated aggregation mechanism significantly enhances the model's ability to parse the semantic structure of complex scenes, making the aggregated features more discriminative.

[0046] Step 5: Based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, the features of the coarse-grained regions are propagated and fused to the corresponding fine-grained regions to obtain cross-level fused features.

[0047] Since fine-grained regions (such as small targets or object edges) often lack sufficient contextual information, relying solely on intra-graph aggregation at the same level can easily lead to semantic ambiguity.

[0048] This embodiment establishes a unidirectional propagation path from coarse-grained regions to fine-grained regions, utilizing the stable global semantics inherent in the coarse-grained regions to guide feature enhancement in the fine-grained regions. This cross-level feature propagation mechanism essentially injects large-scale contextual information into small-scale local features, effectively compensating for the semantic ambiguity of lower-level features, solving the problem of decreased annotation accuracy caused by limited receptive fields in small target regions, and enhancing the consistency of multi-scale features.

[0049] Step 6: Based on aggregated features and cross-level fusion features, determine the semantic annotation results of each image region in the image to be annotated.

[0050] Specifically, the steps include: fusing aggregated features and cross-level fusion features to obtain the final feature representation of each image region; inputting the final feature representation into a classifier to predict the semantic category label of each image region; and generating a semantic annotation map of the image to be annotated based on the semantic category label of each image region.

[0051] In this process, aggregated features represent semantic interaction information at the same level after dynamic graph optimization, while cross-level fused features supplement contextual guidance information from a larger scale. Fusing these two types of features fully leverages the dual gains of adaptive graph structure evolution and multi-scale information complementarity, forming a more complete and robust region feature representation. Subsequently, the classifier makes semantic decisions based on this final feature representation, outputting a category label for each region, and finally mapping it back to the image space to generate a semantic annotation map. This closed-loop process ensures that the annotation results have both local accuracy and maintain global consistency.

[0052] It should be understood that although this embodiment describes the image region annotation method in a specific order of steps, in other embodiments, as long as the above functional logic is satisfied, the execution order of the steps can be appropriately adjusted, or some steps can be executed in parallel. Furthermore, this embodiment mainly elucidates the overall architecture and core principles of the method. Implementation details such as the specific calculation method for dynamic updates, the detailed definition of relationship categories, and the specific parameters for cross-level fusion will be further explained in subsequent embodiments.

[0053] Example 2 Based on Embodiment 1 above, this embodiment further details the specific implementation of dynamically updating connection weights in step 3. In at least some iterative layers of a graph neural network, the process of dynamically updating the connection weights in the initial semantic relation graph based on the node features of the current iterative layer to obtain a dynamic semantic relation graph specifically includes the following sub-steps: Step 301: In the current iteration layer of the graph neural network, calculate the attention weights between nodes based on the node features of the current iteration layer.

[0054] Specifically, the attention weight reflects the potential correlation strength between two nodes in the current semantic feature space. Unlike existing technologies that rely solely on fixed geometric distance or single feature similarity, the attention weight in this embodiment changes in real time as the network depth increases and the feature semantics become richer. For example, in shallow networks, the attention weight may reflect more the similarity of texture or color; while in deep networks, it may reflect more the consistency of high-level semantic categories. This adaptive calculation method ensures that the graph structure can keenly capture the inherent connections between image content at different levels of abstraction.

[0055] Step 302: Combine the connection weights and attention weights from the previous iteration layer to update the connection weights of the current iteration layer. This step is crucial for maintaining the continuity of graph topology evolution in this application.

[0056] The reason for not recalculating the adjacency matrix from scratch in each layer is that completely independent recalculation would cause drastic changes in the graph structure between layers, disrupting the smoothness of feature propagation and even leading to gradient instability. By combining the connection weights of the previous iteration layer, the historical topology is essentially preserved as prior information, and the attention weights of the current layer are used to correct or enhance it. This mechanism is similar to a kind of "inertia" constraint, allowing the evolution of the graph structure to adapt to new semantic changes without losing existing reasonable connection patterns, thus ensuring the stability and convergence speed of model training.

[0057] Step 303: Use the updated connection weights as the connection weights of the dynamic semantic relation graph. The connection weights obtained after the above update constitute the basis of the dynamic graph structure used for feature aggregation in the current iteration layer.

[0058] As a specific implementation method, the process of updating the connection weights and attention weights of the previous iteration layer in step 302 can be achieved through the following refined steps: First, the node features of the first and second nodes in the current iteration layer are concatenated to obtain concatenated features. This concatenation operation preserves the independent semantic information of both nodes, providing a complete input vector for subsequent determination of their relationship. It should be understood that, in addition to concatenation, other embodiments may also use element-wise addition, Hadamard product, or bilinear pooling to fuse the features of two nodes, as long as the joint semantic state of the node pair can be expressed.

[0059] Secondly, the concatenated features are mapped using an attention parameter matrix and processed by an activation function to obtain attention weights. The attention parameter matrix consists of learnable model parameters used to project the high-dimensional concatenated features into a scalar space to quantify the degree of association between nodes. The activation function typically employs non-linear functions such as Sigmoid or LeakyReLU to introduce non-linear expressiveness and normalize the output values ​​to a reasonable range. This process is essentially an end-to-end relation strength predictor, capable of automatically learning which feature combinations correspond to strong connections.

[0060] Next, the attention weights are multiplied by the connection weights between the first and second nodes in the previous iteration layer to obtain the connection weights for the current iteration layer. This can be expressed as: in, Indicates the first Layer nodes With nodes Connection weights between them This represents the activation function. Indicates the first The attention parameter matrix of the layer, and Representing nodes respectively and nodes In the The feature vector of the layer, Demonstrate splicing operation. Indicates the previous iteration layer (the first iteration layer) The connection weights corresponding to the layers. The physical meaning of the product operation is: only when historical connections exist (i.e., The final connection weight will remain large only when the current semantics also support the connection (i.e., the attention weight is high). If the historical connection is zero, the connection will not be generated out of thin air even if the current attention weight is high, which effectively prevents false connections caused by noise. Conversely, if the historical connection exists but the current semantics no longer supports it, the connection weight will gradually decay. This gating update mechanism realizes the robust evolution of the graph structure.

[0061] In practical applications, to balance computational efficiency and the benefits of dynamic updates, it is not necessary to perform the aforementioned update operation at every layer of the graph neural network. This application proposes performing updates at designated iterative layers of the graph neural network. For example, in a network containing four layers of graph convolutions, dynamic updates can be performed at layers 1 and 3, while layers 2 and 4 retain the graph structure of the previous layer. This interval update strategy is based on the fact that feature changes between adjacent layers are usually gradual, and frequent updates result in diminishing marginal returns, while increasing the overhead of matrix operations. By updating at key layers (such as layers where feature semantics undergo significant transitions), major topological changes can be captured while significantly reducing computational load. It should be understood that the selection strategy for designated iterative layers can be flexibly adjusted according to network depth and task complexity; for example, updates can be performed every N layers or only at the last layer, and this application does not limit this.

[0062] Furthermore, to further remove long-distance noise interference and maintain the sparsity of the graph structure, this embodiment also introduces spatial distance constraints. Specifically, when the spatial distance between nodes is greater than a preset distance threshold, the corresponding connection weight is set to zero. This is because semantic relevance in natural images is often localized; regions that are too far apart, even if they have similar features, are often just coincidences rather than genuine semantic connections. By setting a distance threshold, these long-range pseudo-connections can be forcibly severed, making the graph structure more focused on local semantic interactions. As a preferred example, the preset distance threshold can be set to 50 pixels. Under this setting, regions with a center-to-center distance greater than 50 pixels will be considered to have no direct association. Of course, this threshold is not fixed and can be adaptively adjusted according to image resolution, average target size, or average radius of superpixel segmentation. For example, for high-resolution images, the threshold can be appropriately increased to cover a larger receptive field; for dense small target scenes, the threshold can be decreased to improve connection accuracy. This sparsity strategy not only improves annotation accuracy but also significantly reduces the number of non-zero elements participating in matrix operations, thereby further optimizing inference speed.

[0063] In summary, this embodiment constructs a complete dynamic connection weight update scheme through a product update mechanism, a specified layer update strategy, and spatial distance threshold constraints. This scheme effectively maintains the continuity of topological evolution, computational sparsity, and overall operational efficiency while ensuring that the graph structure can adaptively evolve with semantic features. It solves the problem of balancing accuracy and performance in traditional static or fully dynamic graphs.

[0064] Example 3 Based on the above embodiment 1, this embodiment further details the specific implementation of step 4, which uses the corresponding aggregation strategy to differentially aggregate the features of neighboring nodes based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, and obtains aggregated features.

[0065] In traditional graph convolutional networks, information from all neighboring nodes is typically transformed and aggregated using the same weight matrix. This approach implicitly assumes that only one homogeneous interaction pattern exists between nodes. However, in real-world image scenarios, the relationships between regions are highly heterogeneous. For example, components belonging to the same object have a "composition" relationship, while adjacent different objects have a "boundary" relationship. If uniform aggregation is performed without distinction, it can easily lead to overly smoothed features at object boundaries, causing the annotation results to stick or become blurred at semantic boundaries.

[0066] This embodiment introduces a relationship-aware differential aggregation mechanism to decouple the interactions between nodes into multiple independent semantic relationship categories, and assigns exclusive feature transformation parameters to each category, thereby achieving refined modeling of complex scene structures.

[0067] Specifically, this differentiated aggregation process first includes determining the semantic relationship categories between nodes in the dynamic semantic relationship graph. As a preferred implementation, the process of determining semantic relationship categories can be achieved through explicit classification prediction. Specifically, this includes: concatenating the node features of the first node and the second node, inputting the result into a relationship classifier, predicting the probability that the first node and the second node belong to each candidate relationship category; and determining the candidate relationship category with the highest probability as the semantic relationship category between the first node and the second node. The relationship classifier here can be a multilayer perceptron (MLP) or other form of neural network classification head, whose input is a joint representation of the features of the two nodes, and whose output is a probability distribution covering all predefined relationship types. In this way, the model can dynamically determine the most likely semantic interaction type between the node pairs based on their real-time feature states. It should be understood that although this embodiment uses "taking the maximum probability" as a hard decision example, in other implementations, a soft allocation strategy can also be adopted, that is, retaining multiple high-confidence relationship types and their corresponding probabilities for weighted allocation in subsequent aggregation; this application does not limit this approach.

[0068] It should be noted that the above relation classifier adopts a three-layer fully connected neural network structure, consisting of an input layer, intermediate hidden layers, and an output layer cascaded in sequence. Let the feature dimension of a single node be d, and the total number of predefined semantic relation categories be K. The structural parameters of the preferred implementation are as follows: Input layer: 2D in dimension, used to receive the concatenated vector of features from two nodes; The first hidden layer has a dimension of d, followed by a batch normalization layer and a ReLU activation function. A Dropout layer with a dropout probability of 0.2 is also set to suppress overfitting. The second hidden layer has a dimension of d / 2, followed by a batch normalization layer and a ReLU activation function; Output layer: dimension K, followed by a Softmax activation function, outputting the normalized probability distribution of each candidate relation category.

[0069] The specific method for determining the input and output format of the relation classifier is as follows: The feature vectors of any first node and second node in the dynamic semantic relation graph are concatenated along the feature dimension to obtain a joint feature vector, which serves as the input to the relation classifier. The relation classifier outputs a probability vector with a dimension consistent with the total number of predefined semantic relation categories. Each element in the vector corresponds to the predicted probability of the current two nodes belonging to each semantic relation category. The sum of the probability values ​​corresponding to all categories is 1. When using a hard decision strategy, the semantic relation category with the highest predicted probability is selected as the final determined semantic relation category for that node. When using a soft allocation strategy, the complete probability values ​​for all corresponding semantic relation categories of that node are retained for weighted calculation in subsequent feature aggregation steps.

[0070] The relation classifier adopts a two-stage training approach of "weakly supervised pre-training + end-to-end joint fine-tuning", as detailed below: In the weakly supervised pre-training stage, based on the dataset with pixel-level semantic annotation, pseudo-labels for semantic relationships corresponding to each node pair are automatically generated according to preset rules. The relationship classifier is then pre-trained separately using cross-entropy loss as the objective function. The loss calculation logic is as follows: iterate through all node pairs and all predefined semantic relationship categories, calculate the product of the one-hot encoding value of the pseudo-label corresponding to each relationship category and the logarithm of the predicted probability for that category, sum all the product results, take the negative value, and then divide by the total number of node pairs to obtain the relationship classification loss value for this training round. In this stage, an adaptive moment estimation optimizer is used to perform parameter updates, with an initial learning rate set to 0.01% and a preset total of 20 training rounds. In the end-to-end joint fine-tuning phase, the pre-trained relation classifier is embedded into the overall image region labeling network. It undergoes joint end-to-end training together with the feature extraction network, graph neural network, and labeling classifier. The total training loss is a weighted combination of semantic labeling loss and relation classification loss: the semantic labeling loss is the cross-entropy loss corresponding to the semantic labeling task, and the relation classification loss is the cross-entropy loss corresponding to the aforementioned relation classification task. The proportions of the two types of losses are adjusted by balancing weights, with an optimal value of 0.3. During this training phase, all parameters of the relation classifier are updated synchronously with the gradient backpropagation process of the overall network, without the need for additional manually labeled semantic relation real-world tags.

[0071] It should be noted that the training data for the relation classifier mainly comes from three sources: First, weakly supervised data derived from publicly available semantic segmentation datasets. Based on images from publicly available semantic segmentation datasets, multi-scale superpixel segmentation is performed to obtain a set of regions. Combined with the pixel-level semantic labels provided by the dataset, semantic relation pseudo-labels for various node pairs are automatically generated according to preset rules for classifier pre-training. Second, manually labeled validation sets are used. No fewer than 200 images covering typical scenes are selected, and the semantic relations of region pairs are manually labeled by professionals. This is only used to verify classification accuracy and does not participate in model training. Third, self-supervised pre-training data is used. Positive and negative sample pairs are constructed for unlabeled images through data augmentation methods such as cropping and color perturbation. The feature extraction layer of the classifier is pre-trained using a contrastive learning approach to improve its ability to represent visual semantic associations.

[0072] It should be noted that an independent feature transformation weight matrix of dimension d×d is configured for each semantic relation category, and it is initialized using the He normal method. The weight matrices of each relation, as well as the weight matrices and the basic weights of the GNN convolutional layer, are independent of each other and learned separately. Decoupling and aggregation of different relation types are achieved through dedicated feature transformation.

[0073] After determining the semantic relationship categories between nodes, for each semantic relationship category, the features of neighboring nodes with that semantic relationship category are aggregated using a corresponding weight matrix to obtain single-relationship aggregated features. This means that for neighboring node pairs determined to have an "inclusion relationship," the first weight matrix is ​​used for feature transformation; while for neighboring node pairs determined to have an "adjacency relationship," the second weight matrix is ​​used. Each weight matrix is ​​an independently learnable parameter specifically designed to capture the information transmission patterns under a particular relationship type. For example, under an "inclusion relationship," the weight matrix tends to promote feature fusion and enhancement to strengthen the consistency within an object; while under an "adjacency relationship," the weight matrix may learn patterns to suppress feature diffusion to maintain clear boundaries between different semantic regions. This parameter decoupling design allows the model to process multiple semantic flows in parallel on the same graph structure, significantly improving the richness of feature representation.

[0074] Subsequently, the single-relation aggregated features corresponding to multiple semantic relation categories are weighted and fused to obtain the final aggregated features. To adaptively balance the contribution of different relation types to the current node annotation, this embodiment learns the fusion weights of each semantic relation category through an attention mechanism; based on these fusion weights, the multiple single-relation aggregated features are weighted and summed to obtain the aggregated features. Specifically, the fusion weights are not preset fixed values, but are calculated interactively with each single-relation aggregated feature through a shared attention vector. This attention mechanism can automatically identify which relation type information is most critical at the current moment based on the feature context of the current node. For example, when the node is inside an object, the model may assign a higher fusion weight to "containment relation"; while when the node is at the edge of an object, the fusion weight of "adjacency relation" will automatically increase. This adaptive fusion strategy ensures that the aggregated features contain both necessary contextual information and retain key boundary details, effectively solving the boundary confusion problem commonly found in traditional methods.

[0075] To more clearly illustrate the practical application of the aforementioned differentiated aggregation mechanism, a specific scenario configuration is used as an example below. In this configuration, the predefined set of semantic relationship categories includes four types: containment relationship, adjacency relationship, similarity relationship, and hierarchical relationship. "Containment relationship" refers to a region being a component of another region (e.g., a car window and a car). The weight matrix corresponding to this relationship is mainly used to integrate local details to form a complete object representation. "Adjacency relationship" refers to two regions being spatially adjacent but semantically different (e.g., a car and a road). The weight matrix corresponding to this relationship focuses on maintaining the sharpness of semantic boundaries and preventing feature leakage across boundaries. "Similarity relationship" refers to two regions having similar appearances or textures but not necessarily a direct spatial hierarchy (e.g., two different cars). This relationship helps to utilize non-local similar samples to assist in the classification of the current region. "Hyperarchical relationship" reflects the hierarchical affiliation between multi-scale regions (e.g., a coarse-grained "vehicle" region and a fine-grained "tire" region), used to convey cross-level semantic constraints. By assigning independent weight matrices to these four relationships and dynamically fusing them via an attention mechanism, the model can simultaneously handle multiple objectives such as object integrity, boundary clarity, global consistency, and multi-scale coordination within a unified framework. This results in significantly higher annotation accuracy than a single aggregation strategy in complex urban street scenes. It should be understood that the four relationship types described above are merely illustrative. In practical applications, the number and definition of relationship types can be expanded or adjusted according to specific task requirements, such as adding "occlusion relationships" or "symmetry relationships." As long as the core idea of ​​"category-based differential aggregation" is followed, it falls within the scope of this application.

[0076] Example 4 Based on the above embodiment 1, this embodiment further details the specific implementation method of propagating and fusing the features of the coarse-grained region to the corresponding fine-grained region in step 5 based on the hierarchical mapping relationship between the coarse-grained region and the fine-grained region in the multi-scale region set, so as to obtain cross-level fused features.

[0077] In image annotation tasks, fine-grained regions often correspond to small targets or object edge details in an image. Due to their limited receptive field, these regions struggle to obtain sufficient contextual semantic information through feature aggregation at the same level, easily leading to classification ambiguity. In contrast, coarse-grained regions cover a larger spatial range and contain more stable and explicit global semantics. Therefore, this embodiment constructs a unidirectional feature propagation path from coarse-grained regions to fine-grained regions, utilizing high-level global semantics to guide low-level local feature enhancement, thereby effectively compensating for the semantic deficiencies in small target regions.

[0078] Specifically, the process of constructing hierarchical mapping relationships includes: based on the overlap between image regions in a multi-scale region set, constructing a parent-child mapping relationship between fine-grained regions and coarse-grained regions. Here, overlap is a key indicator for measuring whether two regions have a spatial inclusion or belonging relationship. As a preferred implementation, overlap can be calculated using the intersection-over-union (IoU) ratio. For any fine-grained region, calculate its IoU value with all candidate coarse-grained regions, and determine the coarse-grained region with the largest IoU value as the parent node of the fine-grained region. The reason for choosing IoU instead of simple center point distance is that IoU can more accurately reflect the geometric coverage between regions, avoiding mismatches caused by irregular region shapes. It should be understood that, in addition to IoU, other embodiments can also use indicators such as the ratio of intersection area to fine-grained region area (Intersection over Child Area), as long as it can characterize the spatial belonging relationship. This mapping mechanism based on geometric overlap ensures the physical rationality of the semantic information transmission path, that is, only when a small region is indeed located inside a large region will it accept its semantic guidance.

[0079] After establishing the parent-child mapping relationship, the features of the coarse-grained region corresponding to the parent node are scale-aligned based on this relationship. Since the coarse-grained and fine-grained regions originate from feature maps of different scales, their feature resolutions and spatial dimensions are usually inconsistent, and direct fusion would lead to spatial misalignment. Therefore, the parent node features must first be transformed to the same spatial scale as the child nodes. Subsequently, the scale-aligned features are propagated and fused to the fine-grained region corresponding to the child nodes, resulting in cross-level fused features. This process essentially injects large-scale contextual priors into small-scale detailed features. It is crucial to emphasize that this propagation path is strictly unidirectional, allowing information to flow only from the parent node to the child node, and prohibiting reverse flow. This design aims to prevent noise or erroneous semantics in small target regions from contaminating the global features, ensuring the purity and stability of the top-level semantics of the graph structure.

[0080] It should be noted that the complete calculation process of the parent-child mapping relationship in the above embodiments is as follows: First, the multi-scale region set is divided into multiple levels according to the feature map resolution from high to low. A mapping relationship is established only between adjacent scale levels, where the higher-resolution level is the fine-grained sub-level, and the adjacent lower-resolution level is the coarse-grained parent level. The spatial coordinates of all regions in both levels are converted to the original image coordinate system according to the downsampling ratio of the corresponding feature map, eliminating the spatial scale differences caused by different resolutions. For each fine-grained region and each candidate coarse-grained region in the unified coordinate system, a binary spatial mask with the same size as the original image is generated. Pixel positions covered by the region are marked as 1, and uncovered positions are marked as 0, to accurately represent the spatial boundaries and coverage of each region. For any set of fine-grained regions and candidate coarse-grained regions, a pixel-by-pixel logical AND operation is performed on their binary masks to obtain the overlapping region mask. The total number of pixels with a value of 1 in the overlapping region mask is counted as the intersection area of ​​the two regions. Simultaneously, the total number of pixels with a value of 1 in the masks of both the fine-grained and coarse-grained regions is counted to obtain the area of ​​the corresponding region. The overlap rate of the fine-grained region relative to the candidate coarse-grained region is obtained by dividing the intersection area by the area of ​​the fine-grained region itself, thus quantifying the spatial dependence between the two. A preset minimum overlap threshold is set. When the overlap rate between the fine-grained region and the candidate coarse-grained region is lower than this threshold, it is determined that the two do not have a stable spatial dependence relationship. For each fine-grained region, the coarse-grained region with the largest overlap rate among all candidate coarse-grained regions that meet the threshold requirement is selected as its parent node, establishing a one-to-one parent-child mapping relationship. If the overlap rate between the fine-grained region and all coarse-grained regions is lower than the preset threshold, no parent node is assigned to it, and it does not participate in subsequent cross-level feature propagation.

[0081] To more clearly illustrate the engineering implementation of the aforementioned cross-level propagation mechanism, the following section elaborates on the generation process of multi-scale region sets and specific fusion operations. As a specific implementation method, the process of obtaining a multi-scale region set of the image to be labeled includes: extracting features from the image to be labeled using a feature extraction network to obtain a multi-scale feature pyramid; performing superpixel segmentation on the feature maps of each scale in the multi-scale feature pyramid to generate a region set of the corresponding scale, where the multi-scale region set includes the region set of the corresponding scale. Specifically, the feature extraction network can be a convolutional neural network such as ResNet, VGG, or EfficientNet, whose outputs at different stages constitute a feature pyramid from coarse to fine. For example, in a typical four-level pyramid structure, the first layer feature map has a resolution of 1 / 32 of the original image and is used to generate coarse-grained regions reflecting the overall scene layout; the fourth layer feature map has a resolution of 1 / 4 of the original image and is used to generate fine-grained regions reflecting the details of object components. Superpixel segmentation algorithms (such as SLIC or SEEDS) are responsible for dividing the feature map of each scale into several semantically uniform, boundary-fitting region blocks. This combination of "feature pyramid + superpixel segmentation" strategy ensures both the semantic consistency of region division and the preservation of multi-scale spatial hierarchy, providing a high-quality data foundation for subsequent cross-level interactions.

[0082] During feature fusion, the scale-aligned features are propagated and fused to the fine-grained regions corresponding to the child nodes to obtain cross-level fused features. Specific operations include: upsampling the features of the coarse-grained regions corresponding to the parent node, weighting them according to preset cross-level fusion coefficients, and then superimposing and fusing them with the current features of the fine-grained regions corresponding to the child nodes. The upsampling process can employ bilinear interpolation, nearest-neighbor interpolation, or transposed convolution, aiming to enlarge the spatial size of the parent node features to match that of the child node features. The cross-level fusion coefficients (denoted as...) This is used to control the contribution strength of parent node features to child node features. It can be expressed as a formula: in, This represents the fine-grained regional features after fusion. This represents the original features of the fine-grained region. Indicates the characteristics of coarse-grained regions. Indicates an upsampling operation. This represents the preset cross-level fusion coefficient. As a preferred example, It can be set to 0.3. This value is chosen based on the equilibrium point obtained from numerous experimental verifications: if If the size is too large, the detailed features of the child nodes will be overwhelmed by the global features of the parent node, resulting in the loss of the uniqueness of the small target; if... If the value is too small, it cannot provide enough contextual supplementation, and the improvement effect is not obvious.

[0083] certainly, These parameters can also be learnable or dynamically adjusted based on the confidence levels of parent and child nodes; this application does not impose any limitations on this. Through this weighted fusion method, the model can effectively incorporate macroscopic semantic constraints from coarse-grained regions while preserving the original detailed information of fine-grained regions, significantly improving the recognition and annotation accuracy of small target regions in complex backgrounds.

[0084] Example 5 Please see Figure 2 The present invention also provides an image region annotation system based on a visual semantic relationship graph, wherein the image region annotation system is used to perform the above-described image region annotation method, including: The feature extraction module is used to obtain a multi-scale region set of the image to be labeled, and to extract multi-level node features of each image region in the multi-scale region set; The graph construction and update module is used to construct an initial semantic relationship graph based on the multi-level node features. The initial semantic relationship graph includes nodes and initial connection weights between nodes. The relation construction module is used to dynamically update the connection weights in the initial semantic relation graph based on the node features of the current iteration layer in the iterative layer of the graph neural network, so as to obtain a dynamic semantic relation graph. The relation-aware aggregation module is used to perform differential aggregation of the features of neighboring nodes based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, and to obtain aggregated features by adopting the corresponding aggregation strategy. The cross-level propagation module is used to propagate and fuse the features of the coarse-grained region to the corresponding fine-grained region based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, so as to obtain cross-level fused features. The annotation module is used to determine the semantic annotation results of each image region in the image to be annotated based on the aggregated features and the cross-level fusion features.

[0085] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0086] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image region annotation method based on visual semantic relationship graphs, characterized in that, The specific steps include: Step 1: Obtain a multi-scale region set of the image to be labeled, and extract multi-level node features of each image region in the multi-scale region set; Step 2: Construct an initial semantic relationship graph based on the multi-level node features. The initial semantic relationship graph includes nodes and initial connection weights between nodes. Step 3: In the iterative layer of the graph neural network, the connection weights in the initial semantic relationship graph are dynamically updated based on the node features of the current iterative layer to obtain the dynamic semantic relationship graph; Step 4: Based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, use the corresponding aggregation strategy to perform differential aggregation on the features of neighboring nodes to obtain aggregated features; Step 5: Based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, the features of the coarse-grained regions are propagated and fused to the corresponding fine-grained regions to obtain cross-level fused features. Step 6: Based on the aggregated features and the cross-level fusion features, determine the semantic annotation results of each image region in the image to be annotated; The method used to obtain cross-level fusion features is as follows: Based on the overlap between image regions in the multi-scale region set, a parent-child mapping relationship between fine-grained regions and coarse-grained regions is constructed. Based on the parent-child mapping relationship, the features of the coarse-grained region corresponding to the parent node are scale-aligned. The scale-aligned features are propagated and fused into the fine-grained regions corresponding to the child nodes to obtain cross-level fused features. During feature fusion, the scale-aligned features are propagated and fused to the fine-grained regions corresponding to the child nodes to obtain cross-level fused features. Specific operations include: upsampling the features of the coarse-grained regions corresponding to the parent node, weighting them according to preset cross-level fusion coefficients, and then superimposing and fusing them with the current features of the fine-grained regions corresponding to the child nodes. The upsampling process uses bilinear interpolation, nearest-neighbor interpolation, or transposed convolution to enlarge the spatial size of the parent node features to match that of the child node features. The cross-level fusion coefficients control the contribution strength of the parent node features to the child node features; expressed by the formula: in, This represents the fine-grained regional features after fusion. This represents the original features of the fine-grained region. Indicates the characteristics of coarse-grained regions. Indicates an upsampling operation. This represents the preset cross-level fusion coefficient.

2. The image region annotation method based on visual semantic relationship graphs according to claim 1, characterized in that, The method used to obtain the dynamic semantic relationship graph is as follows: In the current iteration layer of the graph neural network, attention weights between nodes are calculated based on the node features of the current iteration layer; The connection weights of the current iteration layer are updated by combining the connection weights of the previous iteration layer with the attention weights. The updated connection weights are used as the connection weights of the dynamic semantic relationship graph.

3. The image region annotation method based on visual semantic relationship graphs according to claim 2, characterized in that, The connection weights of the current iteration layer are updated by combining the connection weights of the previous iteration layer with the attention weights, based on the following method: The node features of the first and second nodes in the current iteration layer are concatenated to obtain the concatenated features. The spliced ​​features are mapped using an attention parameter matrix and then processed by an activation function to obtain the attention weights. The attention weights are multiplied by the connection weights between the first and second nodes in the previous iteration layer to obtain the connection weights of the current iteration layer. Updates are performed at the specified iteration layer of the graph neural network, and when the spatial distance between nodes is greater than a preset distance threshold, the corresponding connection weights are set to zero.

4. The image region annotation method based on visual semantic relationship graph according to claim 1, characterized in that, The method used to obtain the aggregated features is as follows: Determine the semantic relationship categories between nodes in the dynamic semantic relationship graph; For each semantic relationship category, the features of neighboring nodes with that semantic relationship category are aggregated using the corresponding weight matrix to obtain single-relation aggregated features; The aggregated features are obtained by weighted fusion of the single-relation aggregated features corresponding to multiple semantic relation categories.

5. The image region annotation method based on visual semantic relationship graph according to claim 4, characterized in that, The method used to determine the semantic relationship categories between nodes in the dynamic semantic relationship graph is as follows: The node features of the first node and the second node are concatenated and input into the relationship classifier to predict the probability that the first node and the second node belong to each candidate relationship category. The candidate relationship category with the highest probability is determined as the semantic relationship category between the first node and the second node; The aggregated features are obtained by weighted fusion of single-relation aggregated features corresponding to multiple semantic relation categories, including: The fusion weights for each semantic relationship category are learned through an attention mechanism; The aggregated features are obtained by weighted summation of multiple single-relationship aggregated features based on the fusion weights.

6. The image region annotation method based on visual semantic relationship graph according to claim 1, characterized in that, The method used to obtain the multi-scale region set of the image to be labeled is as follows: Feature extraction is performed on the image to be labeled using a feature extraction network to obtain a multi-scale feature pyramid; Superpixel segmentation is performed on the feature maps of each scale in the multi-scale feature pyramid to generate a set of regions of the corresponding scale. The multi-scale region set includes the set of regions of the corresponding scale. The scale-aligned features are propagated and fused into the fine-grained regions corresponding to the child nodes to obtain the cross-level fused features. The specific method is as follows: The features of the coarse-grained region corresponding to the parent node are upsampled and weighted according to a preset cross-level fusion coefficient, and then superimposed and fused with the current features of the fine-grained region corresponding to the child node.

7. The image region annotation method based on visual semantic relationship graph according to claim 1, characterized in that, The method used to construct the initial semantic relationship graph based on the multi-level node features is as follows: Based on the spatial adjacency relationship between image regions in the multi-scale region set, the initial connection weights between nodes in the initial semantic relationship graph are determined. Among them, nodes corresponding to spatially adjacent image regions have an initial connection weight greater than zero, and nodes corresponding to spatially non-adjacent image regions have an initial connection weight of zero.

8. The image region annotation method based on visual semantic relationship graph according to claim 1, characterized in that, The method used to determine the semantic annotation results of each image region in the image to be annotated is as follows: The aggregated features and the cross-level fusion features are fused to obtain the final feature representation of each image region; The final feature representation is input into the classifier to predict the semantic category label of each image region; Based on the semantic category labels of each image region, a semantic annotation map of the image to be annotated is generated.

9. An image region annotation system based on visual semantic relationship graphs, characterized in that, The image region annotation system is used to execute the image region annotation method according to any one of claims 1-8, including: The feature extraction module is used to obtain a multi-scale region set of the image to be labeled, and to extract multi-level node features of each image region in the multi-scale region set; The graph construction and update module is used to construct an initial semantic relationship graph based on the multi-level node features. The initial semantic relationship graph includes nodes and initial connection weights between nodes. The relation construction module is used to dynamically update the connection weights in the initial semantic relation graph based on the node features of the current iteration layer in the iterative layer of the graph neural network, so as to obtain a dynamic semantic relation graph. The relation-aware aggregation module is used to perform differential aggregation of the features of neighboring nodes based on the semantic relationship categories between nodes in the dynamic semantic relationship graph, and to obtain aggregated features by adopting the corresponding aggregation strategy. The cross-level propagation module is used to propagate and fuse the features of the coarse-grained region to the corresponding fine-grained region based on the hierarchical mapping relationship between coarse-grained and fine-grained regions in the multi-scale region set, so as to obtain cross-level fused features. The annotation module is used to determine the semantic annotation results of each image region in the image to be annotated based on the aggregated features and the cross-level fusion features.

Images