Image anaphora segmentation method based on autoregression vertex generation and language structure guidance

Abstract

The invention relates to the technical field of image segmentation, in particular to an image anaphora segmentation method based on autoregression vertex generation and language structure guidance, and the method comprises the steps: obtaining an input image and a corresponding anaphora query statement; acquiring multi-scale visual features of the input image by using a pre-trained visual encoder; obtaining language features of the anaphora query statement by using a pre-trained language encoder; constructing an image semantic scene graph based on the multi-scale visual features; constructing a query language dependency graph based on the language features; utilizing a graph alignment mechanism and a feature alignment mechanism to realize structured multi-modal fusion of the image semantic scene graph and the query language dependency graph; based on a structured multi-modal fusion result, generating a contour point sequence of the target instance by adopting a language-guided autoregression instance generation method; generating a segmentation mask of the target instance according to the contour point sequence; and outputting the segmentation mask as an image anaphora segmentation result, thereby realizing remarkable performance improvement.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and more specifically to an image reference segmentation method guided by autoregressive vertex generation and language structure. Background Art

[0002] With the rapid development of computer vision and natural language processing technology, image referent segmentation, as a cutting-edge task integrating vision and language, has received extensive attention from academia and industry in recent years. This task requires the model to accurately segment the target object in a given image based on natural language descriptions. Its application prospects cover multiple fields such as human-computer interaction, intelligent image editing, and visual question answering.

[0003] Existing image referential segmentation methods can be mainly divided into two categories: two-stage based methods and end-to-end based methods. The two-stage method usually uses an object detector to extract candidate regions first, and then selects the most matching region for segmentation based on the language description. Although this type of method has a clear structure, it is often limited to predefined target categories and has difficulty in processing open domain referential expressions. The end-to-end method attempts to predict the segmentation mask directly from the image and language input, and usually adopts multimodal feature fusion and attention mechanism to achieve cross-modal alignment.

[0004] However, there are still some key issues that need to be addressed in the existing technology. First, most methods perform poorly when dealing with complex referential expressions, especially when the query contains multiple objects and complex relationships between them. Second, existing models often have difficulty accurately capturing fine-grained semantic information in the language, resulting in insufficient accuracy when dealing with queries containing detailed descriptions. In addition, many methods also face challenges when dealing with objects with complex shapes or partially occluded, making it difficult to generate accurate instance outlines.

[0005] More importantly, existing methods generally lack deep semantic understanding of image content and language queries. They usually use simple feature splicing or attention mechanisms to fuse visual and language information. This shallow fusion makes it difficult to fully utilize the structured semantic information of the two modalities. At the same time, most methods use pixel-level prediction when generating segmentation masks. This method makes it difficult to ensure the geometric consistency and topological correctness of the generated results, especially when dealing with objects with complex shapes. Summary of the invention

[0006] To address the above problems, this paper proposes an image referential segmentation method guided by autoregressive vertex generation and language structure. This method aims to achieve precise understanding of complex referential queries and accurate segmentation of target instances, with a particular focus on how to effectively integrate the structured semantic information of vision and language, and how to generate geometrically consistent and semantically relevant instance segmentation results.

[0007] The present invention provides an image reference segmentation method guided by autoregressive vertex generation and language structure, comprising:

[0008] The acquisition steps include:

[0009] Get the input image and the corresponding reference query statement;

[0010] Acquire multi-scale visual features of the input image using a pre-trained visual encoder;

[0011] Using a pre-trained language encoder to obtain language features of the reference query sentence;

[0012] Processing steps include:

[0013] Based on the multi-scale visual features, construct an image semantic scene graph;

[0014] Based on the language features, construct a query language dependency graph;

[0015] Using a graph alignment mechanism and a feature alignment mechanism, structured multimodal fusion of the image semantic scene graph and the query language dependency graph is achieved;

[0016] Based on the result of the structured multimodal fusion, a language-guided autoregressive instance generation method is used to generate a contour point sequence of the target instance;

[0017] Output steps include:

[0018] generating a segmentation mask of the target instance according to the sequence of contour points;

[0019] The segmentation mask is output as a result of image segmentation.

[0020] Preferably, the pre-trained visual encoder is the encoder part of a frozen SAM (SegmentAnything Model) model.

[0021] Preferably, the pre-trained language encoder is a frozen BERT model.

[0022] Preferably, the step of constructing an image semantic scene graph specifically includes:

[0023] Processing the multi-scale visual features using a pre-trained scene graph generation model;

[0024] Extract the target object and its attribute information in the image;

[0025] Identify relationships between target objects;

[0026] An image semantic scene graph is constructed based on the target object, attribute information and relationships.

[0027] Preferably, the step of constructing a query language dependency graph specifically includes:

[0028] Performing dependency syntactic analysis on the reference query sentence;

[0029] Extract syntactic dependencies between words;

[0030] A query language dependency graph is constructed based on the syntactic dependency relationship.

[0031] Preferably, the step of structured multimodal fusion specifically includes:

[0032] Achieving node alignment between the image semantic scene graph and the query language dependency graph through a graph alignment mechanism;

[0033] Fine-grained alignment of visual features and language features is achieved through feature alignment mechanism;

[0034] The word graph propagation mechanism based on the dependency tree structure realizes information fusion at the semantic level.

[0035] Preferably, the graph alignment mechanism comprises:

[0036] Approximate node embeddings via factorized similarity matrices;

[0037] A greedy matching algorithm is used to select the most similar node pairs;

[0038] Generate semantic correspondence between image content and query language.

[0039] Preferably, the language-guided autoregressive instance generation method comprises:

[0040] Using language features to guide the rough localization of target instances;

[0041] Use Transformer decoder autoregression to generate instance contour point sequence;

[0042] Use language features as global context information in the decoding process;

[0043] Use dynamic convolutional masks to dynamically adjust the segmentation strategy based on language features.

[0044] Preferably, it also includes an iterative optimization step:

[0045] Iterate the autoregressive instance generation method multiple times;

[0046] In each iteration, the previous generation result is used as additional input;

[0047] Gradually refine and optimize the segmentation results.

[0048] Preferably, a multi-task learning step is also included:

[0049] On the basis of the main segmentation task, a bounding box regression task is added;

[0050] Added instance center prediction task;

[0051] By jointly optimizing multiple related tasks, the representation learning ability of the model is enhanced.

[0052] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0053] The method of the present invention achieves significant performance improvement by innovatively combining pre-trained visual-language models, structured multimodal fusion, language-guided autoregressive instance generation and other technologies. First, by constructing an image semantic scene graph and a query language dependency graph, this method can better capture the structured semantic information in images and languages. Secondly, the graph alignment mechanism and feature alignment mechanism are used to achieve deep fusion of visual and language information, so that the model can more accurately understand complex referential relationships. Furthermore, the language-guided autoregressive instance generation method can generate segmentation results that are geometrically consistent and highly consistent with the language description, especially when dealing with complex shapes or occluded targets.

[0054] In addition, the method of the present invention also introduces iterative optimization and multi-task learning mechanisms to further improve the accuracy of segmentation and the generalization ability of the model. By refining the segmentation results through multiple iterations, the model can generate more accurate instance contours. Multi-task learning enables the model to learn richer and more general feature representations, improving the performance of the model when processing novel referential expressions and unseen target categories.

[0055] In general, the method of the present invention has achieved technological breakthroughs in many aspects. It can not only better understand complex reference queries, but also generate high-quality segmentation results that are geometrically consistent and semantically relevant. This method has achieved significant performance improvements on existing reference segmentation benchmark datasets, and has also shown great potential in dealing with more complex and ambiguous real-world scenarios. In addition, the modular design of this method enables it to have good scalability and adaptability, laying a solid foundation for future technology iterations and practical applications. By solving the key problems in the prior art, the present invention provides a new and effective solution for the image reference segmentation task, which is expected to promote the further development and wide application of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The figure is a flow chart of the method of the present invention.

[0057] Figure 2 Construct a logic block diagram for the image semantic scene graph of the present invention.

[0058] Figure 3 A logical block diagram is constructed for the query language dependency graph of the present invention.

[0059] Figure 4 This is the logic block diagram of the graph alignment and multimodal fusion of the present invention.

[0060] Figure 5 A logic block diagram is generated for the autoregressive example of the present invention.

[0061] Figure 6 It is an iterative optimization logic block diagram of the present invention.

[0062] Figure 7 This is the logic block diagram of the multi-task learning of the present invention. DETAILED DESCRIPTION

[0063] Please refer to Figure 1-7 The present invention provides an image referential segmentation method guided by autoregressive vertex generation and language structure. The method combines the latest computer vision and natural language processing technologies to achieve accurate understanding of complex query statements and accurate segmentation of target instances.

[0064] Specifically, the method of the present invention comprises the following steps:

[0065] First, in the acquisition step, the method acquires an input image and a corresponding referential query statement. The input image here can be a natural image of various resolutions and contents, and the referential query statement is a natural language expression describing the target object. For example, the input image may be a street view photo, and the referential query statement may be "a lady in a red coat crossing the street".

[0066] Next, the method uses a pre-trained visual encoder to obtain multi-scale visual features of the input image. Preferably, the present invention uses the encoder part of a frozen SAM (SegmentAnything Model) model as the visual encoder. The SAM model is pre-trained on a large-scale segmentation dataset and has a powerful general segmentation capability. By using the SAM encoder, the method can extract rich multi-scale visual features that contain local details and global semantic information of the image.

[0067] At the same time, the method also uses a pre-trained language encoder to obtain the language features of the reference query statement. In a preferred embodiment, the present invention uses a frozen BERT model as a language encoder. The BERT model has excellent language understanding capabilities through pre-training of large-scale corpus. Through BERT encoding, each word in the query statement is mapped to a high-dimensional vector space, which not only contains the semantics of the word itself, but also encodes contextual information.

[0068] In the processing step, the method first constructs an image semantic scene graph based on the multi-scale visual features. Specifically, the method uses a pre-trained scene graph generation model to process multi-scale visual features, extract target objects and their attribute information in the image, and identify the relationship between target objects. For example, for the street view photo mentioned above, the scene graph may contain nodes such as "lady", "red coat", and "road", as well as relationship edges such as "wearing" and "on top". This structured scene representation lays the foundation for subsequent multimodal fusion.

[0069] Next, the method constructs a query language dependency graph based on the language features. Specifically, the method performs dependency syntactic analysis on the referential query sentence and extracts the syntactic dependency relationship between words. For example, for the query "a lady in a red coat crossing the road", the dependency graph will capture the relationship that "ladies" is the central word, "cross" and "wear" modify "ladies" respectively, and "red" modifies "coat". This explicit representation of language structure helps to understand the query intent more accurately.

[0070] After obtaining the structured representation of the image and language, the method uses the graph alignment mechanism and the feature alignment mechanism to achieve the structured multimodal fusion of the image semantic scene graph and the query language dependency graph. This step is one of the core innovations of the present invention, which achieves the deep fusion of visual and language information by considering the alignment of the structure and feature levels at the same time.

[0071] Specifically, the graph alignment mechanism first approximates node embeddings by factoring the similarity matrix. Suppose the node set of the image scene graph is The node set of the query dependency graph is We can calculate the similarity matrix S∈R between nodes m×n :

[0072]

[0073] where sin(·,·) is the cosine similarity function. Then, we get the low-dimensional embedding of the node by matrix decomposition:

[0074]

[0075] Where P i ∈R m×d and P t ∈R n×d are the d-dimensional embeddings of image and text nodes respectively.

[0076] Next, this method uses a greedy matching algorithm to select the most similar node pair. Specifically, we traverse P i For each row in , find the P that is most similar to itt , thereby establishing a semantic correspondence between the image content and the query language.

[0077] In the feature alignment mechanism, this method uses the attention mechanism to achieve fine-grained alignment of visual features and language features. Let the visual feature be F v ∈R h×w×c , the language feature is F l ∈R l×c , we calculate the attention weights:

[0078]

[0079] Where W q and W k is the learnable parameter matrix, d k is the dimension of attention. Then, we can get the aligned visual features:

[0080] F′ v =AF l W v ,

[0081] Where W v is also a learnable parameter matrix.

[0082] Through the above steps, this method achieves a deep fusion of visual and language information, providing rich multimodal context for subsequent instance generation.

[0083] In the instance generation stage, based on the results of the structured multimodal fusion, the method uses a language-guided autoregressive instance generation method to generate a sequence of contour points of the target instance. This step makes full use of the multimodal information fused previously to ensure that the generated instance is highly consistent with the query description.

[0084] Specifically, this method first uses language features to guide the rough positioning of the target instance. We use a multi-layer perceptron (MLP) network to process the fused features and generate the center point heat map and size prediction of the target instance:

[0085] H c , s = MLP(F f ),

[0086] Among them, F f is the fused feature, H c is the center point heat map, and s is the predicted instance size. Next, this method uses the Transformer decoder to autoregressively generate the instance contour point sequence. Let the contour point sequence be {p1, p2, …, p N}, each point p i =(x i ,y i) represents the normalized coordinates. The decoding process can be expressed as: i = Transformer(p ＜ i,F f , F l ),

[0087] where p ＜i represents the previously generated point, F f is the fusion feature, F l is a language feature. It is worth noting that we use the language feature F l As global context information for the decoding process, it ensures that the generated outline is consistent with the language description.

[0088] In order to further improve the accuracy of segmentation, this method uses a dynamic convolution mask to dynamically adjust the segmentation strategy according to language features. Specifically, we generate the convolution kernel parameters based on language features:

[0089] W k =MLP(F l ),

[0090] Then use this dynamically generated convolution kernel to process the feature map:

[0091] M=σ(W k *F f ),

[0092] Where * represents the convolution operation, σ is the sigmoid activation function, and M is the final segmentation mask.

[0093] In the output step, the method generates a segmentation mask of the target instance according to the contour point sequence, and outputs it as the final result of the image representative segmentation.

[0094] It is worth mentioning that the method of the present invention also includes an iterative optimization step. Specifically, the method iteratively executes the autoregressive instance generation method multiple times, and in each iteration, the previous generation result is used as an additional input, thereby gradually refining and optimizing the segmentation result. This iterative optimization strategy significantly improves the accuracy and robustness of the segmentation.

[0095] In addition, in order to further enhance the representation learning ability of the model, this method also introduces multi-task learning. In addition to the main segmentation task, this method also adds bounding box regression tasks and instance center prediction tasks. These auxiliary tasks not only provide additional supervision signals, but also promote the model to learn richer and more general feature representations.

[0096] In general, the image referential segmentation method guided by autoregressive vertex generation and language structure proposed in the present invention achieves accurate understanding of complex referential queries and accurate segmentation of target instances by combining innovative technologies such as pre-trained visual-language models, structured multimodal fusion, and language-guided autoregressive instance generation. The method performs well in various complex scenarios and provides a new and effective solution for the image referential segmentation task. The method of the present invention further refines the construction process of the image semantic scene graph. Specifically, in the step of constructing the image semantic scene graph, the method first uses a pre-trained scene graph generation model to process the multi-scale visual features. The scene graph generation model used here is usually a deep neural network pre-trained on a large-scale image-scene graph dataset, such as models such as VCTree or MSDN. These models can effectively capture the semantic information in the image and the relationship between objects.

[0097] When processing multi-scale visual features, this method adopts a feature pyramid network (FPN) structure to make full use of information at different scales. Preferably, FPN contains a top-down path and lateral connections, which can be expressed as:

[0098] P l =Conv(C l +Upsample(P l+1 )),

[0099] Among them C l is the feature map of layer l, P l is the corresponding FPN feature. This fusion of multi-scale features enables the model to focus on both large-scale semantic information and small-scale detail features.

[0100] Next, the method extracts the target object and its attribute information in the image from the processed features. This step usually involves two subtasks: object detection and attribute classification. In object detection, the method uses the region-based convolutional neural network (R-CNN) family of algorithms, such as Faster R-CNN or Mask R-CNN. These algorithms first generate candidate regions, and then classify and regress bounding boxes for each region.

[0101] For the extraction of attribute information, this method applies a multi-label classifier on each detected object. Let f o is the feature vector of the target, and the attribute prediction can be expressed as:

[0102] p(a|o)=σ(W a f o +b a ),

[0103] Where W a and b ais a learnable parameter and σ is a sigmoid activation function. In this way, for each target, we can get a series of attributes and their probabilities.

[0104] When identifying the relationship between target objects, this method adopts a relational reasoning module based on the attention mechanism. Given the features f of two objects i and f j , and their spatial position encoding p i and p j , the relation classification can be expressed as:

[0105] r ij =softmax(W r [f i ;f j ;p i ;p j ]+b r ),

[0106] Where [;] represents feature concatenation, W r and b r is a learnable parameter. This approach is able to consider both the visual features and spatial relationships of objects, thus accurately inferring complex semantic relationships.

[0107] Finally, the method constructs an image semantic scene graph based on the target object, attribute information and relationship. The scene graph can be represented as a triple set G = {(s, r, o)}, where s is the subject object, r is the relationship, and o is the object object. Each object node also comes with its attribute information. This structured scene representation provides rich semantic information for subsequent multimodal fusion.

[0108] In the process of constructing the query language dependency graph, the method of the present invention first performs dependency syntactic analysis on the reference query statement. Preferably, the method uses a neural network-based dependency parser, such as a BiLSTM-based or Graph-based parser. These parsers can accurately capture the grammatical structure of a sentence by modeling the dependency relationship between words.

[0109] Specifically, given an input sentence S = {w1, w2, ..., w n}, the goal of dependency parsing is to i Find the head word h i and dependency type r i In the transition-based parsing approach, this process can be modeled as a series of transition actions:

[0110] a t = argmax a score(s, a),

[0111] Where s is the current parsing state and a is a possible transition action. The parser builds the dependency tree by iteratively executing the highest scoring action.

[0112] When extracting syntactic dependencies between words, this method considers not only direct head word relationships, but also more complex path dependencies. For example, for the query "a lady wearing a red coat crossing the road", in addition to direct dependencies such as "lady←wear", this method also extracts path dependencies such as "lady←wear←coat←red". This fine-grained language structure information helps to understand complex referential expressions more accurately.

[0113] Finally, the method constructs a query language dependency graph based on the syntactic dependency relationship. The dependency graph can be expressed as D = (V, E), where V is a set of word nodes and E is a set of dependency edges. Each edge e ij ∈E represents the word w i To its head word w j Preferably, the method also adds part-of-speech information and named entity types of words to the dependency graph, further enriching the language representation.

[0114] When implementing structured multimodal fusion, the method of the present invention adopts an innovative graph alignment mechanism. The core idea of ​​this mechanism is to establish a semantic correspondence between image content and query language while maintaining the graph structure.

[0115] Specifically, the graph alignment mechanism first approximates node embeddings by factoring the similarity matrix. Let the node feature matrix of the image scene graph be X i ∈R m×d , the node feature matrix of the query dependency graph is X t ∈R n×d , where m and n are the number of nodes in the two graphs, and d is the feature dimension. This method first calculates the similarity matrix between nodes:

[0116]

[0117] Then, we get the low-dimensional embedding by matrix decomposition:

[0118]

[0119] Where P i ∈R m×k and P t ∈R n×k is a low-dimensional embedding matrix, and k is the embedding dimension (usually smaller than d). This low-dimensional embedding not only reduces the computational complexity, but also captures the potential semantic relationship between nodes.

[0120] Next, this method uses a greedy matching algorithm to select the most similar node pairs. Specifically, for each node i in the image scene graph, find the query dependency graph node j that is most similar to it:

[0121] j = argmax j cos(P i [i],P t [j]),

[0122] Where cos(·,·) represents the cosine similarity. In this way, we have established a mapping from image nodes to query nodes.

[0123] It is worth noting that the method also considers structural consistency in the matching process. Preferably, the method introduces a structural consistency constraint, that is, if two image nodes are connected in the scene graph, their corresponding query nodes should also be close in the dependency graph. This can be achieved by adding a structural similarity term to the matching score:

[0124] score(i,j)=cos(P i [i],P t [j])+λΣ i′∈N(i) Σ j′∈N(j) cos(P i [i′],P t [j′]),

[0125] Where N(i) represents the neighbor set of node i, and λ is the balancing parameter.

[0126] Through the above graph alignment mechanism, this method generates semantic correspondences between image content and query language. This correspondence not only considers the similarity of node features, but also maintains the consistency of graph structure, providing a solid foundation for subsequent fine-grained alignment and instance generation.

[0127] In general, the method of the present invention achieves a deep fusion of visual and language information through sophisticated image scene graph construction, detailed query language dependency analysis, and an innovative graph alignment mechanism. This fusion makes full use of the structured representation of the two modalities, providing rich contextual information for the subsequent referential target segmentation task, so that complex referential expressions can be understood more accurately and accurate segmentation results can be generated. In the core implementation of the present invention, the language-guided autoregressive instance generation method plays a key role. This method cleverly integrates language information into each stage of instance generation, ensuring that the generated segmentation results are highly consistent with the referential query.

[0128] First, the method uses language features to guide the rough positioning of the target instance. Specifically, in a preferred embodiment, the present invention uses a multi-layer perceptron (MLP) network to process the fused multimodal features. The input of this MLP network is the structured multimodal fusion features obtained in the previous steps, and the output is the center point heat map and size prediction of the target instance. Formally, this process can be expressed as:

[0129] H c , s = MLP(F f ),

[0130] Among them, F f is the fusion feature, H c ∈R H×W is the center point heat map, s∈R 2 is the predicted instance size (height and width). This rough positioning provides valuable prior information for subsequent fine segmentation. Next, one of the core innovations of this method is to use the Transformer decoder to autoregressively generate a sequence of instance contour points. This method is inspired by the recent sequence generation models in the field of computer vision, but the present invention uniquely applies it to the task of referential segmentation. Specifically, this method represents the instance contour as a series of sequential points {p1, p2, ..., p N}, where each point p i =(x i ,y i ) represents the normalized coordinates. The decoding process can be formally expressed as:

[0131] p i = Transformer(p ＜ i,F f , F l ),

[0132] In this formula, p ＜i represents the previously generated point sequence, F f is the fusion feature, F l is a language feature. It is worth noting that this method innovatively transforms the language feature F l As global context information for the decoding process, this ensures that when generating each contour point, the model can fully consider the semantic information of the query language, thereby generating instance contours that are highly consistent with the language description.

[0133] In the specific implementation of the Transformer decoder, the present invention adopts a multi-head attention mechanism and a feedforward neural network. The calculation process of each decoding layer can be expressed as:

[0134]

[0135] Among them, MHA stands for multi-head attention, FFN stands for feedforward neural network, and l stands for layer index. This structure allows the model to make full use of previously generated point information, image features, and language features when generating each point.

[0136] In order to further improve the accuracy and flexibility of segmentation, this method uses a dynamic convolution mask to dynamically adjust the segmentation strategy according to language features. This is another important innovation of the present invention. Specifically, this method first generates convolution kernel parameters according to language features:

[0137] W k =MLP(F l ).

[0138] Then use this dynamically generated convolution kernel to process the feature map:

[0139] M=σ(W k *F f ).

[0140] Where * represents the convolution operation, σ is the sigmoid activation function, and M∈R H×W This dynamic convolution mechanism enables the model to flexibly adjust the segmentation strategy according to different language queries, greatly improving the model's ability to handle complex referential expressions.

[0141] In a preferred embodiment of the present invention, an iterative optimization step is also included. Specifically, the method iteratively executes the above-mentioned autoregressive instance generation method multiple times. In each iteration, the previous generation result is used as an additional input, thereby achieving a gradual refinement and optimization of the segmentation result. Formally, the process of the tth iteration can be expressed as:

[0142] M t =InstanceGen(F f , F l , M t-1 ),

[0143] InstanceGen represents the entire instance generation process, M t-1 is the segmentation result of the previous iteration. This iterative optimization strategy significantly improves the accuracy and robustness of segmentation, especially when dealing with complex shapes or occluded targets. In addition, in order to further enhance the representation learning ability and generalization performance of the model, the present invention introduces a multi-task learning mechanism. In addition to the main segmentation task, this method also adds a bounding box regression task and an instance center prediction task. These auxiliary tasks not only provide additional supervision signals, but also promote the model to learn richer and more general feature representations. Specifically, the bounding box regression task can be expressed as:

[0144] b=MLPb (F f ),

[0145] where b∈R 4 represents the coordinates of the bounding box (upper left corner and lower right corner). The instance center prediction task can be expressed as:

[0146] c=MLP c (F f ),

[0147] where c∈R 2 Indicates the coordinates of the instance center.

[0148] These auxiliary tasks share the feature extraction network with the main segmentation task, but have their own prediction heads. During training, this method uses a multi-task loss function:

[0149] L=L seg +λ1L box +λ2L center ,

[0150] Where L seg , L box and L center are the losses for segmentation, bounding box regression, and center prediction, respectively. λ1 and λ2 are weight coefficients for balancing different tasks.

[0151] Through this multi-task learning strategy, the method of the present invention can learn more robust and universal feature representations, so that it can perform well in various complex scenarios. In particular, this method shows excellent generalization ability when dealing with novel referential expressions or unseen target categories.

[0152] In general, the proposed method of image representative segmentation guided by autoregressive vertex generation and language structure has achieved accurate understanding of complex representative queries and accurate segmentation of target instances by innovatively combining advanced technologies such as pre-trained visual-language model, structured multimodal fusion, language-guided autoregressive instance generation, and multi-task learning. This method has not only achieved significant performance improvement on existing representative segmentation benchmark datasets, but also demonstrated strong potential in processing more complex and ambiguous real-world scenarios, providing a new and effective solution for image representative segmentation tasks.

[0153] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image referential segmentation method guided by autoregressive vertex generation and language structure, characterized in that: include: The acquisition steps include: Get the input image and the corresponding reference query statement; Acquire multi-scale visual features of the input image using a pre-trained visual encoder; Using a pre-trained language encoder to obtain language features of the reference query sentence; Processing steps include: Based on the multi-scale visual features, construct an image semantic scene graph; Based on the language features, construct a query language dependency graph; Using a graph alignment mechanism and a feature alignment mechanism, structured multimodal fusion of the image semantic scene graph and the query language dependency graph is achieved; Based on the result of the structured multimodal fusion, a language-guided autoregressive instance generation method is used to generate a contour point sequence of the target instance; Output steps include: generating a segmentation mask of the target instance according to the sequence of contour points; The segmentation mask is output as a result of image segmentation.

2. The method according to claim 1, characterized in that The pre-trained visual encoder is the encoder part of the frozen SAM model.

3. The method according to claim 1, characterized in that The pre-trained language encoder is a frozen BERT model.

4. The method according to claim 1, characterized in that: The step of constructing the image semantic scene graph specifically includes: Processing the multi-scale visual features using a pre-trained scene graph generation model; Extract the target object and its attribute information in the image; Identify relationships between target objects; An image semantic scene graph is constructed based on the target object, attribute information and relationships.

5. The method according to claim 1, characterized in that The step of constructing the query language dependency graph specifically includes: Performing dependency syntactic analysis on the reference query sentence; Extract syntactic dependencies between words; A query language dependency graph is constructed based on the syntactic dependency relationship.

6. The method according to claim 1, characterized in that The steps of structured multimodal fusion specifically include: Achieving node alignment between the image semantic scene graph and the query language dependency graph through a graph alignment mechanism; Fine-grained alignment of visual features and language features is achieved through feature alignment mechanism; The word graph propagation mechanism based on the dependency tree structure realizes information fusion at the semantic level.

7. The method according to claim 6, characterized in that The graph alignment mechanism includes: Approximate node embeddings via factorized similarity matrices; A greedy matching algorithm is used to select the most similar node pairs; Generate semantic correspondence between image content and query language.

8. The method according to claim 1, characterized in that The language-guided autoregressive instance generation method comprises: Using language features to guide the rough localization of target instances; Use Transformer decoder autoregression to generate instance contour point sequence; Use language features as global context information in the decoding process; Use dynamic convolutional masks to dynamically adjust segmentation strategies based on language features.

9. The method according to claim 8, characterized in that It also includes an iterative optimization step: Iterate the autoregressive instance generation method multiple times; In each iteration, the previous generation result is used as additional input; Gradually refine and optimize the segmentation results.

10. The method according to claim 1, characterized in that It also includes multi-task learning steps: On the basis of the main segmentation task, a bounding box regression task is added; Added instance center prediction task; By jointly optimizing multiple related tasks, the representation learning ability of the model is enhanced.

Images