A user-intention-driven controllable image segmentation and semantic capturing method
By constructing a scene graph and using a bimodal Transformer to generate multimodal data, the problems of insufficient understanding of user intent and single results in existing technologies are solved, and efficient and accurate output of multimodal image segmentation and semantic capture is achieved.
Patent Information
- Application Number
- CN202510075951.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-16
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2045-01-16
AI Technical Summary
Existing image content understanding algorithms cannot fully understand user intent, and the generated results are singular and lack multimodal collaborative output, failing to meet the diverse needs of practical applications.
By acquiring input images and user visual cues, a scene graph is constructed and user intent is mapped. Multimodal data is generated using a bimodal Transformer, and the alignment of visual masks and text pairs is optimized through multi-entity contrast learning to generate various multimodal results.
It enables the dynamic generation of multimodal image segmentation and semantic capture results based on user intent, improving generation efficiency and quality, enhancing segmentation accuracy and semantic capture consistency, and meeting diverse user needs.
Smart Images
Figure CN119904868B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image content understanding in artificial intelligence, and in particular to a user-intention-driven controllable image segmentation and semantic capture method. BACKGROUND
[0002] Image content understanding algorithms output visual representations (such as target boxes, object masks, etc.) or text representations that describe semantic information by deeply mining visual features in images. The input of user intention can help existing image content understanding algorithms to more accurately represent the semantic information that users are interested in. Current image content understanding algorithms can output single-modal semantic representation information under the guidance of user-provided intention representation information, but there are a series of defects. For example, existing algorithms mainly generate single-modal (visual or text) representation results, lack of multi-modal collaborative image content understanding output, and multi-modal output can more intuitively and comprehensively represent the content that users are interested in compared to single-modal output. In addition, the existing results do not fully understand the user's intention and the output result is single, which cannot provide users with diverse and selectable results. In actual applications, according to user prompts, the system fully understands the user's intention, and translates the image content into multi-modal and diverse results, which has broad application prospects, such as medical diagnosis, automatic driving, etc. SUMMARY
[0003] The purpose of the present application is to provide a user-intention-driven controllable image segmentation and semantic capture method to solve the problems in the background art.
[0004] To achieve the above-mentioned purpose, the present application provides a user-intention-driven controllable image segmentation and semantic capture method, comprising the following steps:
[0005] S1, obtaining an input image and user visual prompt information;
[0006] S2, constructing a scene graph based on the input image obtained in S1 and mapping the user intention into the scene graph according to the user visual prompt information obtained in S1, and extracting local scene graph features related to the intention;
[0007] S3, inputting the local scene graph features related to the intention obtained in S2 and the labeled double-modal (visual mask and text) data into a diffusion model with a double-modal Transformer as the core to generate related double-modal data, and the double-modal data is a visual mask and text pair;
[0008] S4, optimizing and aligning the visual mask and text pair obtained in step S3 using a multi-entity contrastive learning loss.
[0009] Preferably, S1 comprises:
[0010] S11, receiving an input image provided by a user, normalizing the size of the input image to 1024x1024;
[0011] S12, using the first noun of each text as the center object and using the corresponding bounding box as the user visual cue information.
[0012] Preferably, S2 comprises:
[0013] S21, constructing a global scene graph for the input image;
[0014] S22, extracting a subgraph according to the user visual cue, retaining the nodes and edges connected to the center object of the user visual cue;
[0015] S23, constructing an adapter to optimize the extracted subgraph, predicting the objects in the subgraph that are related to the text pair and the visual mask, and filtering out irrelevant objects;
[0016] S24, adjusting the order of the nodes in the subgraph to be consistent with the order of the objects described in the text;
[0017] S25, combining the optimized node features and edge features to generate intent-related local scene graph features.
[0018] Preferably, S23 comprises:
[0019] S231, updating the node features of the subgraph using a self-attention block;
[0020] S232, mapping layer filtering out the nodes related to the target according to the relevance score of the nodes with respect to the labeled mask pair;
[0021] S233, training the adapter optimization node screening using binary cross-entropy (BCE) loss;
[0022] Preferably, S24 comprises:
[0023] S241, mapping the node features to a predicted ranking matrix using a multi-layer fully connected network;
[0024] S242, minimizing the difference between the predicted ranking matrix in S241 and the real ranking matrix using cross-entropy loss;
[0025] S243, filtering out the relevant nodes and adjusting the order of the nodes.
[0026] Preferably, S3 comprises:
[0027] S31, adding Gaussian noise to the original text sample to obtain a noisy text, and obtaining noisy text features through a text encoder;
[0028] S32, inputting the noisy text features and the obtained local scene graph features into the scene graph guided bimodal Transformer, and then generating text features for text prediction and visual features for mask prediction;
[0029] S33, using a text decoder to decode the text features for text prediction to generate a final text description;
[0030] S34. Use the Mask2Former network structure to generate category-independent binary region masks.
[0031] Preferably, generating text features for text prediction in S32 includes using noisy text features as query input, local scene graph features as key and value input, and obtaining updated text features through multi-head attention, cross self-attention layer and feedforward network. The generation process is as follows:
[0032] f' c =MultiHead s (f c ,f c ,f c );
[0033] u c =FFN(f' c +MultiHead c (f' c ,f g ,f g ));
[0034] h c =φ g (u c +f c );
[0035] Among them, MultiHead s and MultiHead c Represent the multi-head attention layer and the cross self-attention layer respectively; FFN represents the feedforward layer; φ(·) represents the mlp layer; the generated h c Represents refined text features.
[0036] Preferably, generating visual features for mask prediction in S32 includes using local scene graph features as query input, noisy text features as key and value input at the same time, and generating visual features consistent with the text description through multi-head attention, cross self-attention layer and feedforward network for mask prediction.
[0037] Preferably, S4 includes:
[0038] S41, in-sample contrast learning is performed, and an in-sample contrast loss is defined as a sum of a mask-text and a text-mask contrast loss;
[0039] S42, inter-sample contrast learning is performed, and a relationship between a mask and a word across several samples is considered, and an inter-sample contrast loss is defined as a sum of a mask-text and a text-mask contrast loss.
[0040] Therefore, the present application has the following beneficial effects by adopting the above-mentioned user-intention-driven controllable image segmentation and semantic capture method:
[0041] (1) A prompt-centered scene graph adapter is adopted, visual prompt information of a user is introduced into a generation and optimization process of a scene graph, different user requirements can be flexibly coped with, a structured scene graph related to the user prompt is dynamically generated, precise mapping of a user intention and filtering of redundant information are realized;
[0042] (2) A diffusion model with a dual-modal Transformer as a core guided by a scene graph is adopted, noise reduction generation is performed in a diffusion process, and scene graph features and text features are integrated into a generation process at each step, collaborative generation of semantic description and segmentation masks is realized, inconsistency problems caused by separate generation in traditional methods are effectively overcome, and efficiency and quality of generation are improved;
[0043] (3) A multi-entity contrast learning loss is introduced, a text and segmentation mask alignment problem is solved, and resolution capability and alignment of a generation result are further improved;
[0044] (4) Visual prompt information of a user is allowed to be mapped into multiple scene graphs, multiple multi-modal representation results are generated, and therefore, multiple image understanding schemes can be provided to a user, and flexible selection of a user is facilitated.
[0045] The technical solutions of the present application will be further described in detail below with the aid of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS
[0046] Figure 1 is an input and output schematic diagram of an embodiment of the present application;
[0047] Figure 2 is a structural schematic diagram of an embodiment adopted by the present application;
[0048] Figure 3 is a flow schematic diagram of an embodiment of the present application;
[0049] Figure 4 is a structural schematic diagram of a scene graph guided dual-modal Transformer module adopted by the present application;
[0050] Figure 5The generation effect of the embodiment of the present application. DETAILED DESCRIPTION
[0051] The following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.
[0052] Referring to Figure 1 , the present application provides an embodiment method, which maps the user's hints to the scene graph through a hint-centered scene graph adapter based on the input image and the simple input (such as the bounding box of the object) provided by the user, effectively captures the user's intention, and then adopts a scene graph guided diffusion process of a dual-modal Transformer to simultaneously generate multiple multi-modal semantic description pairs (including image segmentation visual masks and matching text descriptions). The technical structure diagram of the present application is shown in Figure 2 .
[0053] As shown in Figure 3 , a user-intention-driven controllable image segmentation and semantic capture method at least includes the following operation processes:
[0054] Step S1, acquiring an input image and user visual hint information. Specifically, it includes:
[0055] S11, receiving an input image provided by the user and normalizing its size to 1024x1024.
[0056] S12, approximately using the first noun of each text as the center object and using its corresponding bounding box as the user visual hint information. The hint information is the key for the model to capture the user's intention, which is used to guide the subsequent scene graph generation and optimization process.
[0057] Step S2, constructing a scene graph based on the image acquired in S1 and mapping the user's intention to the scene graph according to the user visual hint information acquired in S1, and extracting local scene graph features related to the intention. It includes:
[0058] S21, constructing a global scene graph for the image;
[0059] S22, extracting a subgraph according to the user visual hint information, and only retaining the nodes and edges connected to the center object of the user visual hint;
[0060] S23, constructing a scene graph adapter to optimize the extracted subgraph, predicting objects in the subgraph related to the visual mask-text pair, and filtering out irrelevant objects;
[0061] Specifically, the scene graph adapter consists of several self-attention blocks and a mapping layer. First, the node features of the subgraph are fed into the self-attention block to generate updated features. Subsequently, the mapping layer predicts a node relevance score, which represents the relevance of the node with respect to the visual mask-text pair, and calculates its intersection over union (IOU) with the corresponding node in the subgraph. If the IOU exceeds a threshold, the relevance score is assigned to 1, otherwise it is assigned to 0. As a result, the adapter can effectively predict relevant objects in the subgraph while filtering out irrelevant objects.
[0062] S24, adjusting the order of subgraph nodes to make it consistent with the order of objects described in the text;
[0063] The subgraph node order adjustment method specifically includes: using a multi-layer fully connected network to map node features into a sorting matrix; using cross-entropy loss to minimize the gap between the predicted sorting matrix and the true sorting matrix; screening out relevant nodes and adjusting the node order to obtain a refined node feature.
[0064] S25. Combine the optimized node features and edge features to generate the final scene graph features related to the user intention.
[0065] Step S3, such as Figure 3 As shown in the figure, the intent-related scene graph features obtained by S2 and the annotated bimodal (visual mask and text) data are input into the diffusion model with the bimodal Transformer as the core to generate relevant bimodal data, namely visual mask and text pairs. Specifically, it includes:
[0066] S31, adding Gaussian noise to the original text sample, and obtaining the features of the noisy text data through a text decoder;
[0067] Specifically, Gaussian noise is added to the text samples from time step t=0 to t, and then a reverse denoising process is performed to recover the annotations. Unlike existing diffusion models, the present invention introduces a scene graph feature to guide the denoising process.
[0068] S32, inputting the noise-added text features and the obtained scene graph features into a scene graph-guided bimodal Transformer, and then generating text features for text prediction and visual features for mask prediction;
[0069] Specifically, at each time step t, the noisy annotation features and the scene graph features obtained in step S2 are input into the scene graph guided bimodal Transformer, and then the visual features for mask prediction are generated. Figure 4As shown, the scene graph guided dual-modal Transformer of the embodiment of the present application contains N blocks, each of which contains a multi-head self-attention layer, a multi-head cross-attention layer and a feed-forward layer.
[0070] For generating visual features for text prediction, noisy text features f c As query input, scene graph features f g At the same time, as key and value input into the block, after multi-head attention, cross self-attention layer and feed-forward network, the updated text features are obtained, and the generation process is as follows:
[0071] f' c = MultiHead s (f c , f c , f c ),
[0072] u c = FFN (f' c + MultiHead c (f' c , f g , f g )),
[0073] h c = φ g (u c + f c )
[0074] Where MultiHead s and MultiHead c represent multi-head attention layer and cross self-attention layer respectively, FFN represents feed-forward layer, and φ(·) represents mlp layer. The generated h c represents a refined text feature, which is either passed to the subsequent block or used for text prediction. The final noise-free text feature generates a natural language description through the text decoder, which is complete in semantics and highly relevant to the target region.
[0075] And for generating visual features for mask prediction, scene graph features f g are used as query input, noisy text description features f c At the same time, as key and value input into the block, after multi-head attention, cross self-attention layer and feed-forward network, visual features consistent with the text description are generated for mask prediction, and the generation process is as follows:
[0076] f' g = MultiHead s (f g , f g , fg ),
[0077] u g =FFN(f' g +MultiHead c (f' g ,f c ,f c )),
[0078] h g =φ g (u g +f g )
[0079] generated h g Under the guidance of image annotation features, visual features for mask prediction can be accurately provided, which are either passed to subsequent blocks or used for mask prediction.
[0080] S33, decode the text features for text prediction using a text decoder to generate a final text description.
[0081] S34, generate a binary region mask independent of the class using a Mask2Former network structure.
[0082] Step S4, using multi-entity contrastive learning to optimize and align the visual mask-text pairs obtained in step S3, including intra-sample contrastive learning and inter-sample contrastive learning.
[0083] S41, intra-sample contrastive learning, the intra-sample contrastive loss is defined as the sum of mask-word and word-mask contrastive losses.
[0084] Specifically, intra-sample contrastive learning aligns the mask and word embeddings within a single sample, maximizing the mutual information between the mask entity and the matching word. Specifically, it includes: calculating the mask-word matching relationship to obtain a set of word embeddings matching the current mask and a set of mask embeddings matching the current word embedding, using the InfoNCE loss function to calculate the contrastive loss of the mask embedding and its matching word embedding set, and the contrastive loss of the word embedding and its matching mask embedding set, and the total intra-sample contrastive loss is defined as the sum of the mask-word and word-mask contrastive losses.
[0085] S42, inter-sample contrastive learning, considering the relationship between masks and words across several samples, the inter-sample contrastive loss is defined as the sum of mask-text and text-mask contrastive losses.
[0086] The contrast learning between samples is performed, aiming to capture the global interaction information between different samples. Specifically, the global matching score is calculated for each pair of mask and word, in each batch, the mask-text pair is randomly selected, and the contrast loss of the global matching score is optimized, similarly, the total inter-sample contrast loss includes the contrast loss between mask-text and text-mask.
[0087] Figure 5 A visual example effect diagram of implementing an embodiment of the present application is shown. In general, the present application supports generating multiple bimodal (visual mask and text) results, which can provide diversified but accurate results through a simple prompt, effectively reducing user burden and improving overall performance.
[0088] Therefore, the present application adopts the above-mentioned user intention driven controllable image segmentation and semantic capture method, solves the problems that the existing image segmentation and semantic capture method is difficult to capture user intention, the segmentation result and the semantic text result are single and insufficiently associated, the multi-modal output processing capability of the complex scene is limited, and greatly improves the image segmentation accuracy and the consistency of semantic capture.
[0089] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.
Claims
1. A user intent driven controllable image segmentation and semantic capture method, characterized in that, The method comprises the following steps: S1, obtaining an input image and user visual prompt information; S2, constructing a scene graph based on the input image obtained in S1 and mapping user intent into the scene graph according to the user visual prompt information obtained in S1, and extracting local scene graph features related to the intent; comprising: S21, constructing a global scene graph of the input image; S22, extracting a subgraph according to the user visual prompt, and retaining nodes and edges connected to the central object of the user visual prompt; S23, constructing an adapter to optimize the extracted subgraph, predicting objects in the subgraph that are related to the visual mask and the text pair, and filtering out irrelevant objects; comprising: S231, updating the node features of the subgraph using a self-attention block; S232, screening out nodes related to the target according to the relevance score of the nodes to the labeled mask pair by the mapping layer; S233, training the adapter optimization node screening using a binary cross-entropy (BCE) loss; S24, adjusting the order of the nodes in the subgraph to be consistent with the order of the objects described in the text; comprising: S241, mapping the node features into a predicted ranking matrix using a multi-layer fully connected network; S242, minimizing the difference between the predicted ranking matrix in S241 and the real ranking matrix using a cross-entropy loss; S243, screening out related nodes and adjusting the order of the nodes; S25, combining the optimized node features and edge features to generate local scene graph features related to the intent; S3, inputting the local scene graph features related to the intent obtained in S2 and the labeled double-modal data into a diffusion model with a double-modal Transformer as the core to generate related double-modal data, wherein the double-modal data is a visual mask and a text pair; S4, optimizing and aligning the visual mask and the text pair obtained in step S3 using a multi-entity contrastive learning loss.
2. The user intent driven controllable image segmentation and semantic capture method of claim 1, wherein, S1 comprises: S11, receiving an input image provided by a user and standardizing the size to 1024x1024; S12, using the first noun of each text as the central object and using the corresponding bounding box as the user visual prompt information.
3. The user intent driven controllable image segmentation and semantic capture method of claim 1, wherein S3 Comprising: S31, adding Gaussian noise to the original text sample to obtain a noisy text, and obtaining noisy text features through a text encoder; S32, inputting the noisy text features and the obtained local scene graph features into a scene graph guided double-modal Transformer, and then generating text features for text prediction and visual features for mask prediction; S33, decoding the text features for text prediction using a text decoder to generate a final text description; S34, generating a class-agnostic binary region mask using a Mask2Former network structure.
4. The user intent driven controllable image segmentation and semantic capture method of claim 3, wherein: In S32, the generation of the text features for text prediction comprises using the noisy text features as query input, and the local scene graph features as key and value input, passing through a multi-head attention, a cross self-attention layer and a feedforward network to obtain updated text features, and the generation process is as follows: f′ c = MultiHead s (f c ,f c ,f c ); u c = FFN(f c + MultiHead c (f c ,f g ,f g )) ; h c = φ g (u c + f c ); where MultiHead s and MultiHead c represent multi-head attention layer and cross self-attention layer, respectively; FFN denotes feed-forward layer; φ(·) denotes mlp layer; f c is the noisy text feature; f g is the local scene graph feature; h c generated represents the refined text feature.
5. The user intent driven controllable image segmentation and semantic capture method of claim 4, wherein: The visual features for mask prediction generated in S32 include using local scene graph features as query input, noisy text features as input of key and value, passing through a multi-head attention, cross self-attention layer and a feedforward network, generating visual features consistent with the text description for mask prediction, and the generation process is as follows: f′ g = MultiHead s (f g ,f g ,f g ); u g = FFN(f g + MultiHead c (f g ,f c ,f c )) ; h g = φ g (u g + f g ); The generated h g represent visual features.
6. The user intent driven controllable image segmentation and semantic capture method of claim 1, wherein S4 Comprise: S41, intra-sample contrast learning is performed, and the intra-sample contrast loss is defined as the sum of the mask-word and word-mask contrast losses; S42, inter-sample contrast learning is performed, and the relationship between the mask and the word across several samples is considered, and the inter-sample contrast loss is defined as the sum of the mask-text and text-mask contrast losses.
Citation Information
Patent Citations
Visual positioning method based on multi-modal feature alignment
CN117934803A