User intention-driven controllable image segmentation and semantic capture method

Abstract

The invention discloses a user intention-driven controllable image segmentation and semantic capture method, which belongs to the technical field of image content understanding and comprises the following steps: acquiring an input image and user visual prompt information; constructing a scene graph based on the input image, mapping a user intention into the scene graph according to user visual prompt information, and extracting local scene graph features related to the intention; inputting the local scene graph features and the labeled bimodal data into a diffusion model with a bimodal Transform as a core, and generating a related visual mask and text pair; and performing optimization alignment on the obtained visual mask and the text pair by utilizing multi-entity contrast learning loss. According to the user intention-driven controllable image segmentation and semantic capture method provided by the invention, the problems that the user intention is difficult to capture, the segmentation result and the semantic text result are single and insufficient in association, the multi-modal output processing capability of a complex scene is limited and the like in the prior art are solved, and the image segmentation precision and the semantic capture consistency are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image content understanding in artificial intelligence, and in particular to a user intention-driven controllable image segmentation and semantic capture method. Background Art

[0002] Image content understanding algorithms output visual representations (such as target boxes, object masks, etc.) or text representations that describe semantic information by deeply mining the visual features in the image. The input of user intent can help existing image content understanding algorithms more accurately represent the semantic information of interest to users. Current image content understanding algorithms can output unimodal semantic representation information under the guidance of the intent representation information provided by users, but there are a series of defects. For example, existing algorithms mainly generate unimodal (visual or text) representation results, lacking multimodal collaborative image content understanding output, and multimodal output can more intuitively and comprehensively represent the content of interest to users than unimodal output. In addition, the existing results do not fully understand the user's intent and the output results are single, which cannot provide users with diverse and selectable results. In practical applications, according to user prompts, the system can fully understand the user's intent and translate the image content into multimodal and diverse results, which has broad application prospects, such as medical diagnosis, autonomous driving, etc. Summary of the invention

[0003] The purpose of the present invention is to provide a user intention-driven controllable image segmentation and semantic capture method to solve the problems existing in the background technology.

[0004] To achieve the above object, the present invention provides a user intention driven controllable image segmentation and semantic capture method, comprising the following steps:

[0005] S1, obtaining input image and user visual prompt information;

[0006] S2, constructing a scene graph based on the input image obtained by S1 and mapping the user intention to the scene graph according to the user visual cue information obtained by S1, and extracting local scene graph features related to the intention;

[0007] S3, input the intent-related local scene graph features obtained in S2 and the annotated bimodal (visual mask and text) data into the diffusion model with the bimodal Transformer as the core to generate relevant bimodal data, which are visual mask and text pairs;

[0008] S4. Optimize the alignment of the visual mask and text pairs obtained in step S3 using multi-entity contrastive learning loss.

[0009] Preferably, S1 comprises:

[0010] S11, receiving an input image provided by a user, and normalizing its size to 1024×1024;

[0011] S12. Use the first noun of each text as the central object and use its corresponding bounding box as the user visual cue information.

[0012] Preferably, S2 comprises:

[0013] S21, constructing a global scene graph of the image for the input image;

[0014] S22, extracting a subgraph according to the user visual prompt, retaining nodes and edges connected to the central object of the user visual prompt;

[0015] S23, build an adapter to optimize the extracted sub-image, predict objects in the sub-image that are related to the visual mask and text pairs, and filter out irrelevant objects;

[0016] S24, adjusting the order of subgraph nodes to make it consistent with the order of objects described in the text;

[0017] S25. Combine the optimized node features and edge features to generate intent-related local scene graph features.

[0018] Preferably, S23 includes:

[0019] S231, using the self-attention block to update the subgraph node features;

[0020] S232, the mapping layer selects nodes related to the target according to the relevance scores of the nodes with respect to the labeled mask pairs;

[0021] S233. Use binary cross entropy (BCE) loss to train the adapter to optimize node screening.

[0022] Preferably, S24 includes:

[0023] S241, using a multi-layer fully connected network to map node features into a prediction ranking matrix;

[0024] S242, using cross entropy loss to minimize the gap between the predicted sorting matrix and the true sorting matrix in S241;

[0025] S243. Filter out relevant nodes and adjust the order of the nodes.

[0026] Preferably, S3 includes:

[0027] S31, adding Gaussian noise to the original text sample to obtain a noisy text, and obtaining a noisy text feature through a text encoder;

[0028] S32, inputting the noisy text features and the acquired local scene graph features into a scene graph-guided bimodal Transformer, and then generating text features for text prediction and visual features for mask prediction;

[0029] S33, using a text decoder to decode the text features for text prediction to generate a final text description;

[0030] S34. Use the Mask2Former network structure to generate category-independent binary region masks.

[0031] Preferably, generating text features for text prediction in S32 includes using noisy text features as query input, local scene graph features as key and value input at the same time, and obtaining updated text features through multi-head attention, cross self-attention layer and feedforward network. The generation process is as follows:

[0032] f' c =MultiHead s (f c ,f c ,f c );

[0033] u c =FFN(f' c +MultiHead c (f' c ,f g ,f g ));

[0034] h c =φ g (u c +f c );

[0035] Among them, MultiHead s and MultiHead c represent the multi-head attention layer and the cross self-attention layer respectively; FFN represents the feedforward layer; φ(·) represents the mlp layer; the generated h c Represents refined text features.

[0036] Preferably, generating visual features for mask prediction in S32 includes using local scene graph features as query input, noisy text features as key and value input at the same time, and generating visual features consistent with the text description for mask prediction through multi-head attention, cross self-attention layer and feedforward network.

[0037] Preferably, S4 includes:

[0038] S41, perform in-sample contrastive learning, where the in-sample contrastive loss is defined as the sum of mask-word and word-mask contrastive losses;

[0039] S42. Perform inter-sample contrastive learning, considering the relationship between masks and words across several samples. The inter-sample contrastive loss is defined as the sum of mask-text and text-mask contrastive losses.

[0040] Therefore, the present invention adopts the above-mentioned user intention-driven controllable image segmentation and semantic capture method, which has the following beneficial effects:

[0041] (1) A prompt-centric scene graph adapter is used to introduce the user's visual prompt information into the scene graph generation and optimization process. It can flexibly respond to different user needs, dynamically generate structured scene graphs related to user prompts, and achieve accurate mapping of user intentions and filtering of redundant information.

[0042] (2) A scene graph-guided diffusion model with a bimodal Transformer as the core is adopted to perform noise reduction generation in the diffusion process. Scene graph features and text features are integrated into the generation process at each step, achieving the collaborative generation of semantic description and segmentation mask, effectively overcoming the inconsistency problem caused by separate generation in traditional methods and improving the efficiency and quality of generation.

[0043] (3) The multi-entity contrastive learning loss is introduced to solve the problem of insufficient alignment between text and segmentation masks, further improving the resolution and alignment of the generated results;

[0044] (4) It allows the user's visual cue information to be mapped into multiple scene graphs, and generates multiple multimodal representation results accordingly, thereby providing multiple image understanding solutions to the user, facilitating the user's flexible selection.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is an input and output schematic diagram of an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a flow chart of an embodiment of the present invention;

[0049] Figure 4 It is a schematic diagram of the structure of the scene graph guided bimodal Transformer module adopted by the present invention;

[0050] Figure 5This is the generation effect of the embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 The present invention provides an embodiment method, based on the input image and the simple input provided by the user (such as the bounding box of the object), the user's prompt is mapped to the scene graph through a prompt-centric scene graph adapter, effectively capturing the user's intention, and then a scene graph is used to guide the diffusion process of the bimodal Transformer to simultaneously generate multiple multimodal semantic description pairs (including image segmentation visual masks and matching text descriptions). The technical structure diagram of the present invention is as follows Figure 2 shown.

[0053] like Figure 3 As shown, a user intention driven controllable image segmentation and semantic capture method includes at least the following operation process:

[0054] Step S1, obtaining input image and user visual prompt information. Specifically including:

[0055] S11. Receive an input image provided by a user and normalize its size to 1024×1024.

[0056] S12. Approximately use the first noun of each text as the central object and use its corresponding bounding box as the user visual cue information. The cue information is the key for the model to capture user intentions and is used to guide the subsequent scene graph generation and optimization process.

[0057] Step S2, constructing a scene graph based on the image obtained in S1 and mapping the user intention to the scene graph according to the user visual prompt information obtained in S1, and extracting the local scene graph features related to the intention. Including:

[0058] S21, constructing a global scene graph of the image for the image;

[0059] S22, extracting a subgraph according to the user visual prompt information, and retaining only nodes and edges connected to the central object of the user visual prompt;

[0060] S23, constructing a scene graph adapter to optimize the extracted sub-graph, predicting objects in the sub-graph that are related to the visual mask-text pair, and filtering out irrelevant objects;

[0061] Specifically, the scene graph adapter consists of several self-attention blocks and a mapping layer. First, the node features of the subgraph are fed into the self-attention block to generate updated features. Subsequently, the mapping layer predicts a node relevance score, which represents the relevance of the node with respect to the visual mask-text pair, and calculates its intersection over union (IOU) with the corresponding node in the subgraph. If the IOU exceeds the threshold, the relevance score is assigned to 1, otherwise it is assigned to 0. Therefore, the adapter can effectively predict relevant objects in the subgraph while filtering out irrelevant objects.

[0062] S24, adjusting the order of subgraph nodes to make it consistent with the order of objects described in the text;

[0063] The subgraph node order adjustment method specifically includes: using a multi-layer fully connected network to map node features into a sorting matrix; using cross entropy loss to minimize the gap between the predicted sorting matrix and the actual sorting matrix; screening out related nodes and adjusting the node order to obtain a refined node feature.

[0064] S25. Combine the optimized node features and edge features to generate the final scene graph features related to the user intention.

[0065] Step S3, such as Figure 3 As shown in the figure, the intent-related scene graph features obtained by S2 and the annotated bimodal (visual mask and text) data are input into the diffusion model with the bimodal Transformer as the core to generate relevant bimodal data, namely, visual mask and text pairs. Specifically, it includes:

[0066] S31, adding Gaussian noise to the original text sample, and obtaining the data features of the noisy text through a text decoder;

[0067] Specifically, Gaussian noise is added to the text samples from time step t=0 to t, and then a reverse denoising process is performed to restore the annotations. Different from the existing diffusion model, the example of the present invention introduces a scene graph feature to guide the denoising process.

[0068] S32, inputting the noise-added text features and the obtained scene graph features into a scene graph-guided bimodal Transformer, and then generating text features for text prediction and visual features for mask prediction;

[0069] Specifically, at each time step t, the noisy annotation features and the scene graph features obtained in step S2 are input into the scene graph guided bimodal Transformer, and then the visual features for mask prediction are generated. Figure 4As shown, the scene graph-guided bimodal Transformer of an embodiment of the present invention includes N blocks, each of which includes a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward layer.

[0070] For generating visual features for text prediction, the noisy text feature f is used c As query input, the scene graph feature f g At the same time, it is input into the block as key and value, and after multi-head attention, cross self-attention layer and feedforward network, the updated text features are obtained. The generation process is as follows:

[0071] f' c =MultiHead s (f c ,f c ,f c ),

[0072] u c =FFN(f' c +MultiHead c (f' c ,f g ,f g )),

[0073] h c =φ g (u c +f c )

[0074] Among them, MultiHead s and MultiHead c Represent the multi-head attention layer and the cross self-attention layer, FFN represents the feedforward layer, and φ(·) represents the mlp layer. The generated h c Represents a refined text feature, which is either passed to subsequent blocks or used for text prediction. The final noise-free text feature is passed through the text decoder to generate a natural language description that is semantically complete and highly relevant to the target region.

[0075] For generating visual features for predicting masks, the scene graph feature f g As query input, the noisy text description feature f c It is input into the block as key and value at the same time, and after multi-head attention, cross self-attention layer and feedforward network, it generates visual features consistent with the text description for mask prediction. The generation process is as follows:

[0076] f' g =MultiHead s (f g ,f g ,fg ),

[0077] u g =FFN(f' g +MultiHead c (f' g ,f c ,f c )),

[0078] h g =φ g (u g +f g )

[0079] The generated h g Guided by image annotation features, visual features for mask prediction can be accurately provided, which are either passed to subsequent blocks or used for mask prediction.

[0080] S33, using a text decoder to decode the text features used for text prediction to generate a final text description.

[0081] S34. Use the Mask2Former network structure to generate category-independent binary region masks.

[0082] Step S4, using multi-entity contrastive learning to optimize the alignment of the visual mask-text pairs obtained in step S3, including intra-sample contrastive learning and inter-sample contrastive learning.

[0083] S41. Perform in-sample contrastive learning, and the in-sample contrastive loss is defined as the sum of mask-word and word-mask contrastive losses.

[0084] Specifically, in-sample contrastive learning is to align masks and word embeddings at the level within a single sample to maximize the mutual information between masked entities and matching words. Specifically, it includes: calculating the mask-word matching relationship, obtaining the word embedding set that matches the current mask and the mask embedding set that matches the current word embedding, using the InfoNCE loss function to calculate the contrast loss between the mask embedding and its matching word embedding set, as well as the contrast loss between the word embedding and its matching mask embedding set, and the total in-sample contrast loss is defined as the sum of the mask-word and word-mask contrast losses.

[0085] S42. Perform inter-sample contrastive learning, considering the relationship between masks and words across several samples. The inter-sample contrastive loss is defined as the sum of mask-text and text-mask contrastive losses.

[0086] Perform inter-sample contrastive learning to capture global interaction information between different samples. Specifically, for each mask and word pair, calculate the global matching score. In each batch, randomly select mask-text pairs and optimize the contrastive loss of the global matching score. Similarly, the total inter-sample contrastive loss includes the contrastive loss between mask-text and text-mask.

[0087] Figure 5 A visualization example effect diagram of an embodiment of the present invention is shown. In general, the present invention supports the generation of multiple dual-modal (visual mask and text) results, can provide diverse but accurate results through a simple prompt, effectively reduce the user burden and improve overall performance.

[0088] Therefore, the present invention adopts the above-mentioned user intention-driven controllable image segmentation and semantic capture method to solve the problems that the existing image segmentation and semantic capture methods are difficult to capture user intention, the segmentation results and semantic text results are single and insufficiently correlated, and the multimodal output processing capabilities of complex scenes are limited, thereby greatly improving the image segmentation accuracy and the consistency of semantic capture.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A user-intention-driven controllable image segmentation and semantic capture method, characterized in that: The following steps are involved: S1, obtaining input image and user visual prompt information; S2, constructing a scene graph based on the input image obtained by S1 and mapping the user intention to the scene graph according to the user visual cue information obtained by S1, and extracting local scene graph features related to the intention; S3, input the intent-related local scene graph features obtained by S2 and the annotated bimodal data into the diffusion model with the bimodal Transformer as the core to generate relevant bimodal data, which are visual mask and text pairs; S4. Optimize the alignment of the visual mask and text pairs obtained in step S3 using multi-entity contrastive learning loss.

2. The user intention driven controllable image segmentation and semantic capture method according to claim 1, characterized in that: S1 includes: S11, receiving an input image provided by a user, and normalizing its size to 1024×1024; S12. Use the first noun of each text as the central object and use its corresponding bounding box as the user visual cue information.

3. The user intention driven controllable image segmentation and semantic capture method according to claim 2, characterized in that S2 include: S21, constructing a global scene graph of the image for the input image; S22, extracting a subgraph according to the user visual prompt, retaining nodes and edges connected to the central object of the user visual prompt; S23, build an adapter to optimize the extracted sub-image, predict objects in the sub-image that are related to the visual mask and text pairs, and filter out irrelevant objects; S24, adjusting the order of subgraph nodes to make it consistent with the order of objects described in the text; S25. Combine the optimized node features and edge features to generate intent-related local scene graph features.

4. The user intention driven controllable image segmentation and semantic capture method according to claim 3, characterized in that: S23 includes: S231, using the self-attention block to update the subgraph node features; S232, the mapping layer selects nodes related to the target according to the relevance scores of the nodes with respect to the labeled mask pairs; S233. Use binary cross entropy (BCE) loss to train the adapter to optimize node screening.

5. The user intention driven controllable image segmentation and semantic capture method according to claim 3, characterized in that: S24 includes: S241, using a multi-layer fully connected network to map node features into a prediction ranking matrix; S242, using cross entropy loss to minimize the gap between the predicted sorting matrix and the true sorting matrix in S241; S243. Filter out relevant nodes and adjust the order of the nodes.

6. The user intention driven controllable image segmentation and semantic capture method according to claim 5, characterized in that S3 include: S31, adding Gaussian noise to the original text sample to obtain a noisy text, and obtaining a noisy text feature through a text encoder; S32, inputting the noisy text features and the acquired local scene graph features into a scene graph-guided bimodal Transformer, and then generating text features for text prediction and visual features for mask prediction; S33, using a text decoder to decode the text features for text prediction to generate a final text description; S34. Use the Mask2Former network structure to generate category-independent binary region masks.

7. The user intention driven controllable image segmentation and semantic capture method according to claim 6, characterized in that: The text features generated for text prediction in S32 include using noisy text features as query input, local scene graph features as key and value input, and obtaining updated text features through multi-head attention, cross self-attention layer and feedforward network. The generation process is as follows: f′ c =MultiHead s (f c ,f c ,f c ); you c =FFN(f′ c +MultiHead c (f′ c ,f g ,f g )); h c =φ g (u c +f c ); Among them, MultiHead s and MultiHead c They represent the multi-head attention layer and the cross self-attention layer respectively; FFN represents the feed-forward layer; φ(·) represents the mlp layer; f c is the noisy text feature; f g is the local scene graph feature; the generated h c Represents refined text features.

8. The user intention driven controllable image segmentation and semantic capture method according to claim 7, characterized in that: The generation of visual features for mask prediction in S32 includes using local scene graph features as query input, noisy text features as key and value input, and generating visual features consistent with the text description through multi-head attention, cross self-attention layer and feedforward network for mask prediction. The generation process is as follows: f′ g =MultiHead s (f g ,f g ,f g ); you g =FFN(f′ g +MultiHead c (f′ g ,f c ,f c )); h g =φ g (u g +f g ); The generated h g Represents visual features.

9. The user intention driven controllable image segmentation and semantic capture method according to claim 5, characterized in that S4 include: S41, perform in-sample contrastive learning, where the in-sample contrastive loss is defined as the sum of mask-word and word-mask contrastive losses; S42. Perform inter-sample contrastive learning, considering the relationship between masks and words across several samples. The inter-sample contrastive loss is defined as the sum of mask-text and text-mask contrastive losses.

Images