A zero-shot referential and instance segmentation method and system based on dynamic color assignment
By employing a dynamic color allocation algorithm and cross-attention guided denoising technology, the problem of accurate segmentation of computer vision instance segmentation and index expression segmentation in zero-shot scenarios is solved, achieving high-precision and low-cost pixel-level segmentation results.
Patent Information
- Application Number
- CN202611297933.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-07-01
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-06
AI Technical Summary
Existing computer vision instance segmentation and indexed expression segmentation techniques have insufficient generalization performance in zero-shot scenarios, rely on a large amount of labeled data, are difficult to achieve high-precision pixel-level segmentation, and lack standardized algorithmic documentation, resulting in poor reproducibility.
A dynamic color allocation algorithm is adopted, which generates a solid color rendering image and performs clustering through multimodal data preprocessing, cross-attention guided denoising and color analysis, and adaptively generates a binary mask to achieve pixel-level accurate segmentation.
It achieves accurate pixel-level segmentation of arbitrary natural language text references without the need for training or fine-tuning, avoiding dependence on large-scale labeled data and improving segmentation accuracy and robustness.
Smart Images
Figure CN123052091A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a method and system for zero-sample referencing and instance segmentation based on dynamic color allocation. Background Technology
[0002] In computer vision tasks involving instance segmentation and referencing expression segmentation (RES), current mainstream technologies mainly fall into two categories:
[0003] The first category consists of supervised segmentation models based on dedicated network architectures, with UniGS being a typical example. This approach proposes a unified diffusion model representation method, modifying the traditional UNet architecture and adding a mask generation branch. It employs a position-aware color palette, assigning colors to image entities through hard-coded rules and generating color maps, which are then decoded to obtain the segmentation mask. The training and fine-tuning of this approach heavily rely on a large dataset with mask annotations.
[0004] The second category is image-level classification models driven by prompts, with InstructCV being a typical example. This approach transforms the text-image diffusion model into a general visual model through fine-tuning of instructions. Its core employs a color-coding strategy, relying on natural language prompts such as "output the specified color if the image contains the corresponding category" to guide the diffusion model to generate global solid color blocks, thereby completing image-level classification.
[0005] The aforementioned existing technical solutions have the following technical defects:
[0006] First, zero-shot generalization performance is insufficient, and the model is highly dependent on data annotation. Mainstream instance segmentation models such as UniGS, without targeted fine-tuning and supplementary labeled data, are unable to achieve accurate zero-shot segmentation of any free-form text referent. When new categories or complex referential descriptions not present in the input training set are encountered, the model is prone to inference failure.
[0007] Secondly, the lack of pixel spatial matching relationships makes it difficult to directly generate high-precision pixel-level masks. Although large models such as InstructCV can output corresponding colors based on text instructions, the output results only represent the global category of the entire image and cannot establish spatial correspondence with the local regions of each instance in the image. Conventional image generation models output continuous RGB images with light and shadow and texture, which can easily confuse the visual features of different instances of the same type. They cannot extract pixel-level instance masks with discrete boundaries and accurate positioning at low cost.
[0008] Third, the core processes lack standardized algorithmic documentation, resulting in poor reproducibility of the solutions. Existing literature only provides conceptual explanations of key steps such as dynamic color allocation rules, local region pure color generation guidance logic, and cluster-based mask extraction, without providing clear mathematical expressions and standardized algorithmic steps, which reduces the feasibility of the solutions and their ability to support technical verification. Summary of the Invention
[0009] In view of the above-mentioned shortcomings of current computer vision instance segmentation and index representation segmentation, this invention provides a zero-sample index representation and instance segmentation method and system based on dynamic color allocation, which can achieve pixel-level accurate semantic segmentation without training.
[0010] To achieve the above objectives, the present invention adopts the following technical solution:
[0011] A zero-sample referencing and instance segmentation method based on dynamic color assignment, characterized in that the method includes:
[0012] Acquire multimodal data and preprocess it, the multimodal data including image data and natural language task instructions;
[0013] Based on the image data and natural language task instructions, execute a dynamic color allocation algorithm to generate rendering prompt instructions;
[0014] The rendering prompts and the image data are input into the image generation model, cross-attention guided denoising is performed, and a solid color rendering image is output.
[0015] The solid color rendering image is subjected to color analysis and mask extraction to generate a binary mask.
[0016] According to one aspect of the invention, the multimodal data further includes a number N of targets to be segmented, and the task type is identified by the number N of targets to be segmented, the task type including indexed segmentation and instance segmentation, when When the task type is determined to be a representative segmentation, when When the task type is determined to be instance segmentation.
[0017] According to one aspect of the present invention, the execution of the dynamic color allocation algorithm to generate rendering prompt instructions specifically includes:
[0018] Establish a benchmark for perceptual uniform color measurement;
[0019] Based on the task type and the number N of targets to be segmented, a dynamic color allocation algorithm is executed in the perceptual uniform color metric space to generate a color set.
[0020] Based on the color set, the color mapping relationship is encoded into rendering prompt instructions using a preset structured template.
[0021] According to one aspect of the present invention, the dynamic color allocation algorithm includes:
[0022] When the task type is referential segmentation, the background color is determined according to the natural language task instruction, and a color with a perceptual color difference greater than a preset threshold from a preset set of high-recognition pure colors is selected as the referential target color.
[0023] When the task type is instance segmentation, a maximum-minimum distance greedy strategy is adopted to iteratively select colors from the candidate pool, so as to maximize the minimum perceptual color difference between the newly selected color and the set of already selected colors, and force the minimum perceptual color difference to be greater than or equal to a preset threshold.
[0024] According to one aspect of the present invention, the preset structured natural language template includes an index expression segmentation template and an instance segmentation template.
[0025] According to one aspect of the present invention, the step of inputting the rendering prompt instruction and the image data into the image generation model for cross-attention guided denoising specifically comprises:
[0026] The rendering prompt instructions and the image data are conditionally encoded to generate a text embedding sequence and a visual feature map;
[0027] Based on the text embedding sequence and visual feature map, color assignment is mapped to spatial regions and guided through the cross-attention layer of U-Net;
[0028] Based on the guidance results, a classifier is used to freely guide the execution of conditional diffusion denoising.
[0029] According to one aspect of the present invention, the step of performing color analysis and mask extraction on the solid color rendering image to finally generate a binary mask specifically involves:
[0030] The solid color rendering image is converted to a color space to obtain a set of pixel features;
[0031] The corresponding clustering strategy is applied to the pixel feature set to generate clusters;
[0032] Calculate the average color difference within each cluster, and perform noise reduction processing on clusters whose average color difference exceeds a preset noise threshold.
[0033] The denoised clusters are mapped to binary masks.
[0034] According to one aspect of the invention, the clustering strategy is determined based on the task type and the number N of targets to be segmented: when the task type is a target that can be segmented and When the task type is instance segmentation and N is a known fixed value, K-Means clustering is used, and the number of clusters is fixed at 2. When the task type is instance segmentation and N is unknown or extremely large, DBSCAN density clustering is used.
[0035] According to one aspect of the invention, the image generation model includes a text encoder, an image encoder, a denoising U-Net, and an image decoder.
[0036] A zero-sample referencing and instance segmentation system based on dynamic color assignment, the system being applied to any of the zero-sample referencing and instance segmentation methods described above, the system comprising:
[0037] The data acquisition module acquires multimodal data and preprocesses it, the multimodal data including image data and natural language task instructions;
[0038] The instruction generation module executes a dynamic color allocation algorithm to generate rendering prompt instructions based on the image data and natural language task instructions.
[0039] The image generation module inputs the rendering prompt instructions and the image data into the image generation model, performs cross-attention guided denoising, and outputs a solid color rendering image;
[0040] The result generation module performs color analysis and mask extraction on the solid color rendering image, and finally generates a binary mask.
[0041] The advantage of this invention is that it can achieve precise pixel-level segmentation of arbitrary natural language text references without any model training or fine-tuning. Specifically:
[0042] First, through the dynamic color allocation algorithm, it can adaptively generate a set of highly recognizable pure colors that meet the minimum color difference constraint according to the task type (referring to representative segmentation or instance segmentation) and the number of targets N, breaking through the dependence of traditional methods on fixed categories and preset color palettes.
[0043] Second, by reducing the complex natural language referential relationships to color rendering tasks and using the cross-attention mechanism in the image-to-image conditional diffusion model for spatial guidance, it is possible to generate a pure color rendering image that maintains a strict pixel-level spatial correspondence with the original image instance without modifying any network parameters. This effectively avoids the dependence on large-scale labeled data and the instance confusion caused by texture and lighting in natural images.
[0044] Third, by performing color clustering (K-Means or DBSCAN) and introducing a color distance threshold for noise filtering, discrete and accurate binary masks can be stably extracted from solid color rendering images, effectively suppressing color gradients and artifact interference that may occur in edge regions of the generative model, and finally achieving high-precision and robust zero-sample referencing and instance segmentation. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a schematic flowchart of a zero-sample referencing and instance segmentation method based on dynamic color allocation according to the present invention.
[0047] Figure 2 This is a schematic diagram of the structure of a zero-sample referencing and instance segmentation system based on dynamic color allocation according to the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Example 1
[0050] like Figure 1 As shown, a zero-sample referencing and instance segmentation method based on dynamic color allocation is presented, the method specifically including the following steps:
[0051] Step S1: Acquire multimodal data and preprocess it. The multimodal data includes image data, natural language task instructions, number of targets to be segmented, and task type.
[0052] The specific steps for obtaining multimodal data in step S1 are as follows:
[0053] The image data to be processed is obtained through a local file storage system, a user-interactive upload interface, or a specified network URL. The obtained image data is decoded and its size is normalized, outputting a normalized tensor, denoted as: Where H and W represent the height and width of the image (in pixels), respectively, and channel dimension 3 corresponds to the RGB color space.
[0054] The system receives natural language task instruction strings typed by the user through a text input box in the graphical user interface (GUI) or a command-line interface, denoted as text variable T. Simultaneously, it obtains the number N of targets to be segmented, as input by the user. This number can be specified directly by the user through a numeric input box, or automatically inferred from T by the system's built-in lightweight semantic parser.
[0055] Task types include Referring Expression Segmentation (RES) and Instance Segmentation, identified by the number N of targets to be segmented. When... When, it indicates that the task type is a segmentation of representation. When the task type is instance-based, it indicates that the task type is instance-based.
[0056] In this embodiment, for the specified representation segmentation scenario, the input is an image data containing multiple people and a complex background. (resolution) The user's text variable T is: "A girl wearing a red baseball cap and drinking water." The number of targets to be segmented, N, is set to 1.
[0057] In this embodiment, for the instance segmentation scenario, the input is an image containing approximately 200 extremely similar targets (such as scattered garlic cloves), where the number of targets N is unknown.
[0058] Step S2: Based on the image data and natural language task instructions, execute the dynamic color allocation algorithm to generate rendering prompt instructions;
[0059] Step S21: Establish a color measurement benchmark;
[0060] To eliminate the non-linear discrepancy between Euclidean distance in the RGB color space and human visual perception, a unified color measurement standard is established. Candidate pure colors are converted from the RGB color space to the CIELAB color space, and the conversion function is denoted as... Using a standard D65 illuminator and a 2° observer angle, any two colors within this metric space... Its perceived color difference is defined as:
[0061]
[0062] in, , representing lightness, red-green axis chromaticity, and blue-yellow axis chromaticity, respectively.
[0063] Step S22: Execute the dynamic color allocation algorithm according to the task type;
[0064] (1) When the task type is instance partitioning ( A greedy strategy of minimax distance is used to generate N color sets in the metric space. This maximizes the perceived color difference between any two colors and is not less than 1. Specifically:
[0065] 1. From a standard high-contrast seed color set Select from (such as red, green, blue, yellow, cyan, magenta, etc.) .
[0066] 2. For , In the candidate pool (by RGB cubes with sampling step size) Select the k-th color from the uniform sampling (formulation):
[0067]
[0068] Simultaneously enforce constraints: ,in, The minimum distinguishable color difference threshold, The preferred value is 30.
[0069] Output color set Each color is an RGB integer value.
[0070] In this embodiment, estimation Generate a set of 200 colors by executing a maximum-minimum distance greedy algorithm. ,satisfy .
[0071] (2) When the task type is referential segmentation, let the target color be... The background color is , Preferred ; From a preset set of highly recognizable solid colors (preferably pure green) Or pure red The selection criteria are: the element with the largest color difference from the background and unrelated to any potential color preferences implied in T. This satisfies the following:
[0072]
[0073] The output color set is .
[0074] In this embodiment, for the finger delivery segmentation scenario, the following is selected: (Pure green) (Black), Verification .
[0075] Step S23: Based on the color assignment results, generate natural language prompts using a preset structured template.
[0076] Based on the task type and color set, color mapping rules are encoded into natural language prompts P using a structured template. The preset structured template P is as follows:
[0077] (1) Refers to the segmentation template :
[0078] ="Render the target that satisfies the reference condition 'T' as a solid color RGB ( The remaining areas are rendered as RGB ( )".
[0079] In the embodiments, =Render the girl wearing a red baseball cap and drinking water as pure green. The remaining areas are rendered in black. ".
[0080] (2) Instance segmentation template :
[0081] ="Render the i-th target instance in the image as a solid RGB color ( (i=1,…,N), ensuring that the color difference between adjacent instances in the LAB color space is greater than 1. ".
[0082] Step S3: Input the rendering prompt instruction and the image data into the image generation model, perform cross-attention guided denoising, and output a solid color rendering image;
[0083] Step S31: Perform conditional encoding on the rendering prompt instruction and the image data to generate a text embedding sequence and a visual feature map;
[0084] The rendering cue instruction P is encoded into a text embedding sequence using a pre-trained text encoder (such as CLIP ViT-H / 14). Where L is the token length and d is the embedding dimension.
[0085] Image data is processed using a pre-trained image encoder (such as a VAE). Encoding as visual feature maps , where h and w are the spatial dimensions after downsampling, and c is the number of latent space channels.
[0086] Step S32: Based on the text embedding sequence and visual feature map, color assignment is mapped to spatial regions and guided through cross-attention;
[0087] In the decoding layer at each time step, the denoising U-Net uses a cross-attention mechanism to dynamically align the color allocation rules (such as "target A is rendered as pure red, target B as pure green") contained in the rendering cue instruction P with the image spatial features, thereby accurately projecting each pure color identifier to the region where its corresponding target instance is located. Let the intermediate latent space features at step t be... The single-head cross attention calculation is as follows:
[0088] , ,
[0089]
[0090] In the formula, This is an intermediate feature at step t. , , This is a learnable linear projection matrix in U-Net. This refers to the dimension of attention.
[0091] For pointer-to-segmentation, its pointer-to-segmentation template Includes "Render the target that satisfies the reference condition 'T' as a solid color RGB ( The remaining areas are rendered as RGB ( The prompt ")" indicates that the attention weights are focused on the image region where the target is located, guiding U-Net to preferentially generate images that are close to the target in that region. The solid color feature generates a color close to the background color in non-target areas. The characteristics of pure color;
[0092] For instance segmentation scenarios, multi-head attention processes N color assignment constraints in parallel ( This allows different instance regions to be guided independently and without interference to their respective corresponding pure colors, thus achieving a spatial correspondence between instances and colors.
[0093] Step S33: Based on the guidance results, use a classifier to freely guide the execution of conditional diffusion denoising;
[0094] A pre-trained image-to-image diffusion model (such as Stable Diffusion v2.1) is used to perform conditional denoising within the latent space. Initial latent variables are obtained by sampling from standard Gaussian noise. Then proceed according to the reverse time step Perform denoising updates. Deterministic sampling using DDIM (Denoising Diffusion Implicit Models) is employed, with the update formula as follows:
[0095]
[0096] In the formula, For the U-Net denoising network, For noise dispatch coefficient, for The cumulative product, Standard Gaussian noise, Let be the noisy latent variable at step t. The random noise coefficient introduced in DDIM sampling.
[0097] To enhance the model's adherence to rendering cue instructions P, a classifier-free guidance (CFG) approach is employed. Within the same time step t, unconditional noise predictions are computed in parallel. With conditional noise prediction The final combined noise estimate is:
[0098]
[0099] In the formula, For guiding strength (preferred) ), Empty message.
[0100] After the output undergoes all T-step denoising, the final latent variables are obtained. .
[0101] Step S34: Spatial correspondence preservation and VAE decoding are performed on the denoised and updated result to output a solid color rendering image.
[0102] In each denoising step, the image data I is compared with the current latent band noise variable. The concatenation along the channel dimension is then input into U-Net, forcing the generated result to match the visual feature map in spatial structure. Strict alignment. The final generated latent variables. The corresponding pixel space rendering image, its solid color areas Maintaining a pixel-level correspondence with the spatial location of the target instance in the original image data I, satisfying:
[0103]
[0104] In the formula, This represents the effective pixel support domain of the rendered image.
[0105] Will The input is processed by a VAE decoder, reconstructed into a pixel-space RGB image, resulting in a local solid color rendering image. .
[0106] Step S4: Perform color analysis and mask extraction on the solid color rendering image to finally generate a binary mask.
[0107] Step S41: Perform color space conversion on the solid color rendering image to obtain a set of pixel features;
[0108] right Each pixel Convert its RGB values to LAB color space:
[0109]
[0110] This transformation maps all pixels to the same perceptually uniform color metric space as in step S21, ensuring that subsequent clustering and color difference calculations have a consistent metric benchmark.
[0111] Step S42: Based on the task type and the number of targets N, execute the corresponding clustering strategy on the pixel feature set to generate several clusters;
[0112] (1) refers to the division of representation ( );
[0113] For pixel set Perform color K-Means clustering, with a fixed number of clusters. These correspond to the foreground and background, respectively. The optimization objective is to minimize all pixels to their nearest cluster. The sum of the squared distances between the centers is:
[0114]
[0115] Iterative update of cluster centers Each pixel is assigned to the nearest cluster center, forming two subsets of pixels (clusters). and Next, the mean of each subset is recalculated as the new cluster center. :
[0116]
[0117] Output two clusters , and its cluster centers , .
[0118] because The target and the background are respectively a highly recognizable solid color and a pure black. They are naturally highly separable in the LAB space, and the clustering results are stable and converge rapidly.
[0119] (2) Instance segmentation, and the number of targets N is known. );
[0120] K-Means color clustering is used, with a fixed number of clusters. This is strictly consistent with the number N of the color set allocated in step S22. For the pixel set... Perform the above K-Means iterative optimization to determine the cluster centers. It will automatically converge to The LAB values corresponding to each solid color in the output are near the output clusters. and its cluster centers .
[0121] (3) Instance segmentation, and the number of targets N is unknown or the number of targets is extremely large.
[0122] When the number of target instances N is unknown or extremely large (e.g., hundreds of similar instances), DBSCAN is used to adaptively discover color clusters without needing to pre-specify the number of clusters. The specific parameter settings are as follows:
[0123] Neighborhood radius: (Preferred) (in the LAB space)
[0124] Minimum number of samples: (Preferred) )
[0125] pixel p - Neighborhood is defined as:
[0126]
[0127] In the formula, Represents the LAB color vector of another pixel, if If p is the core point, then the core point and its neighborhood are merged into clusters based on density reachability. This process is repeated for all pixels to generate several clusters. and the geometric center (mean) of each cluster. .in, It is determined automatically by the algorithm.
[0128] DBSCAN can automatically discover color clusters of any shape and number, effectively dealing with color gradients and edge blur noise that may occur in the rendering of dense scenes.
[0129] Step S43: Calculate the average color difference within each cluster, and denoise the clusters whose average color difference exceeds a preset noise threshold.
[0130] For each cluster Calculate the average color difference from the pixel within the cluster to the cluster center:
[0131]
[0132] like (Preferred) If the cluster is determined to contain too much light and shadow gradation noise, a second subdivision or removal is performed.
[0133] Step S44: Map the denoised clusters to a binary mask.
[0134] Map each valid cluster to a binary mask:
[0135]
[0136] For the segmentation of the pointer, select the target color. The cluster with the closest distance in the LAB space is used as the final mask:
[0137]
[0138] Output: Set of discrete pixel-level segmentation masks (When N=1 in RES, only output) When splitting instances, input N masks. If DBSCAN finds a certain number of clusters... If the matched valid cluster is used, the missing instance mask is set to all zeros.
[0139] Example 2
[0140] like Figure 2 As shown, a zero-sample referencing and instance segmentation system based on dynamic color assignment is disclosed. This system is applied to the zero-sample referencing and instance segmentation method described in Embodiment 1. The system includes:
[0141] The data acquisition module M1 acquires multimodal data and preprocesses it, the multimodal data including image data and natural language task instructions;
[0142] The instruction generation module M2 executes a dynamic color allocation algorithm to generate rendering prompt instructions based on the image data and natural language task instructions;
[0143] Image generation module M3 inputs the rendering prompt instructions and the image data into the image generation model, performs cross-attention guided denoising, and outputs a solid color rendering image;
[0144] The result generation module M4 performs color analysis and mask extraction on the solid color rendering image, and finally generates a binary mask.
[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A zero-sample referencing and instance segmentation method based on dynamic color allocation, characterized in that, The method includes: Acquire multimodal data and preprocess it, the multimodal data including image data and natural language task instructions; Based on the image data and natural language task instructions, execute a dynamic color allocation algorithm to generate rendering prompt instructions; The rendering prompts and the image data are input into the image generation model, cross-attention guided denoising is performed, and a solid color rendering image is output. The solid color rendering image is subjected to color analysis and mask extraction to generate a binary mask.
2. The zero-sample referencing and instance segmentation method according to claim 1, characterized in that, The multimodal data also includes the number of targets N to be segmented. The task type is identified by the number of targets N, and the task type includes target segmentation and instance segmentation. When the task type is determined to be a representative segmentation, when When the task type is determined to be instance segmentation.
3. The zero-sample referencing and instance segmentation method according to claim 2, characterized in that, The specific steps for executing the dynamic color allocation algorithm and generating rendering prompt instructions are as follows: Establish a benchmark for perceptual uniform color measurement; Based on the task type and the number N of targets to be segmented, a dynamic color allocation algorithm is executed in the perceptual uniform color metric space to generate a color set. Based on the color set, the color mapping relationship is encoded into rendering prompt instructions using a preset structured template.
4. The zero-sample referencing and instance segmentation method according to claim 3, characterized in that, The dynamic color allocation algorithm includes: When the task type is referential segmentation, the background color is determined according to the natural language task instruction, and a color with a perceptual color difference greater than a preset threshold from a preset set of high-recognition pure colors is selected as the referential target color. When the task type is instance segmentation, a maximum-minimum distance greedy strategy is adopted to iteratively select colors from the candidate pool, so as to maximize the minimum perceptual color difference between the newly selected color and the set of already selected colors, and force the minimum perceptual color difference to be greater than or equal to a preset threshold.
5. The zero-sample referencing and instance segmentation method according to claim 4, characterized in that, The preset structured natural language templates include an index expression segmentation template and an instance segmentation template.
6. The zero-sample referencing and instance segmentation method according to claim 1, characterized in that, The specific steps of inputting the rendering prompt instructions and the image data into the image generation model for cross-attention guided denoising are as follows: The rendering prompt instructions and the image data are conditionally encoded to generate a text embedding sequence and a visual feature map; Based on the text embedding sequence and visual feature map, color assignment is mapped to spatial regions and guided through the cross-attention layer of U-Net; Based on the guidance results, a classifier is used to freely guide the execution of conditional diffusion denoising.
7. The zero-sample referencing and instance segmentation method according to claim 1, characterized in that, The specific steps for performing color analysis and mask extraction on the solid color rendering image to ultimately generate a binary mask are as follows: The solid color rendering image is converted to a color space to obtain a set of pixel features; The corresponding clustering strategy is applied to the pixel feature set to generate clusters; Calculate the average color difference within each cluster, and perform noise reduction processing on clusters whose average color difference exceeds a preset noise threshold. The denoised clusters are mapped to binary masks.
8. The zero-sample referencing and instance segmentation method according to claim 2, characterized in that, The clustering strategy is determined based on the task type and the number N of targets to be segmented: when the task type is a target that achieves segmentation and When the task type is instance segmentation and N is a known fixed value, K-Means clustering is used, and the number of clusters is fixed at 2. When the task type is instance segmentation and N is unknown or extremely large, DBSCAN density clustering is used.
9. The zero-sample referencing and instance segmentation method according to any one of claims 1 to 8, characterized in that, The image generation model includes a text encoder, an image encoder, a denoising U-Net, and an image decoder.
10. A zero-sample referencing and instance segmentation system based on dynamic color allocation, characterized in that, The system is applied to the zero-sample referencing and instance segmentation method according to any one of claims 1 to 9, and the system comprises: The data acquisition module acquires multimodal data and preprocesses it, the multimodal data including image data and natural language task instructions; The instruction generation module executes a dynamic color allocation algorithm to generate rendering prompt instructions based on the image data and natural language task instructions. The image generation module inputs the rendering prompt instructions and the image data into the image generation model, performs cross-attention guided denoising, and outputs a solid color rendering image; The result generation module performs color analysis and mask extraction on the solid color rendering image, and finally generates a binary mask.