Multi-granularity, multi-level instance segmentation method and apparatus, electronic equipment, storage medium

Abstract

This application provides a multi-granularity, multi-level instance segmentation method, apparatus, electronic device, and storage medium, belonging to the field of image processing technology. The method includes: fusing features from multimodal information of a target image to obtain a first fused feature map at multiple scales; scoring the first fused feature map with the highest resolution to obtain a response score for each pixel; using the pixels with the highest response scores as seed points and aggregating the neighborhood features of the seed points to obtain an instance feature map; performing mask prediction based on each instance feature map to obtain an instance mask and its hierarchy; merging multiple instance masks level by level to obtain a grouped mask and an overall mask; and performing instance segmentation on the target image based on the instance masks at multiple granularities and the instance masks at multiple levels. The multi-granularity, multi-level instance segmentation method, apparatus, electronic device, and storage medium provided in this application can achieve multi-granularity, multi-level instance segmentation of images.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and more specifically, relates to a multi-granularity, multi-level instance segmentation method and apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, significant progress has been made in image instance segmentation and open vocabulary segmentation: one type is the end-to-end unified architecture, which integrates semantics, instance, and panoramic segmentation into a unified framework, emphasizing minimal post-processing and overall optimization; another type is cue-based segmentation, which relies on cue points / boxes / text to quickly generate class-independent masks in zero-shot scenarios, improving transfer and interaction efficiency; another type is open vocabulary / text-controlled segmentation based on vision-language pre-training, which supports expanding categories by text and predicting unseen classes; in addition, RGB + depth segmentation that integrates depth information enhances boundary and robustness under weak texture and complex lighting conditions, and specialized methods and data for crowded / occluded environments also focus on occlusion robustness and target differentiation.

[0003] Overall, the aforementioned technologies have advantages in unified modeling, interactive prompts, open categories, and geometric enhancement, but they mostly remain at a single granularity or a single level, and cannot provide individual / union / grouping or object / part / subpart. Summary of the Invention

[0004] The purpose of this application is to provide a multi-granularity, multi-level instance segmentation method, apparatus, electronic device, and storage medium to achieve multi-granularity, multi-level instance segmentation of images.

[0005] A first aspect of this application provides a multi-granularity, multi-level instance segmentation method, including:

[0006] The multimodal information of the target image is acquired, and multi-scale feature fusion is performed on the multimodal information to obtain a first fused feature map at multiple scales; the multimodal information includes text control information, RGB image and its corresponding depth image;

[0007] Based on the pre-trained scoring model, each pixel in the first fused feature map with the highest resolution is scored to obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance.

[0008] According to the corresponding response scores from high to low, the first K pixels are selected as seed points. With each seed point as the center, the neighborhood features of the seed point are gathered from the first fused feature map at multiple scales to obtain the instance feature map of the target instance where the seed point is located.

[0009] Mask prediction is performed based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; the level of the instance mask includes object level, component level and sub-component level;

[0010] Multiple object-level instance masks are merged level by level to obtain group masks and overall masks respectively;

[0011] The target image is segmented based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks, and overall masks.

[0012] A second aspect of this application provides a multi-granularity, multi-level instance segmentation apparatus, comprising:

[0013] The data acquisition module is used to acquire multimodal information of the target image, perform multi-scale feature fusion on the multimodal information, and obtain a first fused feature map at multiple scales; the multimodal information includes text control information, RGB image and its corresponding depth image;

[0014] The location scoring module is used to score each pixel in the first fused feature map with the highest resolution based on a pre-trained scoring model, and obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance;

[0015] The seed selection module is used to select the top K pixels as seed points according to the corresponding response scores from high to low. Centered on each seed point, the neighborhood features of the seed point are gathered from the first fused feature map at multiple scales to obtain the instance feature map of the target instance where the seed point is located.

[0016] The mask prediction module is used to perform mask prediction based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; the level of the instance mask includes object level, component level and sub-component level;

[0017] The mask merging module is used to merge multiple object-level instance masks step by step to obtain group masks and overall masks respectively;

[0018] The instance segmentation module is used to perform instance segmentation on the target image based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks, and overall masks.

[0019] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the multi-granularity, multi-level instance segmentation method described above.

[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-granularity, multi-level instance segmentation method described above.

[0021] The beneficial effects of the multi-granularity, multi-level instance segmentation method and apparatus, electronic device, and storage medium provided in this application are as follows:

[0022] This embodiment first performs multi-scale feature fusion on the multimodal information of the target image to obtain a first fused feature map at multiple scales. Then, in the first fused feature map with the highest resolution, seed points are selected based on the response score of each pixel. Centered on each seed point, the neighborhood features of the seed point on the first fused feature maps at multiple scales are aggregated to obtain the instance feature map of the target instance where the seed point is located. Mask prediction is performed based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask. The object-level instance mask is treated as an individual, and multiple object-level instance masks are merged level by level to obtain group masks and overall masks, thus forming mask outputs of multiple granularities such as individual / group / overall. Instance segmentation of the target image is performed based on the instance masks of multiple granularities and the instance masks of multiple levels, which can realize multi-granularity and multi-level instance segmentation of the target image. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a multi-granularity, multi-level instance segmentation method provided in an embodiment of this application;

[0025] Figure 2 An RGB image of a target image provided in one embodiment of this application;

[0026] Figure 3 A depth image of a target image provided in one embodiment of this application;

[0027] Figure 4This is a schematic diagram of the structure of a mapping network provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram of a deep reliability weighting graph provided in an embodiment of this application;

[0029] Figure 6 A flowchart illustrating a multi-granularity, multi-level instance segmentation method provided in another embodiment of this application;

[0030] Figure 7 A structural block diagram of a multi-granularity, multi-level instance segmentation device provided in an embodiment of this application;

[0031] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0036] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a multi-granularity, multi-level instance segmentation method provided in an embodiment of this application. The multi-granularity, multi-level instance segmentation method provided in this embodiment can be executed by an electronic device, and the method may include:

[0037] S101: Obtain multimodal information of the target image, perform multi-scale feature fusion on the multimodal information, and obtain the first fused feature map at multiple scales; the multimodal information includes text control information, RGB image and its corresponding depth image.

[0038] In this embodiment, the text control information is used to specify the target of the image segmentation task. Taking a rice plant image as an example, the corresponding text control information could be: segment the leaves of the rice plant; the RGB image is used to indicate the color visual information of the target image, including leaf color and texture information, etc. Figure 2 As shown; depth images are used to indicate the spatial distance information of each target instance in the target image, such as the depth difference between rice plant leaves and the background, and the positional relationship between rice plants, etc. Figure 3 As shown.

[0039] By fusing the text control information, RGB image, and corresponding depth image of the target image at multiple scales, a first fused feature map at multiple scales can be obtained. For example, the concept of an existing feature pyramid model can be adopted to align and fuse multimodal information in a multi-scale space to obtain a feature pyramid of the target image.

[0040] S102: Based on the pre-trained scoring model, each pixel in the first fused feature map with the highest resolution is scored to obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance.

[0041] In this embodiment, each pixel can be scored on the first fusion feature map with the highest resolution (i.e., the first fusion feature map with the largest scale, such as stride=4) based on a pre-trained scoring model, and the response score of each pixel as the center of the target instance can be output.

[0042] The scoring model can be implemented using a single-channel scoring head with a single-layer 3×3 lightweight convolution, batch normalization (BN), and SiLU activation function. The parameters for the 3×3 lightweight convolution are set as follows: stride=1, padding=1. For larger targets, the convolution can be set to dilation=2 to expand the receptive field. During training, the instance center region is used as a positive sample (obtained by small-radius erosion or Gaussian diffusion from the ground truth mask). Regions with uncertain boundaries and unreliable depth are marked as "ignore" and not considered for loss. Focal loss or binary cross-entropy (BCE) + online hard example mining (OHEM) is used to balance the positive and negative samples.

[0043] In this embodiment, the high-resolution layer has finer spatial semantics. Therefore, pixel scoring on the first fused feature map with the highest resolution can accurately locate the instance center and extract local differences.

[0044] S103: Select the top K pixels as seed points according to the corresponding response scores from high to low. Centered on each seed point, gather the neighborhood features of the seed point from the first fusion feature map of multiple scales to obtain the instance feature map of the target instance where the seed point is located.

[0045] In this embodiment, the top-K (e.g., 100–300) pixels with the highest response scores can be selected as seed points. Centered on each seed point, the neighborhood features are aggregated from the first fusion feature map at multiple scales to obtain the instance feature map of the target instance where the seed point is located.

[0046] The range of neighborhood features can be dynamically adjusted based on the scale of the first fused feature map. For example, for high-resolution layers (i.e., larger-scale layers), the instances are smaller, and the neighborhood window radius is smaller (e.g., 7×7) to capture local structures in detail; for low-resolution layers (i.e., smaller-scale layers), the instances are larger, and the neighborhood window is expanded accordingly (e.g., 15×15) to cover the overall contour.

[0047] Specifically, the range of neighborhood features can be determined using the following formula:

[0048] ;in is the neighborhood radius at the current scale, and h is a constant, usually taken as 1.5–2.5.

[0049] In this embodiment, the adaptive design of neighborhood features can ensure contextual consistency at different scales: small targets focus on local details, while large targets cover the global structure.

[0050] For example, with each seed point as the center, neighborhood features are aggregated from the first fusion feature maps at multiple scales to obtain the instance feature map of the target instance where the seed point is located. Specifically, this includes: selecting neighborhood features of the seed point from the first fusion feature maps at multiple scales, dividing the neighborhood features at each scale into a preset number of sub-grids (e.g., 7×7), performing max pooling or average pooling on the feature values ​​within each sub-grid to obtain fixed-dimensional feature blocks, flattening the feature blocks to form fixed-length vectors, and weighting and summing the fixed-length vectors at each scale through an attention module to obtain the instance feature map of the target instance where the seed point is located.

[0051] S104: Perform mask prediction based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; the level of the instance mask includes object level, component level and sub-component level.

[0052] In this embodiment, mask prediction can be performed based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask. Taking a rice plant image as an example, the instance mask can be a rice plant mask, a leaf mask, or a lesion mask. Among them, the rice plant mask is an object-level mask, the leaf mask is a component-level mask, and the lesion mask is a sub-component-level mask.

[0053] S105: Merge multiple object-level instance masks step by step to obtain group masks and overall masks respectively.

[0054] In this embodiment, based on obtaining multiple instance masks, multi-granularity construction can be performed. Specifically, each object-level instance mask can be treated as an individual, and the overall mask can be generated by the pixel union of all individuals of the same type. Grouping individuals of the same type can obtain group masks, and the set mapping between the union and the individuals is recorded.

[0055] S106: Instance segmentation of the target image is performed based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks and global masks.

[0056] In this embodiment, the target image is segmented based on the instance masks of multiple granularities and multiple levels obtained in the above steps, resulting in multi-granularity and multi-level segmentation results of the target image.

[0057] As can be seen from the above, this embodiment first performs multi-scale feature fusion on the multimodal information of the target image to obtain a first fused feature map at multiple scales; then, in the first fused feature map with the highest resolution, seed points are selected based on the response score of each pixel. With each seed point as the center, the neighborhood features of the seed point on the first fused feature maps at multiple scales are aggregated to obtain the instance feature map of the target instance where the seed point is located; based on each instance feature map, mask prediction is performed to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; by treating the object-level instance mask as an individual and merging multiple object-level instance masks step by step, group masks and overall masks can be obtained, thus forming mask outputs of multiple granularities such as individual / group / overall; based on the instance masks of multiple granularities and the instance masks of multiple levels, instance segmentation of the target image can be performed, realizing multi-granularity and multi-level instance segmentation of the target image.

[0058] Among them, the multi-granularity and multi-level instance segmentation results of the target image can be used for different scenarios. Taking the target image as a rice plant image as an example, the detection results of different granularities such as single block, group or whole field can be selected according to the needs, and used to accurately locate diseased plants or provide regional early warning.

[0059] In one embodiment of this application, mask prediction is performed based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask, including:

[0060] Each instance feature map is fused with the first fused feature map with the lowest resolution to obtain a low-resolution feature map;

[0061] Input the low-resolution feature map into the first prediction model to obtain the low-resolution mask;

[0062] Upsample the low-resolution mask to the same scale as the target image to obtain the instance mask corresponding to the instance feature map;

[0063] For each region corresponding to an instance mask, features of that region are selected from the first fused feature map at multiple scales, and the selected features are aggregated into an instance vector.

[0064] Input the instance vector into the second prediction model to obtain the level corresponding to the instance mask.

[0065] In this embodiment, considering that after obtaining the instance feature map, if the mask is predicted directly on the high-resolution layer, the gradient propagation will be slow due to the large spatial size of the high-resolution layer. Therefore, this embodiment performs mask prediction on a lower-resolution feature layer (such as stride=8), and after integrating high-level semantics, it upsamples to recover details.

[0066] Specifically, each instance feature map obtained in step S103 can be fused with the first fusion feature map with the lowest resolution, and then input into the first prediction model to obtain a low-resolution mask. The low-resolution mask is upsampled to the original image size through bilinear interpolation or deconvolution. The upsampled mask, the edge features of the RGB image, and the boundary features of the depth image are then input into a shallow convolution kernel (1×1 or 3×3) for computation to obtain the instance mask. Using the above method, the boundaries of the upsampled mask can be refined by combining RGB edges and depth boundaries, thereby improving the consistency between the mask and the target instance contour.

[0067] Specifically, the first prediction model can adopt the following mask prediction header structure:

[0068] ;

[0069] Among them, the first two layers Convolutional layers (3×3) are responsible for feature extraction and local smoothing. Batch normalization (BN) and the SiLU activation function are used to stabilize training and enhance non-linear representations. Finally... The convolution (3×3 convolution) outputs a single-channel mask prediction map.

[0070] Based on the obtained instance masks, within the region corresponding to each instance mask, the same convergence method as in step S103 is used to converge the first fusion features at multiple scales to obtain an instance vector. Then, the instance vector is input into the second prediction model to obtain the level corresponding to that instance mask.

[0071] Specifically, the second prediction model can be implemented using lightweight fully connected networks such as linear layers or multilayer perceptrons (MLPs). This lightweight fully connected network can output the hierarchical attributes of instance vectors, such as objects, parts, or sub-parts, which belongs to the structural level prediction. This lightweight fully connected network can be called a part prediction head. In addition, the second prediction model also includes a semantic category prediction head, which is used to output the probability distribution of instance vectors in the semantic space (e.g., rice plant, leaf, disease spot, etc.). This prediction head is used to provide semantic priors to guide the model to distinguish instances of different semantic types during the overall training phase, so as to form a semantic-structure mapping relationship.

[0072] It should be noted that semantic category information is only used as an auxiliary supervision signal during the training phase and as a semantic annotation reference during inference, and does not participate in hierarchical logical judgment.

[0073] In one embodiment of this application, multiple object-level instance masks are merged level by level to obtain group masks and overall masks, including:

[0074] The pixels corresponding to all object-level instance masks are merged to obtain the overall mask;

[0075] An adjacency graph is built based on the center points of multiple object-level instance masks;

[0076] Clustering is performed on multiple central points in the adjacency graph based on their local density and distance to obtain multiple connected components;

[0077] The instance masks belonging to the center points of each connected component are merged to obtain the group mask.

[0078] In this embodiment, after obtaining the instance mask corresponding to multiple instance feature maps and the level of the instance mask, a multi-granularity, multi-level structured result can be obtained.

[0079] Specifically, in multi-granularity construction, object-level instance masks can be treated as individuals, and the overall mask is generated by the pixel union of similar individuals, recording the set mapping between the union and the individuals. Simultaneously, an adjacency graph can be constructed on the image plane for the center points of multiple object-level instance masks, and then clustered based on local density and geometric similarity to obtain several connected components with adaptive scales. The specific process is as follows:

[0080] (1) Adjacency graph construction:

[0081] Based on the K-Nearest Neighbors (KNN) algorithm, for each instance mask center point, search for the k nearest neighboring instance masks of the same class (default k=8), and calculate the following combined distance:

[0082] ;

[0083] in, and Let be the coordinates of any two center points. and It is a direction vector. and For depth features, , These are the preset weight parameters.

[0084] (2) Adaptive clustering:

[0085] The above adjacency graph is input into the density clustering algorithm DBSCAN, to... As a similarity metric, DBSCAN automatically estimates the number of clusters and divides them into several connected subsets based on local point density: when the center points are densely distributed and have similar features, they are classified into the same connected component; when the center points span large spatial gaps or have significant depth / hierarchical differences, they are divided into different components.

[0086] For each component, the member mask is unioned and lightly smoothed to generate a group mask, and the relationship mapping between "group ↔ individual set" is recorded.

[0087] In multi-level construction, the instance mask at the object level can be used as the parent mask, and the instance mask at the component level can be used as the child mask. For each child mask... Calculate the relationship with all parent masks Crossover ratio:

[0088] ;

[0089] If the maximum crossover ratio ≥0.7, and the center is offset from the centroid of a certain parent mask. If a child mask is determined to be a child mask of the parent mask, the mapping between the parent mask and the child mask is recorded. This rule ensures that each child mask belongs to only one parent mask, thereby maintaining the consistency of the overall hierarchical structure.

[0090] Using the same method, the component-level instance mask can be used as the parent mask, and the child component-level instance mask can be used as the child mask to calculate the mapping relationship between the component-level mask instances and the child component-level mask instances.

[0091] As can be seen from the above, this embodiment can obtain accurate multi-granularity structured results by merging the instance masks at the object level step by step; by determining the parent mask and child mask based on the hierarchy corresponding to multiple instance masks, and by calculating the intersection-union ratio between the parent mask and child mask to determine the mapping relationship between the parent mask and child mask, accurate multi-level structured results can be obtained.

[0092] In one embodiment of this application, multi-scale feature fusion is performed on multi-modal information to obtain a first fused feature map at multiple scales, including:

[0093] Multi-scale feature fusion is performed on the RGB image and its corresponding depth image to obtain a second fused feature map with multiple scales.

[0094] Based on text control information, the second fusion feature maps at multiple scales are modulated to obtain the first fusion feature maps at multiple scales.

[0095] In this embodiment, the appearance / texture features of the target image are first extracted based on the RGB image, outputting multi-scale features (e.g., stride=4, 8); the geometric / boundary features of the target image are extracted based on the depth image, and the corresponding scale features are output simultaneously. Then, the multi-scale features of the RGB image and the multi-scale features of the depth image are aligned, concatenated along the channel dimension at the same scale, and then compressed to a unified dimension through lightweight convolutional projection to obtain the fused second feature map. Using this method, complementary information between the two modalities can be preserved while avoiding channel redundancy. The spatial dimensions and stride of each scale are recorded, and position / coordinate encoding is added to each scale to facilitate subsequent upsampling operations.

[0096] Finally, based on text control information, the second fusion feature map at multiple scales is modulated to obtain the first fusion feature map aligned with the prompt semantics.

[0097] In one embodiment of this application, the text control information includes text semantic information, granular control information, and hierarchical control information;

[0098] Based on text control information, the second fusion feature maps at multiple scales are modulated to obtain the first fusion feature maps at multiple scales, including:

[0099] Semantic encoding is performed on the text semantic information, granular control information, and hierarchical control information to obtain text semantic encoding vector, granular embedding vector, and hierarchical embedding vector, respectively. For the second fusion feature map at each scale, the text semantic encoding vector, granular embedding vector, and hierarchical embedding vector are input into the mapping network corresponding to that scale to obtain the channel modulation parameters corresponding to that scale. Based on the channel modulation parameters, the features at that scale are amplified or suppressed channel by channel to obtain the third fusion feature map after channel modulation.

[0100] Based on the depth information of each pixel in the depth image, the depth weight corresponding to each pixel is determined to obtain a depth reliability weight map; the similarity between the features of each pixel and the text control information is calculated to obtain a semantic attention map; the semantic attention map and the depth reliability weight map are multiplied pixel by pixel to obtain a spatial weight map; the third fusion feature map is adjusted based on the spatial weight map to obtain a spatially modulated fourth fusion feature map.

[0101] The weight bias coefficients corresponding to the fourth fusion feature map at each scale are adjusted to obtain the first fusion feature map at that scale. Specifically, for the fourth fusion feature map with a resolution greater than the preset resolution, the corresponding weight coefficient is greater than 1, and for the fourth fusion feature map with a resolution less than or equal to the preset resolution, the corresponding weight coefficient is less than 1.

[0102] In this embodiment, on the second fused feature map at each scale, the features are modulated in three steps based on text control information, in the order of "channel → space → granularity / hierarchical bias", and finally the modulated first fused feature map is obtained.

[0103] (1) Channel conditionalization (learnable scaling and bias):

[0104] Textual control information is encoded into a set of channel-level coefficients, which amplify or suppress features at the current scale channel by channel. Intuitively, channels more relevant to the cues are amplified, while weaker channels are suppressed, thus focusing attention on representations more consistent with semantics. The coefficients are learnable, and each scale is independent, facilitating the division of labor between high- and low-resolution layers.

[0105] Specifically, the channel-level scaling and bias coefficients are not fixed constants, but are dynamically generated by a learnable conditional mapping network based on the input text control information. This text control information includes text semantic information, granular control information, and hierarchical control information. Correspondingly, semantic encoding is performed on the text semantic information, granular control information, and hierarchical control information to obtain text semantic encoding vectors, granular embedding vectors, and hierarchical embedding vectors, respectively.

[0106] Text semantic embedding vector Used to represent the semantic content of input instructions, such as specific object categories like "rice plant" and "leaf"; granular embedding vector With hierarchical embedding vectors These are independent structural control signals, used to supplement the "analytical perspective" and "structural hierarchy" information at the task execution level. Specifically, It represents the level of detail in the model's observations across a spatial range, and is used to control the weight balance between overall and local features, i.e., to solve the problem of "how much detail to observe"; This indicates the hierarchical position of the model in the semantic structure, used to indicate whether the current feature belongs to the object level, component level, or sub-component level, i.e., solving the "which level to look at" problem. Text semantic embedding vector and hierarchical embedding vectors It is a "control command" that complements the semantics of the text, that is Tell the model "what to recognize," and and Tell the model "from which angle and with what level of detail should it be identified".

[0107] For example, the text control information is: "Detect disease spots on rice plants", and the text semantic encoding vector is... Semantic categories including "rice plant" and "disease spots" are included; furthermore, granular embedding is also required. To determine whether the analysis is at the "individual plant" level or the "group" level, hierarchical embedding is used. The determination is made by extracting features from "leaf level" or "vein level / lesion level" structures.

[0108] Text semantic encoding vector With granularity, hierarchical embedding vectors After splicing, the data is input into a scale-specific mapping network. The channel modulation parameters for the current scale are obtained:

[0109] ;

[0110] in and Control the amplification and offset of the channels separately. Network structure such as Figure 4 As shown, the concatenated vector First, the parameters are reduced in dimension by a linear transformation layer, then nonlinearly activated by SiLU, and finally generated by a linear transformation layer.

[0111] Training via backpropagation, The network adaptively learns the correspondence between semantic cues and channel responses, amplifying channels that are semantically relevant to the cues and suppressing those with weaker relationships, thereby enhancing semantic consistency and optimizing feature division of labor. Parameters for each scale are generated independently to adapt to the semantic hierarchy of different resolution layers.

[0112] (2) Spatial weighting:

[0113] The previous channel modulation step has enhanced the responses of different semantic channels based on semantic cues, but it still does not explicitly distinguish the importance of spatial location. Therefore, in this step, a spatial weight map is further generated to perform supplementary modulation in the spatial dimension.

[0114] First, a "semantic attention map" is generated using semantically modulated features, which reflects the degree of matching between each pixel and the current text. Then, a depth reliability weight map is obtained from the depth image, and the semantic attention map is multiplied pixel-by-pixel with the depth reliability weight map to obtain:

[0115] ;

[0116] in: This represents the final spatial weight graph. Indicates the intensity of semantic attention. This represents the depth reliability weight.

[0117] Based on this, the second fused feature map can be modulated using the spatial weight map:

[0118] This represents the second fused feature map after spatial weighting. This indicates the characteristics after channel modulation. .

[0119] In this embodiment, the depth reliability weight map obtained based on the depth image can be described in detail as follows:

[0120] Calculate the gradient information of the depth image;

[0121] The region whose gradient information is less than the first gradient threshold is defined as the main region, and the weight of the main region is set to the first value.

[0122] Regions whose gradient information is greater than the second gradient threshold are identified as mutation regions; the weights of mutation regions are set to the second value.

[0123] In the depth image, regions other than the main region and abrupt change regions are defined as transition regions, and the weight of the transition regions is set to a third value; wherein the first value is greater than the third value, and the third value is greater than the second value.

[0124] A deep reliability weight map is obtained based on the weights corresponding to the main region, the mutation region, and the transition region.

[0125] like Figure 5 The diagram illustrates a deep reliability weight map. The spatial weight map is applied to the feature map through pixel-by-pixel multiplication, achieving regional weighting at the spatial level, which complements channel-level modulation. The former controls "semantic channel selection," while the latter controls "spatial saliency region selection." The combination of both significantly improves the quality of target region extraction. Therefore, based on the spatial weight map, responses from background or low-confidence regions can be suppressed, highlighting semantically relevant regions and achieving spatial saliency enhancement.

[0126] (3) Overall bias of granularity and hierarchy:

[0127] In this embodiment, an overall weight bias coefficient can be applied to feature maps of different scales. This achieves a semantic balance where high-resolution layers focus on local details and low-resolution layers focus on the overall outline.

[0128] Specifically, high-resolution layers (with smaller strides) contain rich texture and edge information, therefore... >1.0, to enhance the detail response of objects and components / sub-components; low-resolution layers (larger stride) are more suitable for expressing global shape and object relationships, therefore, they are selected. <1.0, to smooth the response and enhance the overall stability of the group mask / global mask.

[0129] The adjusted scale feature is represented as follows:

[0130]

[0131] in: This is the spatially weighted feature map of the s-th layer; This is the feature map after overall offset adjustment; These are the global weight coefficients for this layer, used to amplify or suppress the intensity of the feature response. (Coefficient) The value range is 0.80 to 1.25, with an initial value of 1.00. This value can be fixed as a constant during the initial training phase, followed by adaptive optimization through backpropagation. If the semantic confidence is low in some layers or the control bits are missing, it can automatically fall back to... = 1.00 to ensure model stability.

[0132] By offsetting the granularity and hierarchy as a whole, high-resolution layers are ensured to focus on depicting details, while low-resolution layers maintain the global outline, achieving a multi-level division of labor where "the upper layer focuses on details and the lower layer preserves the structure".

[0133] After channel conditioning, spatial weighting, and overall biasing, the first fused feature map at each scale is obtained.

[0134] In one embodiment of this application, after obtaining the response score for each pixel, the method further includes:

[0135] Geometric scale estimation of the target image is performed based on RGB images, depth estimation of the target image is performed based on depth images, and the scale prior of the target image is determined based on the results of geometric scale estimation and depth estimation.

[0136] The response score of each pixel is adjusted based on the spatial weight map and scale prior to obtain the final response score;

[0137] Specifically, the first K pixels are selected as seed points according to their corresponding response scores from highest to lowest, including:

[0138] The first K pixels are selected as seed points according to the corresponding final response scores from high to low.

[0139] In this embodiment, the scale prior of the target image can be determined based on the RGB image and the depth image, and then the response score of each pixel can be adjusted based on the scale prior and the spatial weight map. The specific process is as follows:

[0140] (1) Geometric scale estimation:

[0141] Calculate the local gradient variance or Laplacian energy for the input image to estimate the texture density. :

[0142] Regions with denser textures typically correspond to smaller target scales.

[0143] (2) Depth estimation:

[0144] The relative scale information of the target is estimated from the depth map Z(x, y), and the scale weight is obtained through normalization mapping. The greater the depth (i.e., the farther away from the camera), the smaller the corresponding target size. Therefore, its reciprocal is taken as the weight factor, defined as follows:

[0145]

[0146] in: Indicates position Scale weights, This represents the depth value of the corresponding pixel. To prevent division by zero of small constants.

[0147] (3) Fusion and Regularization:

[0148] The geometric scale weights and depth scale weights are combined into a comprehensive prior:

[0149] ;

[0150] The final scores of candidate points are weighted and adjusted during seed selection.

[0151] ;

[0152] In this way, the model automatically improves the response in small target regions and suppresses ineffective activations in large target or background regions, thereby achieving scale-adaptive seed selection.

[0153] In one embodiment of this application, the multi-granularity, multi-level instance segmentation method further includes:

[0154] Calculate the intersection-union ratio between every two instance masks at the same level;

[0155] For every two instance masks with an intersection-to-union ratio greater than the overlap threshold, calculate multiple overlapping pixels between the two instance masks; each pair of instance masks includes a first instance mask and a second instance mask.

[0156] Calculate the first distance between each overlapping pixel and the center point of the first instance mask, and the second distance between each overlapping pixel and the center point of the second instance mask;

[0157] Overlapping pixels with a first distance less than a second distance are assigned to the first instance mask, and overlapping pixels with a first distance greater than a second distance are assigned to the second instance mask.

[0158] In this embodiment, for the multi-granularity and multi-level results obtained in the above embodiments, structural consistency and non-overlapping constraints at the same level can be applied, and conflict resolution can be completed according to the principle of "minimum perturbation" to achieve consistency between levels.

[0159] Specifically, the IoU (Intersection over Union) can be calculated pairwise for instance masks at the same level, and a conflict is identified when the IoU exceeds the overlap threshold. The overlap threshold is adjusted according to the level, such as block≈0.05, plant≈0.20, leaf≈0.30, and lesion≈0.40.

[0160] When local overlap occurs in masks at the same level, to avoid structural breaks or area anomalies caused by simple cropping, we adopt a distance-transform-based redistribution strategy:

[0161] (1) Overlapping region extraction: Extraction of two mask regions , Find the intersection , to obtain the set of overlapping pixels.

[0162] (2) Distance mapping calculation: Calculate the Euclidean distance transformation for the skeleton or outer edge of each of the two masks respectively:

[0163] ;

[0164] Obtain the distance from each overlapping pixel to the corresponding mask backbone (center or boundary).

[0165] (3) Normalized weighted comparison: After normalizing the distance, compare the relative closeness of the two according to the proportion:

[0166]

[0167]

[0168] like > Then the pixel is reassigned to the mask. Conversely, belonging to .

[0169] (4) Smoothing and Correction:

[0170] Apply 3×3 guided filtering or morphological smoothing to the allocation boundaries to remove isolated pixels while maintaining boundary continuity.

[0171] In addition, granularity consistency verification can be performed, including:

[0172] (1) Union Coverage of Individuals: The union mask is formed by combining the individual masks pixel by pixel and is used to represent the whole. When any individual is modified or deleted, the union must be recalculated to ensure that the overall result is equal to the sum of all individuals. Its scope is global, ensuring that "the whole completely covers all members".

[0173] (2) Grouped Coverage of Subsets: Grouped masks are applied to local subsets. Only the individual masks of the members within the group need to be combined pixel by pixel. When an individual within the group is modified, the grouped mask only needs to be regenerated within the group to ensure the consistency between the group and the subset. Its scope is local, ensuring that "the subset equals the sum of the individuals within the group".

[0174] If a block mask or its union shows a serious deviation during consistency checks (such as a missing rate or overlap rate exceeding a threshold), then the block is considered to contain an "invalid sub-body." The corresponding processing logic is as follows:

[0175] If the area of ​​a submask is less than 0.5% of the total area of ​​the block mask, or if the intersection-union ratio of the submask and the block mask satisfies:

[0176]

[0177] The mask is then deemed invalid.

[0178] Invalid masks will be downgraded or deleted, and the group mapping table will be updated synchronously.

[0179] When the cumulative invalid mask exceeds a set percentage (e.g., 20%), the entire packet will be recalculated or manually reviewed.

[0180] The updated mask set is used to regenerate the grouped union. This ensures consistent overall granularity.

[0181] In summary, please refer to the following: Figure 6 This embodiment provides a multi-granularity, multi-level instance segmentation method, which sequentially executes S1: input and preprocessing module, S2: multimodal feature extraction and fusion module, S3: text-controlled access and cross-modal conditionalization module, S4: instance representation and mask generation, S5: multi-granularity / multi-level result construction, and S6: consistency and non-overlapping constraints at the same level. This process unifies multimodal fusion, text controllability, and granularity / level consistency into a unified model, ultimately producing a structured mask set and relationship table for individuals / unions / groups and objects / components / subcomponents. Based on this, multi-granularity, multi-level instance segmentation can be achieved.

[0182] The method described in this embodiment can achieve the following beneficial effects:

[0183] (1) The model simultaneously produces individual / union (all) / group and object / component / subcomponent results, and completes the structured constraints with verifiable rules that the union covers the individual, the group covers the subset, and the parent mask covers all sub-masks, so as to avoid the problem of independent results and inconsistent standards.

[0184] (2) For crowded and strong occlusion, set non-overlapping constraints with IoU≤τ (overlap threshold) at the same level, and perform minimal changes such as pixel redistribution, boundary refinement, and bounding clipping on the instance mask that does not meet the constraints; and refresh the union / grouping and parent-child mapping in a coordinated manner to stably produce a set of instances with no (or low) overlap.

[0185] (3) Implement gating / scaling-bias modulation on multi-scale fusion features to significantly amplify the representations related to the prompts and suppress the irrelevant ones; achieve text-controllable instance and component segmentation, and coordinate with the above consistency rule within the same framework.

[0186] (4) Select seeds by high-level scoring → Multi-scale convergence → Low-resolution mask prediction → Links that are upsampled by stride and lightly attached to the edge, taking into account the small target boundaries and overall efficiency in crowded scenes.

[0187] Based on the same inventive concept, this application also provides a multi-granularity multi-level instance segmentation apparatus for implementing the multi-granularity multi-level instance segmentation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the multi-granularity multi-level instance segmentation apparatus provided below can be found in the limitations of the multi-granularity multi-level instance segmentation method described above, and will not be repeated here.

[0188] This application provides a multi-granularity, multi-level instance segmentation device, such as... Figure 7 As shown, the multi-granularity, multi-level instance segmentation device 20 includes: a data acquisition module 21, a position scoring module 22, a seed selection module 23, a mask prediction module 24, a mask merging module 25, and an instance segmentation module 26.

[0189] The data acquisition module 21 is used to acquire multimodal information of the target image, perform multi-scale feature fusion on the multimodal information, and obtain a first fused feature map at multiple scales; the multimodal information includes text control information, RGB image and its corresponding depth image;

[0190] The location scoring module 22 is used to score each pixel in the first fused feature map with the highest resolution based on a pre-trained scoring model, and obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance;

[0191] The seed selection module 23 is used to select the first K pixels as seed points in descending order of the corresponding response scores. With each seed point as the center, the neighborhood features of the seed point are gathered from the first fusion feature map of multiple scales to obtain the instance feature map of the target instance where the seed point is located.

[0192] The mask prediction module 24 is used to perform mask prediction based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; the level of the instance mask includes object level, component level and sub-component level;

[0193] The mask merging module 25 is used to merge multiple object-level instance masks step by step to obtain group masks and overall masks respectively;

[0194] The instance segmentation module 26 is used to segment the target image based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks and global masks.

[0195] In one embodiment of this application, the mask prediction module 24 is specifically used for:

[0196] Each instance feature map is fused with the first fused feature map with the lowest resolution to obtain a low-resolution feature map;

[0197] Input the low-resolution feature map into the first prediction model to obtain the low-resolution mask;

[0198] Upsample the low-resolution mask to the same scale as the target image to obtain the instance mask corresponding to the instance feature map;

[0199] For each region corresponding to an instance mask, features of that region are selected from the first fused feature map at multiple scales, and the selected features are aggregated into an instance vector.

[0200] Input the instance vector into the second prediction model to obtain the level corresponding to the instance mask.

[0201] In one embodiment of this application, the mask merging module 25 is specifically used for:

[0202] The pixels corresponding to all object-level instance masks are merged to obtain the overall mask;

[0203] An adjacency graph is built based on the center points of multiple object-level instance masks;

[0204] Clustering is performed on multiple central points in the adjacency graph based on their local density and distance to obtain multiple connected components;

[0205] The instance masks belonging to the center points of each connected component are merged to obtain the group mask.

[0206] In one embodiment of this application, the data acquisition module 21 is specifically used for:

[0207] Multi-scale feature fusion is performed on the RGB image and its corresponding depth image to obtain a second fused feature map with multiple scales.

[0208] Based on text control information, the second fusion feature maps at multiple scales are modulated to obtain the first fusion feature maps at multiple scales.

[0209] In one embodiment of this application, the data acquisition module 21 is further configured to:

[0210] Based on text control information, the second fusion feature maps at multiple scales are modulated to obtain the first fusion feature maps at multiple scales, including:

[0211] Semantic encoding is performed on the text semantic information, granular control information, and hierarchical control information to obtain text semantic encoding vector, granular embedding vector, and hierarchical embedding vector, respectively. For the second fusion feature map at each scale, the text semantic encoding vector, granular embedding vector, and hierarchical embedding vector are input into the mapping network corresponding to that scale to obtain the channel modulation parameters corresponding to that scale. Based on the channel modulation parameters, the features at that scale are amplified or suppressed channel by channel to obtain the third fusion feature map after channel modulation.

[0212] Based on the depth information of each pixel in the depth image, the depth weight corresponding to each pixel is determined to obtain a depth reliability weight map; the similarity between the features of each pixel and the text control information is calculated to obtain a semantic attention map; the semantic attention map and the depth reliability weight map are multiplied pixel by pixel to obtain a spatial weight map; the third fusion feature map is adjusted based on the spatial weight map to obtain a spatially modulated fourth fusion feature map.

[0213] The weight bias coefficients corresponding to the fourth fusion feature map at each scale are adjusted to obtain the first fusion feature map at that scale. Specifically, for the fourth fusion feature map with a resolution greater than the preset resolution, the corresponding weight coefficient is greater than 1, and for the fourth fusion feature map with a resolution less than or equal to the preset resolution, the corresponding weight coefficient is less than 1.

[0214] In one embodiment of this application, after obtaining the response score for each pixel, the location scoring module 22 is specifically used for:

[0215] Geometric scale estimation of the target image is performed based on RGB images, depth estimation of the target image is performed based on depth images, and the scale prior of the target image is determined based on the results of geometric scale estimation and depth estimation.

[0216] The response score of each pixel is adjusted based on the spatial weight map and scale prior to obtain the final response score;

[0217] Specifically, the first K pixels are selected as seed points according to their corresponding response scores from highest to lowest, including:

[0218] The first K pixels are selected as seed points according to the corresponding final response scores from high to low.

[0219] In one embodiment of this application, the instance segmentation module 26 is specifically used for:

[0220] Calculate the intersection-union ratio between every two instance masks at the same level;

[0221] For every two instance masks with an intersection-to-union ratio greater than the overlap threshold, calculate multiple overlapping pixels between the two instance masks; each pair of instance masks includes a first instance mask and a second instance mask.

[0222] Calculate the first distance between each overlapping pixel and the center point of the first instance mask, and the second distance between each overlapping pixel and the center point of the second instance mask;

[0223] Overlapping pixels with a first distance less than a second distance are assigned to the first instance mask, and overlapping pixels with a first distance greater than a second distance are assigned to the second instance mask.

[0224] See Figure 8 , Figure 8 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 8 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of the data acquisition module 21, position scoring module 22, seed selection module 23, mask prediction module 24, mask merging module 25, and instance segmentation module 26 are shown.

[0225] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0226] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0227] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store preset constants such as a first gradient threshold, a second gradient threshold, and an overlap threshold.

[0228] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in the multi-granularity and multi-level instance segmentation method provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be elaborated here.

[0229] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0230] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0231] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0232] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0234] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0235] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0236] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-granularity multi-level instance segmentation method, characterized in that, include: The multimodal information of the target image is acquired, and multi-scale feature fusion is performed on the multimodal information to obtain a first fused feature map at multiple scales; The multimodal information includes text control information, RGB images and their corresponding depth images; Based on the pre-trained scoring model, each pixel in the first fused feature map with the highest resolution is scored to obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance. According to the corresponding response scores from high to low, the first K pixels are selected as seed points. With each seed point as the center, the neighborhood features of the seed point are gathered from the first fused feature map at multiple scales to obtain the instance feature map of the target instance where the seed point is located. Mask prediction is performed based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; The instance mask has three levels: object level, component level, and sub-component level. Multiple object-level instance masks are merged level by level to obtain group masks and overall masks respectively; The target image is segmented based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks, and overall masks.

2. The multi-granularity multi-level instance segmentation method of claim 1, wherein, The step of performing mask prediction based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask includes: Each instance feature map is fused with the first fused feature map with the lowest resolution to obtain a low-resolution feature map; The low-resolution feature map is input into the first prediction model to obtain the low-resolution mask; The low-resolution mask is upsampled to the same scale as the target image to obtain the instance mask corresponding to the instance feature map; For each region corresponding to an instance mask, features of that region are selected from the first fused feature map at multiple scales, and the selected features are aggregated into an instance vector. The instance vector is input into the second prediction model to obtain the level corresponding to the instance mask.

3. The multi-granularity multi-level instance segmentation method of claim 1, wherein, The step of merging multiple object-level instance masks level by level to obtain group masks and overall masks includes: The pixels corresponding to all object-level instance masks are merged to obtain the overall mask; An adjacency graph is built based on the center points of multiple object-level instance masks; Clustering is performed on the local density and distance of multiple center points in the adjacency graph to obtain multiple connected components; The instance masks belonging to the center points of each connected component are merged to obtain the group mask.

4. The multi-granularity, multi-level instance segmentation method as described in claim 1, characterized in that, Multi-scale feature fusion is performed on the multimodal information to obtain a first fused feature map at multiple scales, including: Multi-scale feature fusion is performed on the RGB image and its corresponding depth image to obtain a second fused feature map with multiple scales. Based on the text control information, the second fusion feature maps at multiple scales are modulated to obtain the first fusion feature maps at multiple scales.

5. The multi-granularity, multi-level instance segmentation method as described in claim 4, characterized in that, The text control information includes text semantic information, granular control information, and hierarchical control information; Based on the text control information, the second fusion feature maps at multiple scales are modulated to obtain a first fusion feature map at multiple scales, including: Semantic encoding is performed on the text semantic information, the granularity control information, and the hierarchical control information to obtain text semantic encoding vector, granular embedding vector, and hierarchical embedding vector, respectively. For the second fusion feature map at each scale, the text semantic encoding vector, granular embedding vector, and hierarchical embedding vector are input into the mapping network corresponding to that scale to obtain the channel modulation parameters corresponding to that scale. Based on the channel modulation parameters, the features at that scale are amplified or suppressed channel by channel to obtain the third fusion feature map after channel modulation. Based on the depth information of each pixel in the depth image, the depth weight corresponding to each pixel is determined to obtain a depth reliability weight map; the similarity between the features of each pixel and the text control information is calculated to obtain a semantic attention map; the semantic attention map and the depth reliability weight map are multiplied pixel by pixel to obtain a spatial weight map; the third fusion feature map is adjusted based on the spatial weight map to obtain a spatially modulated fourth fusion feature map. The weight bias coefficients corresponding to the fourth fusion feature map at each scale are adjusted to obtain the first fusion feature map at that scale. Specifically, for the fourth fusion feature map with a resolution greater than the preset resolution, the corresponding weight coefficient is greater than 1, and for the fourth fusion feature map with a resolution less than or equal to the preset resolution, the corresponding weight coefficient is less than 1.

6. The multi-granularity, multi-level instance segmentation method as described in claim 5, characterized in that, After obtaining the response score for each pixel, the process also includes: Geometric scale estimation is performed on the target image based on the RGB image, depth estimation is performed on the target image based on the depth image, and scale prior of the target image is determined based on the results of geometric scale estimation and depth estimation. The response score of each pixel is adjusted based on the spatial weight map and the scale prior to obtain the final response score; The step of selecting the first K pixels as seed points according to their corresponding response scores from high to low includes: The first K pixels are selected as seed points according to the corresponding final response scores from high to low.

7. The multi-granularity, multi-level instance segmentation method as described in claim 1, characterized in that, Also includes: Calculate the intersection-union ratio between every two instance masks at the same level; For every two instance masks with an intersection-to-union ratio greater than the overlap threshold, calculate multiple overlapping pixels between the two instance masks; each pair of instance masks includes a first instance mask and a second instance mask. Calculate a first distance between each overlapping pixel and the center point of the first instance mask, and a second distance between each overlapping pixel and the center point of the second instance mask; Overlapping pixels with a first distance less than a second distance are assigned to the first instance mask, and overlapping pixels with a first distance greater than a second distance are assigned to the second instance mask.

8. A multi-granularity, multi-level instance segmentation device, characterized in that, include: The data acquisition module is used to acquire multimodal information of the target image, perform multi-scale feature fusion on the multimodal information, and obtain a first fused feature map at multiple scales. The multimodal information includes text control information, RGB images and their corresponding depth images; The location scoring module is used to score each pixel in the first fused feature map with the highest resolution based on a pre-trained scoring model, and obtain the response score of each pixel; the response score of each pixel is used to characterize the probability that the pixel is the center of the target instance; The seed selection module is used to select the top K pixels as seed points according to the corresponding response scores from high to low. Centered on each seed point, the neighborhood features of the seed point are gathered from the first fused feature map at multiple scales to obtain the instance feature map of the target instance where the seed point is located. The mask prediction module is used to perform mask prediction based on each instance feature map to obtain the instance mask corresponding to the instance feature map and the level of the instance mask; The instance mask has three levels: object level, component level, and sub-component level. The mask merging module is used to merge multiple object-level instance masks step by step to obtain group masks and overall masks respectively; The instance segmentation module is used to perform instance segmentation on the target image based on instance masks of multiple granularities and instance masks of multiple levels; the instance masks of multiple granularities include object-level instance masks, group masks, and overall masks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images