Image segmentation method, electronic device, and computer program product

Abstract

The present disclosure provides an image segmentation method, an electronic device and a computer program product. The image segmentation method of the present disclosure comprises: downsampling a first image acquired by a drone to determine a second image; extracting structural features of the first image to determine the structural features of the first image; and extracting semantic features of image representation content of the second image to determine the semantic features of the second image; under the same spatial dimension, fusing the structural features of the first image and the semantic features of the second image to determine visual features; inferring the visual features and the acquired text instructions to determine the mask features of the target to be segmented in the first image; and based on the mask features of the target to be segmented in the first image, performing hierarchical decoding on feature maps of different scales to obtain a mask image, the feature maps of different scales being obtained in the process of extracting the structural features of the first image.

Description

Technical Field

[0001] This disclosure relates to an image segmentation method, an electronic device, and a computer program product. Background Technology

[0002] Traditional image segmentation methods often rely solely on single-scale or single-modal features for segmentation. Under the imaging conditions of drones at high altitudes with tilted angles and extreme scale differences, it is difficult to simultaneously maintain global semantic understanding and local pixel restoration capabilities when processing high-resolution aerial images. Summary of the Invention

[0003] This disclosure provides an image segmentation method, an electronic device, and a computer program product.

[0004] According to one aspect of this disclosure, an image segmentation method is provided, comprising: downsampling a first image acquired by a drone to determine a second image; extracting structural features from the first image to determine structural features of the first image; extracting semantic features from the image representation content of the second image to determine semantic features of the second image; fusing the structural features of the first image and the semantic features of the second image in the same spatial dimension to determine visual features; reasoning about the visual features and acquired text instructions to determine mask features of a target to be segmented in the first image; and performing layered decoding of feature maps at different scales based on the mask features of the target to be segmented in the first image to obtain a mask image, wherein the feature maps at different scales are obtained during the process of extracting structural features from the first image.

[0005] According to the image segmentation method disclosed herein, a low-resolution second image is obtained by downsampling a first image captured by a UAV. Structural and semantic features are extracted from both the original first and second images, respectively, and then these two features are fused under a unified spatial dimension to form a visual feature. Subsequently, this visual feature is used for inference processing in conjunction with acquired text instructions to determine the mask features of the target to be segmented in the image. Finally, based on this mask feature, feature maps at different scales are decoded hierarchically to generate an accurate mask image. This method integrates local details and global semantic information, effectively improving the accuracy and efficiency of image segmentation, and enabling accurate segmentation of small targets and image regions rich in detail.

[0006] According to at least one embodiment of the image segmentation method of this disclosure, in the same spatial dimension, the structural features of the first image and the semantic features of the second image are fused to determine visual features, including: converting the structural features of the first image to the spatial dimension of the semantic features of the second image; performing channel projection on the structural features of the converted first image; aligning the feature dimensions of the structural features of the converted first image with the feature dimensions of the semantic features of the second image; and cross-fusion the semantic features of the second image after feature dimension alignment and the structural features of the first image after feature dimension alignment to determine visual features.

[0007] The image segmentation method according to at least one embodiment of the present disclosure performs cross-fusion of semantic features of a second image aligned with feature dimensions and structural features of a first image aligned with feature dimensions to determine visual features, including: applying attention weighting to the structural features of the first image aligned with feature dimensions based on the semantic features of the second image aligned with feature dimensions; and applying the weighted structural features inversely to the semantic features aligned with feature dimensions to determine visual features.

[0008] According to at least one embodiment of the image segmentation method of this disclosure, structural feature extraction is performed on the first image to determine the structural features of the first image, including: multi-scale structural feature extraction of image edges, image textures and / or targets to be segmented in the first image to obtain feature maps of different scales; and fusion of feature maps of different scales to determine the structural features of the first image.

[0009] According to at least one embodiment of the image segmentation method of this disclosure, a mask image is obtained by hierarchically decoding feature maps of different scales based on the mask features of the target to be segmented in the first image, including: spatially modulating feature maps of different scales based on the mask features of the target to be segmented in the first image; performing mask prediction on the spatially modulated feature maps of different scales based on the mask features of the target to be segmented in the first image to obtain candidate masks of the target to be segmented in the first image; and linearly fusing the candidate masks of the target to be segmented in the first image to obtain a mask image.

[0010] According to at least one embodiment of the image segmentation method of this disclosure, reasoning is performed on the visual features and the acquired text instructions to determine the mask features, thought chain text, and response of the target to be segmented in the first image.

[0011] The image segmentation method according to at least one embodiment of the present disclosure supervises the process of determining a masked image by weighted calculation of text cross-entropy loss, binary cross-entropy loss, and / or Dice loss.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform an image segmentation method according to any embodiment of this disclosure.

[0013] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the image segmentation method of any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image segmentation method of any embodiment of this disclosure. Attached Figure Description

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 This is a schematic diagram of the overall process of an image segmentation method according to one embodiment of the present disclosure.

[0017] Figure 2 This is a flowchart illustrating the determination of structural features of a first image in an image segmentation method according to one embodiment of the present disclosure.

[0018] Figure 3 This is a flowchart of a method for determining visual features according to one embodiment of the present disclosure.

[0019] Figure 4 This is a flowchart of determining visual features in an image segmentation method according to yet another embodiment of the present disclosure.

[0020] Figure 5 This is a flowchart of an image segmentation method for obtaining a mask image according to one embodiment of the present disclosure.

[0021] Figure 6 This is an architecture diagram of an image segmentation method according to one embodiment of the present disclosure.

[0022] Figure 7 This is a comparison diagram of a multimodal language model of an image segmentation method according to one embodiment of the present disclosure with other methods.

[0023] Figure 8 This is a visualization of the image segmentation result of a multimodal language model according to an embodiment of the image segmentation method of this disclosure.

[0024] Figure 9 This is a schematic structural block diagram of an image segmentation apparatus according to one embodiment of the present disclosure.

[0025] Figure 10 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0026] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0027] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] In the scenario of drone power line inspection, maintenance personnel need to quickly locate areas with corona discharge on suspension insulator strings or potential hazards around tower bases covered by weeds. However, traditional image segmentation methods can only identify a limited number of categories such as pre-trained insulators and weeds, and cannot understand complex semantic instructions. At the same time, aerial images have high resolution, small targets, and complex backgrounds. Single-scale features can easily lead to blurred edges or missed detections. Pure text-driven image segmentation models lack the ability to perceive local structural details (such as cracks and corrosion), resulting in incomplete boundaries in the final output mask image, which is often misaligned with the actual target position, affecting the efficiency of fault diagnosis and the reliability of automated decision-making.

[0029] To address this, this disclosure proposes an image segmentation method. The method involves acquiring a first image captured by a UAV and generating a second image through downsampling. Structural features from the first image and semantic features from the second image are extracted separately and fused under a unified spatial dimension to form visual features that combine local details and global semantics. These features are then combined with textual instructions to generate mask features. Based on the mask features, multi-scale feature maps are hierarchically decoded to output the final mask image. This method achieves cross-path collaborative modeling of high-resolution structural information and low-resolution semantic information, supports open-vocabulary segmentation driven by natural language instructions, significantly improves the segmentation accuracy of complex small targets and blurred boundaries, and reduces computational redundancy by reusing multi-scale features, thus balancing high performance and engineering practicality.

[0030] The image segmentation method disclosed herein enables lightweight, real-time segmentation on terminal devices, such as on-site identification of insulator damage areas during power line inspections and real-time marking of pest-infested plots by agricultural drones, without relying on network data transmission. On mobile terminals, users can use voice or text commands, and the system will instantly generate and overlay masks for use in security monitoring, intelligent navigation, or assisting visually impaired individuals. On the server side, it can batch process massive amounts of aerial imagery for high-precision tasks such as building outline extraction in urban planning, damaged area statistics in disaster assessment, and land cover classification in land surveying.

[0031] Figure 1 A schematic diagram illustrating the overall flow of an image segmentation method according to one embodiment of this disclosure is shown. Figure 1 The method M100 shown includes steps S110 to S150. This method can be executed by a mobile device, a terminal device, or a server.

[0032] In step S110, the first image acquired by the drone is downsampled to determine the second image.

[0033] Preferably, the original high-resolution first image (e.g., 1024×1024 pixels) is compressed to a lower resolution (e.g., 448×448 pixels) by a fixed ratio using a standard image resampling algorithm (e.g., bilinear interpolation or average pooling) to generate the second image.

[0034] In step S120, structural features are extracted from the first image to determine its structural features; and semantic features are extracted from the image representation content of the second image to determine its semantic features.

[0035] By extracting local geometric and texture details from the high-resolution first image, the structural features of the first image are obtained; and by extracting global semantic context from the image representation content of the low-resolution second image, the semantic features of the second image are obtained, providing a complementary and complementary enhanced basic representation for subsequent cross-path fusion and text instruction-guided reasoning.

[0036] Preferably, structural features are extracted from the first image using a high-resolution visual encoder based on the SAM model.

[0037] Preferably, the second image is subjected to global semantic feature extraction using a low-resolution visual encoder based on the CLIP model.

[0038] In step S130, the structural features of the first image and the semantic features of the second image are fused in the same spatial dimension to determine the visual features.

[0039] By engaging cross-path information interaction within a unified spatial dimension, a visual representation is generated that includes both local geometric details and global semantic context. Without introducing additional image data, high-resolution image boundaries and textures are combined with semantic flow, taking into account both global context and local details. This reduces the edge localization error of small targets while keeping computational and memory usage almost unchanged.

[0040] In step S140, reasoning is performed on the visual features and the acquired text instructions to determine the mask features of the target to be segmented in the first image.

[0041] Preferably, the fused visual features and the input natural language instructions are input into a large language model, and cross-modal alignment and joint reasoning are performed in the semantic space to finally generate mask features. These mask features encode the semantic attributes, spatial location clues, and instance boundary information of the target to be segmented in the image.

[0042] Optionally, reasoning is performed on visual features and acquired text instructions to determine the mask features, thought chain text, and response of the target to be segmented in the first image. This visualizes how the large language model understands instructions and locates the target to be segmented, thereby improving the interpretability of the large language model and the efficiency of human-computer collaboration.

[0043] In step S150, based on the mask features of the target to be segmented in the first image, the feature maps of different scales are decoded in layers to obtain the mask image. The feature maps of different scales are obtained during the process of extracting structural features from the first image.

[0044] The mask features are gradually mapped back to the original image space. Through layer-by-layer modulation and fusion of multi-scale structural feature maps, a high-resolution, well-defined, and fully filled pixel-level mask image is finally output. By fully utilizing the multi-level spatial information retained in the structural feature extraction stage, accurate reconstruction from semantic representation to pixel coordinates is achieved while maintaining computational efficiency.

[0045] Preferably, based on the mask features of the target to be segmented in the first image, a mask image is obtained by performing layered decoding of feature maps at different scales through a three-layer multi-scale decoder.

[0046] Therefore, the image segmentation method disclosed herein combines local geometric and texture detail extraction from a high-resolution image (i.e., the first image) with global semantic context understanding from a low-resolution image (i.e., the second image). It integrates visual features within the same spatial dimension using a cross-path fusion strategy, performs text-guided reasoning based on these visual features, and finally generates a precise pixel-level mask image through hierarchical decoding of multi-scale structural feature maps. This not only achieves high-precision localization and segmentation of the target but also maintains high computational efficiency and memory usage efficiency, significantly reduces errors in small target edge localization, and significantly improves the accuracy and interpretability of the segmentation results, providing strong technical support for UAV image analysis in complex environments.

[0047] Regarding step S120, extracting structural features from the first image and determining the structural features of the first image, in some embodiments of this disclosure, may include, for example: Figure 2 Steps S1201 to S1202 are shown.

[0048] In step S1101, multi-scale structural feature extraction is performed on the image edges, image textures and / or the target to be segmented in the first image to obtain feature maps at different scales.

[0049] From the original high-resolution first image, a multi-scale perception mechanism is used to extract multi-level structural information covering the global layout to local details. The focus is on structural elements such as the geometric boundaries, image texture and target contours of the image to obtain a set of feature maps with spatial hierarchical relationships.

[0050] In step S1102, feature maps of different scales are fused to determine the structural features of the first image.

[0051] Feature maps at different scales are fused through a collaborative mechanism across spatial and channel dimensions to obtain the structural features of the first image. These structural features not only preserve local details such as edges and textures but also integrate structural information such as target contours and overall layout.

[0052] Therefore, it effectively enhances the perception of small targets, complex boundaries, and occluded scenes, and significantly improves the accuracy of subsequent semantic-guided segmentation.

[0053] Regarding step S130, in the same spatial dimension, the structural features of the first image and the semantic features of the second image are fused to determine visual features. Structural features of the first image are extracted to determine its structural features. In some embodiments of this disclosure, this may include, for example... Figure 3 Steps S1301 to S1303 are shown.

[0054] In step S1301, the structural features of the first image are converted to the spatial dimension of the semantic features of the second image.

[0055] Preferably, the structural features of the first image are spatially resampled to align the spatial dimension of the structural features of the first image with the spatial dimension of the semantic features of the second image.

[0056] In step S1302, channel projection is performed on the structural features of the converted first image, and the feature dimensions of the structural features of the converted first image and the semantic features of the second image are aligned.

[0057] Preferably, the transformed structural and semantic features are dimensionally aligned using a learnable linear projection matrix.

[0058] In step S1303, the semantic features of the second image after feature dimension alignment and the structural features of the first image after feature dimension alignment are cross-fused to determine the visual features.

[0059] Therefore, it effectively solves the problem of heterogeneity of multimodal features in spatial and channel dimensions, and improves the accuracy and robustness of semantic-guided segmentation.

[0060] Regarding step S1303, the semantic features of the second image after feature dimension alignment and the structural features of the first image after feature dimension alignment are cross-fused to determine visual features. In some embodiments of this disclosure, this may include, for example: Figure 4 Steps S410 to S420 are shown.

[0061] In step S410, attention weighting is applied to the structural features of the first image after feature dimension alignment based on the semantic features of the second image after feature dimension alignment.

[0062] By dynamically weighting the spatial locations of structural features using semantic features, we can focus more on target regions that are semantically related to the current text instruction in subsequent processing.

[0063] In step S420, the weighted structural features are applied inversely to the semantic features aligned with the feature dimensions to determine the visual features.

[0064] The semantically guided weighted structural features are used to locally calibrate and refine the original semantic features, thereby constructing visual features that are more semantically accurate and spatially consistent.

[0065] Therefore, this two-way closed-loop fusion mechanism significantly improves the target positioning accuracy and anti-interference capability under open vocabulary instructions.

[0066] Regarding step S150, based on the mask features of the target to be segmented in the first image, the feature maps of different scales are decoded layer by layer to obtain a mask image. In some embodiments of this disclosure, this may include, for example... Figure 5 Steps S1501 to S1503 are shown.

[0067] In step S1501, based on the mask features of the target to be segmented in the first image, spatial modulation is performed on feature maps of different scales.

[0068] The mask features are mapped to a set of spatial modulation parameters, which are applied to the feature map at the corresponding scale to enhance the response in the target-related region and suppress activation in the non-target region.

[0069] In step S1502, based on the mask features of the target to be segmented in the first image, mask prediction is performed on the spatially modulated feature maps of different scales to obtain candidate masks of the target to be segmented in the first image.

[0070] The multi-scale structural feature maps, after semantically guided spatial modulation, are transformed into preliminary pixel-level mask prediction results through a lightweight convolutional head, forming candidate masks at multiple scales.

[0071] In step S1503, the candidate masks of the target to be segmented in the first image are linearly fused to obtain a mask image.

[0072] Candidate masks are linearly weighted and fused using learnable weights to generate a mask image.

[0073] Therefore, by using masked images generated from multi-level feature information, semantic-to-pixel accurate reconstruction can be achieved in one go, thus improving the accuracy of image segmentation.

[0074] In one specific embodiment, the process of determining the mask image is supervised by weighted calculation of text cross-entropy loss, binary cross-entropy loss, and / or Dice loss. This achieves multi-dimensional collaborative supervision of the mask generation process. Text cross-entropy guides semantic consistency, ensuring that the segmentation results conform to natural language instructions. Binary cross-entropy optimizes pixel-level classification accuracy. Dice loss enhances the overall integrity and boundary fit of the target region. The weighted combination of these three factors effectively balances local details and global structure, significantly improving the segmentation accuracy of small targets, blurred edges, and open-vocabulary instructions in complex scenes, while avoiding overfitting or boundary degradation problems caused by a single loss function.

[0075] The technical solution of this disclosure will be further explained below with a specific application example.

[0076] like Figure 6As shown, the input consists of the raw, high-resolution first image captured by the drone and free natural language commands (i.e., text commands); the output of the multimodal language model (PixDLM) is a pixel mask. In addition, there are explanatory answers and thought chain texts generated by the large language model. The main body of the multimodal language model consists of three major modules: Dual-Path Vision Encoder, Multimodal Large Language Model (Multimodal LLM), and Hierarchical Reasoning Decoder.

[0077] First, in the global scope path (i.e., the global feature path), the first image is downsampled to a lower resolution to reduce token cost and make it usable for large language model inference. A low-resolution visual encoder using a CLIP-based ViT model is employed to extract global semantic features from the second image. Then, in the fine scope path (i.e., the pixel-level detail path), a high-resolution structural encoder using a SAM Hiera-L model is employed to extract structural features from the first image to preserve boundary and small object details. The spatial resolution of the structural features in the first image is significantly higher than that of the semantic features in the second image, which is used to inject structural information during the decoding or alignment stages.

[0078] The structural features of the first image are downsampled and projected, while the semantic features of the second image are projected onto a shared dimension. : Where Resizer represents the layer-by-layer resampling operation. For lightweight learnable matrices, Indicates structural features, This represents the structural features after spatial resampling and channel projection. Represents semantic features, This represents the semantic features after channel projection.

[0079] Furthermore, an attention mechanism is employed to facilitate interaction between the projected and aligned structural and semantic features: , This formula represents the semantic weighting and transformation of structural features based on semantic features. For learnable matrices, Norm is used for layer normalization / residual processing. This represents the structural features enhanced by semantic attention. Indicates a shared dimension.

[0080] The structural features enhanced by semantic attention are routed back to the original semantic features. ,in, This represents the semantic features after structural feedback calibration. yes Convolution or linear mapping, It is a learnable scaling factor used to balance the injected structural information. By layering this bidirectional mechanism across the global scope path, coarse-to-fine semantic structure coupling can be achieved. Finally, multi-path fusion, with linear fusion at the final stage of the global scope path branches, produces visual token representations, i.e., visual features, for use by large language models. , in, Indicates visual characteristics, This represents the final semantic features after structure-to-semantic back-feedback calibration. This represents the final structural features after semantic-to-structural attention enhancement. This fusion preserves key structural information without significantly increasing the number of tokens.

[0081] The visual features of the first image, produced and aligned by the dual-path encoder, are input together with the acquired text instructions (i.e., the UAV reasoning problem) into a large language model (e.g., LLaMA-2-13B). A learnable mask token is added to the input sequence, so that the large language model naturally encodes the mask semantics into the hidden state of the mask token during the language reasoning process. This allows the multimodal reasoning results to be stored in the representation of the mask token in vector form, thus obtaining mask features for subsequent refinement of the decoder.

[0082] The hierarchical inference decoder is a three-level decoder that receives high-resolution visual features at each level. (From several intermediate layers of the high-resolution path of the SAM model) and the mask predicted by the previous layer. (when hour (A coarse mask can be obtained by projecting the hidden state of the Mask Token), and features can be modulated through the mask. , This represents a sigmoid or other smoothing map, the purpose of which is to amplify high-confidence regions and suppress noisy regions. This represents enhanced visual features after semantically guided spatial modulation.

[0083] The enhanced visual features, after semantically guided spatial modulation, are fused with Mask Token embeddings from a large language model using multi-head attention / convolution, and the current layer mask is output. The prediction is supervised by both BCE loss and Dice loss to improve pixel reconstruction quality.

[0084] Among them, after obtaining all intermediate masks Then use learnable layer weight coefficients They are linearly fused to produce the final mask image. : This layering process achieves a coarse-to-fine refinement from semantic mask to pixel mask, which can effectively restore small targets and boundary details while maintaining semantic consistency.

[0085] Specifically, a hybrid loss function is adopted to balance text understanding and pixel output quality; the overall loss of the multimodal language model is... , in, , , It is text cross-entropy, used to maintain the pre-training capability and instruction understanding of large language models on language tasks; It is a pixel-level binary cross-entropy (BCE) that supervises the pixel-by-pixel matching of the output mask and the ground-truth mask; It uses Dice loss to assess regional overlap and improve the alignment of boundaries with small targets.

[0086] During training, most pre-trained model parameters (e.g., CLIP, SAM, LLM) are frozen, and only the LoRA layer and newly added modules (e.g., MultiPath Alignment, Hierarchical Decoder) are fine-tuned. DeepSpeed ​​ZeRO2, AdamW optimizer, and learning rate are used. Training hyperparameters (8 batches, etc.) Using an RTX3090 processor, batch size = 2, gradient accumulation in 10 steps, and training for 10 epochs, this approach achieves effective fine-tuning and parameter budget control. This strategy allows only about 4.19M parameters to be trained, a very small proportion of the total 7.3B parameters, saving GPU memory while preserving pre-trained knowledge.

[0087] In one specific embodiment, high-resolution images of multiple scenes are collected from CODrone and other drone image resources, covering urban roads, parks, ports, residential areas, construction sites, etc., and metadata such as shooting altitude (30m / 60m / 100m) and lighting conditions (day / night) are recorded. Images with higher reasoning difficulty are retained through manual screening, including those with significant scale changes, dense small targets, occlusion / blurring, or tilted perspectives. Approximately 10,000 images are retained after screening.

[0088] Based on the rotated bounding box annotations provided by CODrone, an initial instance mask was generated using the SAM2 model. Further refinement of the mask boundaries was achieved using semi-automatic annotation tools such as ISAT, particularly for thin structures and small targets, to improve mask quality. The aim was to generate high-quality pixel-level masks, laying the foundation for subsequent semantic annotation and verification.

[0089] Input image descriptions, mask information, and category labels into the GPT-5 model, and guide the model to generate structured output using three types of prompt templates (spatial relationships, attribute descriptions, and scene understanding). The output is organized in JSON format, including the question, reasoning chain, and answer, ensuring consistent formatting for subsequent automatic parsing and validation.

[0090] The automatically generated "question-inference chain-mask" triples are reviewed by annotators with remote sensing or computer vision backgrounds. The evaluation dimensions include: target uniqueness and identifiability rate: whether the question can uniquely identify the target object; inference chain consistency score (CoT-Score): the degree of matching between the reasoning process and visual evidence; and mask-text alignment rate (MaskAlign-Rate): whether the mask area is consistent with the text description.

[0091] The validated data is divided into training, validation, and test sets in a 3:2:5 ratio to ensure a balanced distribution of inference types across the subsets, which are then used to train and validate the multimodal language model.

[0092] This disclosure is based on the constructed DRSeg UAV inference segmentation dataset for training and testing. This dataset contains over 10,000 high-resolution aerial images and provides three types of inference annotations (spatial, attribute, and scene). The corresponding natural language instructions and pixel-level instance masks are generated by an automated inference annotation process (GPT-5 + manual verification). Furthermore, this disclosure employs a 3:2:5 training, verification, and testing partitioning method to ensure a balanced distribution across scenes, scales, and inference types, thereby systematically evaluating the model's language inference capabilities and pixel-level expressive power.

[0093] All experiments were implemented in the PyTorch framework and trained on an 8×NVIDIA RTX 3090 GPU environment, using DeepSpeed ​​ZeRO2 for memory optimization and distributed management of large model parameters. The model's default input image size is 1024×1024, and it uses a multi-path alignment module and a three-level inference decoder for inference. Inference time statistics are based on the average inference latency per image, with a measured average of 1.12s / image, corresponding to 0.89 FPS, which still demonstrates acceptable real-time performance while maintaining high-resolution structure and multi-level inference.

[0094] This disclosure employs two types of metrics in quantitative evaluation: Generalized IoU (gIoU) and Classical IoU (cIoU) to measure the degree of overlap between the prediction mask and the ground truth mask; independent performance is calculated for three inference types to analyze the model's performance under different semantic difficulty conditions. All metrics are based on the DRSeg test set.

[0095] like Figure 7 As shown, the proposed PixDLM model outperforms existing multimodal reasoning and segmentation methods across the entire DRSeg test set, including SegEarth-R1, PixelLM-7B, LISA-7B-v1, Seg-Zero-7B, GeoPix-7B, and other baseline large-scale visual models. Compared to general LMMs that lack pixel-level reasoning capabilities, the PixDLM model achieves significant improvements in all three reasoning tasks. Specifically, spatial reasoning improves gIoU by approximately +7.5% and cIoU by +8.2% compared to the best baseline; attribute reasoning improves gIoU by +6.3% and cIoU by +7.1%; and scene reasoning, due to its involvement in complex global semantic relationships, shows an even greater improvement over the baseline, with gIoU reaching 62.73% and cIoU reaching 64.37%, both the highest values ​​on the current test set. Figure 8 As shown, the PixDLM model demonstrates its significant advantages in complex drone scenarios involving multiple entities, scales, and perspectives.

[0096] These results clearly demonstrate that the dual-path alignment mechanism, Mask Token semantic injection, and three-level hierarchical decoding structure disclosed herein can achieve powerful and robust pixel-level inference capabilities under free natural language instructions. More detailed results show that the performance improvement is most significant in scenes with substantial scale variations, dense small objects, and complex cross-viewpoint scenarios.

[0097] Based on any of the above embodiments, this disclosure also provides an image segmentation apparatus.

[0098] Figure 9This is a schematic block diagram of an image segmentation apparatus according to one embodiment of the present disclosure.

[0099] like Figure 9 As shown, the image segmentation device includes: Image sampling module 9002 downsamples the first image acquired by the UAV to determine the second image; The feature extraction module 9004 performs structural feature extraction on the first image to determine the structural features of the first image; and performs semantic feature extraction on the image representation content of the second image to determine the semantic features of the second image. The feature fusion module 9006 fuses the structural features of the first image and the semantic features of the second image in the same spatial dimension to determine the visual features; The inference module 9008 infers the visual features and the acquired text instructions to determine the mask features of the target to be segmented in the first image. The image decoding module 9010 performs layered decoding on feature maps of different scales based on the mask features of the target to be segmented in the first image to obtain a mask image. The feature maps of different scales are obtained during the process of extracting structural features from the first image.

[0100] The image segmentation device described above can be in the form of computer software, and each module of the image segmentation device can be implemented through computer software modules.

[0101] The specific implementation process of the functions and roles of each module in the above image segmentation device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0102] This disclosure also provides an electronic device 1000. Figure 10 A schematic diagram of the hardware implementation using the processing system is shown.

[0103] The hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this figure, but this does not indicate that there is only one bus or one type of bus.

[0104] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0105] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0106] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0107] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0108] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0114] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. An image segmentation method, characterized in that, include: The first image acquired by the drone is downsampled to determine the second image; Structural features are extracted from the first image to determine its structural features; And extract semantic features from the image representation content of the second image to determine the semantic features of the second image; Within the same spatial dimension, the structural features of the first image and the semantic features of the second image are fused to determine the visual features; The visual features and the acquired text instructions are used to infer the mask features of the target to be segmented in the first image. as well as Based on the mask features of the target to be segmented in the first image, the feature maps at different scales are decoded in layers to obtain the mask image. The feature maps at different scales are obtained during the process of extracting structural features from the first image.

2. The image segmentation method as described in claim 1, characterized in that, Within the same spatial dimension, the structural features of the first image and the semantic features of the second image are fused to determine visual features, including: The structural features of the first image are transformed into the spatial dimension of the semantic features of the second image; Channel projection is performed on the structural features of the transformed first image, and the feature dimensions of the structural features of the transformed first image are aligned with the feature dimensions of the semantic features of the second image. The semantic features of the second image after feature dimension alignment and the structural features of the first image after feature dimension alignment are cross-fused to determine the visual features.

3. The image segmentation method as described in claim 2, characterized in that, The semantic features of the second image (aligned by feature dimensions) and the structural features of the first image (aligned by feature dimensions) are cross-fused to determine visual features, including: Based on the semantic features of the second image after feature dimension alignment, attention weighting is applied to the structural features of the first image after feature dimension alignment. The weighted structural features are then applied inversely to the semantic features aligned with the feature dimensions to determine the visual features.

4. The image segmentation method as described in claim 1, characterized in that, Structural feature extraction is performed on the first image to determine its structural features, including: Multi-scale structural feature extraction is performed on the image edges, image textures, and / or the target to be segmented in the first image to obtain feature maps at different scales; By fusing feature maps at different scales, the structural features of the first image are determined.

5. The image segmentation method as described in claim 1, characterized in that, Based on the mask features of the target to be segmented in the first image, the feature maps at different scales are decoded hierarchically to obtain a mask image, including: Based on the mask features of the target to be segmented in the first image, spatial modulation is performed on feature maps of different scales; Based on the mask features of the target to be segmented in the first image, mask prediction is performed on the feature maps of different scales after spatial modulation to obtain the candidate mask of the target to be segmented in the first image. The candidate masks of the target to be segmented in the first image are linearly fused to obtain a mask image.

6. The image segmentation method as described in claim 1, characterized in that, The visual features and the acquired text instructions are used to infer the mask features, thought chain text, and response of the target to be segmented in the first image.

7. The image segmentation method as described in claim 1, characterized in that, The process of determining the mask image is supervised by calculating the text cross-entropy loss, the binary cross-entropy loss, and / or the Dice loss using weighted calculations.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the image segmentation method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the image segmentation method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method according to any one of claims 1 to 7.

Images