An unsupervised image segmentation model and segmentation method for drone scenes

Abstract

The present invention provides an unsupervised image segmentation model and segmentation method for drone scenes, the model includes a self-attention feature extraction module, a linear segmentation head, a self-hint generator, a SAM segmentation module and an image level fusion module; the self-attention feature extraction module extracts the key features of the original drone image, obtains the self-attention map, and sends it to the linear segmentation head and the self-hint generator respectively; the linear segmentation head uses a lightweight segmentation head to extract a rough mask; the self-hint generator is used for bounding box hint generation and point hint generation; the SAM segmentation module is used to obtain two segmentation maps with different granularity based on box hints and point hints; the image level fusion module filters out the segmentation results that do not contain foreground objects, fuses the filtered results with the extracted rough mask and outputs them, and obtains a high-quality segmentation map. The present invention greatly reduces the difficulty of annotation caused by the high resolution and wide field of view of drone images, and significantly improves task efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle image recognition and segmentation based on computer vision, and in particular relates to an unsupervised image segmentation model and a segmentation method for unmanned aerial vehicle scenes. Background Art

[0002] The task of object localization aims to segment the foreground objects in the image, which plays an important role in visual perception systems such as autonomous driving and security monitoring. Achieving high-precision segmentation results usually requires fully supervised training on large-scale carefully annotated datasets. However, annotating large-scale datasets requires extremely high manpower and time costs. In this case, researchers began to focus on methods that reduce dependence on dataset annotation, such as semi-supervised, weakly supervised, and unsupervised learning. In drone scenarios, the need for unsupervised image segmentation and object localization is particularly urgent. Drones are often used in monitoring, surveying, search and rescue tasks. The images taken by drones have the characteristics of high resolution and wide field of view. However, this also increases the difficulty and complexity of data annotation. The limitations of traditional fully supervised methods are particularly prominent in this case. Therefore, studying an unsupervised image segmentation model and method that does not rely on a large amount of annotated data is of great significance for improving the efficiency and accuracy of drone image processing.

[0003] Earlier methods mainly rely on handcrafted methods or dataset-level image search, which are computationally difficult to scale to large datasets. In recent years, with the introduction of self-supervised visual feature learning models, such as DINO (Emerging Properties in Self-Supervised Vision Transformers), which can learn feature representations of objects on unlabeled data, it has greatly promoted the development of unsupervised object localization. LOST (Localizing Objects with Self-Supervised Transformers and no Labels) locates the entire object by determining the foreground seed from the self-supervised feature map; TokenCut (Self-supervised Transformers for Unsupervised Object Discovery using Normalized Cut) locates objects through dual partitioning; SelfMask (Unsupervised Salient Object Detection with Spectral Cluster Voting) calculates segmentation on different features and votes for the best result. WSCUOD (Weakly-supervised Contrastive Learning for Unsupervised Object Discovery) extends weakly supervised contrastive learning to unsupervised object discovery, extracting the most discriminative regions in image representations through principal component analysis. Found (Unsupervised Object Localization: Observing the Background To Discover Objects) proposes to find background seed inversions on feature maps to obtain foreground objects, and train a lightweight model to obtain object segmentation results in the image. PaintSeg (Training-free Segmentation via Painting) uses a generative model to segment image objects based on heterogeneous visual cues.

[0004] The best unsupervised target localization models mentioned above mainly rely on feature maps extracted by self-supervised models and pixel-level similarity, but lack image-level information supervision. This limitation makes it difficult for the model to obtain high-quality mask results. These unsupervised models are not accurate enough in locating objects, lack completeness in segmenting objects, and have unclear boundary positioning. There are also a large number of erroneous pixel segmentation results, resulting in low final target localization accuracy.

[0005] At the same time, the general segmentation model SAM (Segment Anything Model) demonstrates strong object understanding capabilities, providing fine-grained object segmentation results based on specific cues (such as points or boxes). However, in the absence of supervised cues, SAM generates a set of segmentation results for an image instead of focusing on salient objects, which limits its applicability in unsupervised object localization tasks. Summary of the invention

[0006] In view of the above problems, the present invention provides an unsupervised image segmentation model for drone scenes in a first aspect, comprising a self-attention feature extraction module, a linear segmentation head, a self-hint generator, a SAM segmentation module and an image level fusion module;

[0007] The self-attention feature extraction module is used to extract key features of the original drone image, obtain the self-attention map, and send it to the linear segmentation head and the self-hint generator respectively;

[0008] The linear segmentation head extracts a coarse mask using a lightweight segmentation head to generate a coarse mask;

[0009] The self-hint generator extracts the rough outline of the foreground target from the attention map, takes the circumscribed rectangle of the outline, and thus obtains the box hint for the foreground target of the drone image; grid sampling is designed based on the size of the original drone image, and the center point of each grid is taken to obtain the point hint of the drone image;

[0010] The SAM segmentation module includes an image encoder, a cue encoder and a mask decoder; the original drone image is encoded by the image encoder, the cue is encoded by the cue encoder, and finally two different fine-grained segmentation maps are obtained by the mask decoder, namely, a fine-grained point cue mask based on a large number of point cues, high fineness but not focusing on foreground objects, and a box cue mask based on insufficient fineness of box cues but focusing on foreground objects;

[0011] The image-level fusion module filters the point hint mask through the box hint mask, and filters out the segmentation results that do not contain the foreground object by calculating the intersection-over-union ratio; then the filtered results are fused with the extracted rough mask to obtain a final high-quality segmentation map.

[0012] Preferably, the self-attention feature extraction module uses a DINO model with frozen parameters, and obtains a backbone network model through self-supervised training based on ViT. The visual encoder of the DINO model consists of multiple layers of encoders, each of which has a feedforward network and a multi-head attention mechanism for attention. The self-attention feature map is extracted from the last layer of DINO to obtain the self-attention map.

[0013] Preferably, the step of extracting a rough mask using a lightweight segmentation head is as follows:

[0014] By calculating the similarity between blocks, the block with the lowest attention in the self-attention depth map is identified and used as the background seed. Then, based on the background seed, the foreground target is extracted with the help of the standardized cutting method. The image processing edge smoothing algorithm is used for the foreground target, and the loss is calculated with the image before processing, so as to train a lightweight convolution head to achieve rough feature extraction and generate a rough mask.

[0015] Preferably, in the SAM segmentation module, a single bounding box obtained from the hint generator is input into the hint encoder of SAM, the original image is input into the SAM image encoder, and the image segmentation result is obtained according to the bounding box, which is defined as the bounding box mask Mb; in addition, a point hint grid is generated based on the image size, input into SAM, and low-quality and repeated masks are filtered out. This process produces an overall image segmentation, which is defined as the point mask Mp.

[0016] Preferably, the self-hint generator extracts key features from the last layer of self-attention based on the self-attention map, constructs a scaling factor according to the image block size, extracts the feature CLS of the classification mark, and standardizes the remaining features. The similarity matrix is ​​modified according to the threshold (set to 1e-5), and then the degree of each node needs to be calculated, and the angle matrix D is constructed, the generalized eigensystem (DA) y=λDy is solved, and the eigenvector associated with the second smallest eigenvalue is obtained. The bisection of the graph is calculated by the average value of the eigenvector to obtain the foreground and background partitions, the foreground partition is selected using the maximum absolute value, and the seed point is found. The connected component is selected according to the seed point, and the minimum bounding box of the object is extracted according to the mask of the connected component. Finally, the bounding box is rescaled to the size of the original image according to the scaling ratio, and the box hint is obtained.

[0017] Preferably, the image-level fusion module accepts two inputs, namely, a coarse mask obtained by segmentation with the help of a linear segmentation head, and a single box-based mask M obtained by a SAM segmentation module. b and a set of masks S based on point cues;

[0018] Using M b As a benchmark for screening the set S, by calculating M b and The intersection and union (IoU) between them determines each Whether it corresponds to foreground or background segmentation, it is defined as follows:

[0019]

[0020] Through the background filtering module, the foreground object mask set M is filtered out from the mask set S based on the point prompt s ; Then, the intersection-guided method is used to evaluate the overlap between each mask in the set and the roughly extracted object Mp; the overlap significance T of each mask in the set and Mp is evaluated 1 and prediction accuracy T 2 , to decide whether a single mask in the current evaluation set is used to enhance the rough mask; the calculation formula is as follows:

[0021]

[0022] Where Pij represents the rough prediction result, Sij represents each mask from the Ms set;

[0023] The mask suitable for enhancement is combined with those pixels M that were not segmented by the SAM segmentation module in the prediction n Added to the candidate set C; finally, the n masks in the candidate set are stacked into an n-channel tensor T, where T∈R n×W×H , and then processed by a 1×1 convolution to obtain the output image.

[0024] The second aspect of the present invention provides an unsupervised image segmentation method for a drone scene, comprising the following process:

[0025] Acquire images through drone photography;

[0026] Inputting the image into the unsupervised image segmentation model of the drone scene as described in the first aspect to perform object segmentation;

[0027] Output the target image after segmentation processing.

[0028] The third aspect of the present invention provides an unsupervised image segmentation device for drone scenes, the device comprising at least one processor and at least one memory, the processor and the memory being coupled; the memory stores a computer execution program of the unsupervised image segmentation model for drone scenes as described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes an unsupervised image segmentation method for drone scenes.

[0029] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program or instruction of the unsupervised image segmentation model of the drone scene as described in the first aspect. When the program or instruction is executed by a processor, the processor executes an unsupervised image segmentation method for the drone scene.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Based on the SAM model and the DINO model, the present invention proposes a new unsupervised image segmentation model SelfLoc, which utilizes the SAM refined segmentation capability and the DINO unsupervised feature extraction capability to realize and explore a new drone image segmentation paradigm that does not rely on annotations. This enables the realization of usable drone image segmentation results without supervision. The present invention proposes two core modules, namely the self-prompt generator module and the image-level fusion module, which realize the foreground position of the model self-prompt image and the image-level fusion.

[0032] This model is applied in drone scenarios and fully utilizes its feature of not requiring annotation. This feature greatly reduces the difficulty of annotation caused by the high resolution and wide field of view of drone images, and significantly improves task efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, what is described below is only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 It is the overall structural block diagram of the unsupervised image segmentation model of the present invention.

[0035] Figure 2 The structure diagram of the self-prompt generator module in the embodiment

[0036] Figure 3 Image level fusion module structure diagram

[0037] Figure 4 Schematic diagram of the fusion of fine-grained segmentation mask and coarse mask

[0038] Figure 5 This is a simplified structural diagram of the unsupervised image segmentation device in Example 2. DETAILED DESCRIPTION

[0039] Embodiment 1:

[0040] The invention will be further described below in conjunction with specific embodiments.

[0041] This paper proposes a new unsupervised target localization framework: SelfLoc (SelfLoc:

[0042] High Quality Unsupervised Object Localization with Self-Prompt SAM, a high-quality unsupervised object localization model based on self-prompt. The overall process is as follows Figure 1 As shown, the original drone image of the present invention is subjected to the DINO model with frozen parameters to extract the key features of the image and obtain the self-attention map. The extracted features are processed in two branches, which are used for rough feature extraction and self-prompt generation respectively: for rough feature extraction, the block with the lowest attention in the self-attention depth map is identified by calculating the similarity between the blocks, and this block is used as the background seed. Then, based on the background seed, the foreground target is extracted with the help of the standardized cutting method, and the image processing edge smoothing algorithm is used for the foreground target, and the loss is calculated with the image before processing, so as to train a lightweight convolution head to realize rough feature extraction and generate a rough mask;

[0043] For self-hint generation, we first apply standardized cutting technology to the attention map to extract the rough outline of the foreground target from the attention map, and take the circumscribed rectangle of the outline to obtain the box hint for the foreground target in the drone image; then we design the grid sampling directly based on the original drone image size, take the center point of each grid, and get the point hint of the drone image.

[0044] After obtaining the hint of the image itself, the original drone image is encoded by the image encoder, and the hint is encoded by the hint encoder, and then they are finally passed through the decoder to obtain two different fine-grained segmentation maps, namely, a fine-grained point hint mask based on a large number of point hints, high precision but not focusing on foreground objects, and a box hint mask based on insufficient precision of box hints but focusing on foreground objects.

[0045] The point segmentation map is further filtered through the box-based segmentation map, and the segmentation results that do not contain foreground objects are filtered out by calculating the intersection-over-union ratio. Then, the filtered results are fused with the extracted rough mask through a 1*1 convolution to obtain the final high-quality segmentation map.

[0046] (I) Self-attention feature extraction module

[0047] Given an input image of size H×W, ViT divides the image into image blocks of resolution size K*K, and the number of image blocks per image is HW / K 2 , an additional learnable token, called CLS, can obtain the aggregate information of the image based on this token. The visual encoder of the DINO model consists of multiple layers of encoders, each of which has a feedforward network and a multi-head attention mechanism for attention. The DINO model is a backbone network model obtained through self-supervised training based on ViT. Therefore, for the unsupervised image segmentation task applied in the present invention, the self-attention map is obtained by extracting the self-attention feature map from the last layer of DINO.

[0048] (II) Rough Target Extraction

[0049] By calculating the similarity between blocks, the block with the lowest attention in the self-attention depth map is identified and used as the background seed. Then, based on the background seed, the foreground target is extracted with the help of the standardized cutting method. The image processing edge smoothing algorithm is used for the foreground target, and the loss is calculated with the image before processing, so as to train a lightweight convolution head to achieve rough feature extraction and generate a rough mask.

[0050] After obtaining the attention map, a rough segmentation result can be obtained through a linear segmentation head. Following the idea of ​​the Found model, the idea of ​​first identifying the background and then inverting it to obtain the foreground is adopted. Specifically, for an image, the self-supervised pre-trained DINO model is first used to extract deep features. The least focused blocks in the self-attention depth map are identified, called background seeds. Subsequently, the background is determined by locating blocks similar to the background seeds.

[0051]

[0052] The above equation mainly shows the calculation method of background seed, where A pirepresents the attention score between the CLS token and each patch in the attention head i, argmin represents finding the patch that minimizes the cumulative attention score, and s represents the calculated seed. Then, a coarse mask is extracted using a lightweight segmentation head consisting of a 1x1 convolution. Subsequently, the predicted mask obtained from the segmentation head is refined using the background mask obtained based on DINO and by applying a bilateral solver. This refined mask will be used as a pseudo-label for training the segmentation head. The resulting coarse mask is input to the image fusion module.

[0053] 3. Self-prompt generator

[0054] The self-hint generator aims to generate sparse hints for objects in an image without human intervention. It mainly consists of two parts: bounding box hint generation and point hint generation. The structure is as follows Figure 2 Specifically, the last layer of self-attention map generated by DINO is obtained, and key features are extracted from the attention map of the image. Standardized cut can divide the image into two disjoint sets by minimizing the standardized energy. The definition is as follows:

[0055]

[0056] Among them, Ncut(A,B) represents the standardized cut value, which is used to evaluate the index of partitioning the graph into subsets A and B, cut(A,B) represents the sum of the edge weights between set A and set B, and assoc(A,V) represents the cumulative value of the edge weights connecting the nodes in set A with all nodes in the entire graph V.

[0057] The foreground salient object in the image has a low correlation with the entire image. In order to obtain the foreground salient target, the present invention selects the largest connected component as the final foreground object area. Therefore, by applying standardized cutting to the extracted features, the foreground target area in the image is obtained, and the circumscribed rectangle is extracted from the foreground area, that is, the frame prompt for the foreground object is obtained from the image itself in an unsupervised manner. Specifically, on the basis of the attention map obtained by DINO, the key feature is extracted from the last layer of self-attention. The scaling factor is constructed according to the image block size, the feature CLS of the classification mark is extracted, and the remaining features are standardized. The similarity matrix is ​​modified according to the threshold (set to 1e-5). Then, it is necessary to calculate the degree of each node and construct the angle matrix D. Solve the generalized eigensystem (DA) y=λDy to obtain the eigenvector associated with the second smallest eigenvalue. The bisection of the graph is calculated by the average value of the eigenvector to obtain the foreground and background partitions. The foreground partition is selected using the maximum absolute value and the seed point is found. The connected component is selected according to the seed point, and the minimum bounding box of the object is extracted according to the mask mask of the connected component. Finally, the bounding box is rescaled to the size of the original image according to the scaling ratio, and the frame prompt is obtained.

[0058] The self-hint generator aims to generate sparse hints for objects in an image without human intervention. It mainly consists of two parts: bounding box hint generation and point hint generation. The structure is as follows Figure 2 As shown. Specifically, the last layer of self-attention map generated by DINO is obtained, and key features are extracted from the attention map of the image. Normalized cutting can divide the image into two non-overlapping sets by minimizing the normalized energy. The foreground salient object in the image has a low correlation with the entire image. In order to obtain the foreground salient target, the present invention selects the largest connected component as the final foreground object area. Therefore, by applying normalized cutting to the extracted features, the foreground target area in the image is obtained, and the circumscribed rectangle of the foreground area is extracted, that is, the box prompt for the foreground object is obtained from the image itself in an unsupervised manner.

[0059] In addition, in order to provide a more refined image segmentation result and supplement the foreground target area that is not segmented in the rough target extraction process, the present invention follows the grid prompt method designed by SAM, performs uniform grid sampling on the input image, obtains a set of point prompts, designs grid sampling based on the size of the original drone image, takes the center point of each grid, and obtains the point prompt of the drone image. Therefore, point prompts that can provide refined segmentation and box prompts that assist in locating foreground targets and screening SAM segmentation results are obtained. The two prompts are respectively sent to the prompt encoder of the SAM segmentation module to provide location information embedding.

[0060] (IV) SAM segmentation module

[0061] In order to further improve the quality of segmentation, the SAM model is introduced for refined segmentation. The SAM architecture consists of an image encoder, a hint encoder, and a mask decoder. The heavyweight image encoder based on Vision Transformer converts the input image into an embedded representation, while the hint encoder converts various types of hints (including points, boxes, masks, etc.) into compact embedded representations. Subsequently, these embeddings from the two parts are integrated through the mask decoder to generate accurate segmentation results. SAM is a hint-driven visual large model, and the granularity of its segmentation depends on the accuracy of the hints provided, which usually requires manual supervision. To utilize the capabilities of SAM in an unsupervised manner, the model must be able to autonomously generate hints to guide the segmentation process. Therefore, with the help of the point hints and box hints generated in an unsupervised manner by the self-hint generation module, the obtained single box hint is input into the hint encoder of SAM, and the original image is input into the SAM image encoder. Finally, the image segmentation result is obtained based on the box hint, which is defined as the bounding box mask M b In addition, to provide more segmentation information, the point hint is fed into the SAM segmentation module and low-quality and duplicate masks are filtered out. This process produces an overall image segmentation, defined as the point mask M p , typically including tens to hundreds of mask results.

[0062] (V) Image-level fusion module

[0063] The image-level fusion module accepts two inputs, namely, the rough mask obtained by segmentation with the help of the linear segmentation head, and the single bounding box hint and point hint generated by the self-hint generator, which are input into the SAM segmentation module together with the original image, and the resulting single box-based mask (denoted as M b ) and a set of masks based on point cues (denoted as S). The segmentation results produced by SAM from self-generated point cues sometimes include not only foreground object masks but also background masks, which may interfere with the integration of the final results. In contrast, the segmentation obtained from the predicted bounding boxes as sparse cues tends to focus on the objects in the image. Therefore, using M b As a benchmark for screening the set S, by calculating M b and The intersection and union (IoU) between them is used to determine each Whether it corresponds to foreground or background segmentation, it is defined as follows:

[0064]

[0065] Therefore, through the background filtering module, the foreground object mask set M is filtered out from the mask set S based on the point prompt sInspired by WSSSAM, we then use an intersection-guided approach to evaluate each mask in the set against the roughly extracted object M p This involves evaluating the overlap between each mask in the set and M p The overlap significance T 1 and prediction accuracy T 2 , to decide whether a single mask in the current evaluation set can be used to enhance the rough mask. The calculation formula is as follows, where P ij Represents the rough prediction result, S ij Indicates that it comes from M s Each mask of the collection.

[0066]

[0067]

[0068] The mask suitable for enhancement is combined with those pixels M that were not segmented by SAM in the prediction n Add to the candidate set C. Finally, the n masks in the candidate set are stacked into an n-channel tensor T, where T∈R n×W×H , and then processed by a 1×1 convolution to obtain the output mask. The structure is as follows Figure 3 As shown in Figure 2, the fusion process of fine-grained segmentation mask and coarse mask is as follows: Figure 4 shown.

[0069] (VI) Experimental results prove

[0070] Table 1 Inference results on three salient object detection datasets

[0071]

[0072] Method: method;

[0073] ECSSD, DUTS-TE, and DUT-OMRON are the names of data sets;

[0074] mIoU: mean intersection over union;

[0075] Acc: accuracy;

[0076] F β max : A comprehensive evaluation index based on precision and recall;

[0077] As shown in Table 1, this embodiment tests the inference accuracy of the model of the present invention on three salient object detection datasets. Compared with the best previous models, the model of the present invention is superior in standard mask mIoU, accuracy Acc, β index F β maxThe three indicators have achieved industry-leading results, especially mIoU has achieved significant improvements.

[0078] Table 2 Reasoning results on three target discovery datasets

[0079] method VOC07 dataset VOC12 dataset COCO20k dataset rOSD 54.5 55.3 48.5 LOD 53.6 55.1 48.5 DINO-seg 45.8 46.2 42.0 LOST 61.9 64.0 50.7 DSS 62.7 66.4 52.2 FreeSolo 44.0 49.7 35.2 TokenCut 72.3 75.3 62.7 SelfMask 72.3 75.3 62.7 DINOSAUR - 70.4 67.2 WSCUOD 70.6 72.1 63.5 Found <![CDATA[ 72.5 ]]> <![CDATA[ 76.1 ]]> 62.9 Segmentation method of the present invention 74.3 77.7 <![CDATA[ 64.5 ]]>

[0080] Method: method;

[0081] VOC07, VOC12, and COCO20k are all names of datasets;

[0082] Table 2 shows the inference results of the SelfLoc model of the present invention on three object discovery datasets. Evaluation is performed on three popular object discovery datasets: PASCAL VOC07, PASCAL VOC2012, and COCO20k datasets. COCO20k is a subset of 19,817 images randomly selected from the COCO2014 dataset. Results using the correct localization (CorLoc) metric are reported. CorLoc measures the percentage of correct boxes, i.e., the intersection-over-union ratio of the predicted box with a true box is greater than 0.5. Compared with current state-of-the-art methods, such as WSCUOD and Found, SelfLoc outperforms all other methods on both VOC07 and VOC12 datasets. Although DINOSAUR achieves higher performance on the COCO20K dataset, the method of the present invention takes less training time and GPU usage.

[0083] Table 3 Reasoning results on three target discovery datasets

[0084]

[0085] Experiments are conducted on four commonly used camouflage object test datasets: CAMO, COD10k, NC4k, and CHAMELEON, which contain 250 images, 2026 images, 4121 images, and 76 images, respectively. Four well-known evaluation indicators are used, including S-measure (S α ); weighted F-measure That is, the weighted F measure; mIoU, that is, the mean intersection over union ratio; and mean absolute error (M), that is, the mean absolute error. For the sake of fairness, all the experimental data provided above are from the official data provided by each model.

[0086] As shown in Table 3, our method achieves state-of-the-art results on all four datasets, surpassing all previous methods. Specifically, compared with the second-best UCOS-DA, our method improves by 3.68% on average IoU, 1.48% on maximum F-measure, 2.08% on weighted F-measure, and 0.88% on average on structure metric. The outstanding performance of SelfLoc in saliency segmentation and camouflaged object segmentation also demonstrates the effectiveness and robustness of our method.

[0087] Embodiment 2:

[0088] like Figure 5 As shown, the present invention also provides an unsupervised image segmentation device for drone scenes, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; the memory stores a computer execution program of the unsupervised image segmentation model of the drone scene as described in Example 1; when the processor executes the computer execution program stored in the memory, the processor can execute an unsupervised image segmentation method for drone scenes. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0089] The device may be provided as a terminal, a server or other forms of devices.

[0090] Figure 5is a block diagram of a device shown as an example. The device may include one or more of the following components: a processing component, a memory, a power component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component generally controls the overall operation of the electronic device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component may include one or more processors to execute instructions to complete all or part of the steps of the above method. In addition, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0091] The memory is configured to store various types of data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0092] The power supply assembly provides power to various components of the electronic device. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device. The multimedia assembly includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia assembly includes a front camera and / or a rear camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0093] The audio component is configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. The I / O interface provides an interface between the processing component and the peripheral interface module, and the above-mentioned peripheral interface module can be a keyboard, a click wheel, a button, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

[0094] The sensor assembly includes one or more sensors for providing various aspects of status assessment for the electronic device. For example, the sensor assembly can detect the on / off state of the electronic device, the relative positioning of components, such as the display and keypad of the electronic device, and the sensor assembly can also detect the position change of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device and the temperature change of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0095] The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0096] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0097] Embodiment 3:

[0098] The present invention also provides a computer-readable storage medium, which stores a computer program or instruction of an unsupervised image segmentation model for a drone scene as described in Example 1. When the program or instruction is executed by a processor, the processor can execute an unsupervised image segmentation method for a drone scene.

[0099] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0100] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk (such as CD-ROM, CD-R, CD-RW, DVD-20ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tape, etc. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0101] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0102] It should be understood that the storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a terminal or server.

[0103] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0104] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0105] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0106] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An unsupervised image segmentation model for drone scenes, characterized by: It includes self-attention feature extraction module, linear segmentation head, self-hint generator, SAM segmentation module and image-level fusion module; The self-attention feature extraction module is used to extract key features of the original drone image, obtain the self-attention map, and send it to the linear segmentation head and the self-hint generator respectively; The linear segmentation head extracts a coarse mask using a lightweight segmentation head to generate a coarse mask; The self-hint generator extracts the rough outline of the foreground target from the attention map, takes the circumscribed rectangle of the outline, and thus obtains the box hint for the foreground target of the drone image; Design grid sampling based on the original drone image size, take the center point of each grid, and get the point prompt of the drone image; The SAM segmentation module includes an image encoder, a cue encoder and a mask decoder; the original drone image is encoded by the image encoder, the cue is encoded by the cue encoder, and finally two different fine-grained segmentation maps are obtained by the mask decoder, namely, a fine-grained point cue mask based on a large number of point cues, high fineness but not focusing on foreground objects, and a box cue mask based on insufficient fineness of box cues but focusing on foreground objects; The image level fusion module filters the point hint mask through the box hint mask, and filters out the segmentation results that do not contain the foreground object by calculating the intersection-over-union ratio; then the filtered results are fused with the extracted rough mask to obtain a final high-quality segmentation map; The image-level fusion module accepts two inputs, namely the coarse mask obtained by segmentation with the help of the linear segmentation head and the single box-based mask obtained by the SAM segmentation module. M b and a collection of masks based on point hints S； use M b As a filter set S The benchmark is calculated by M b and The intersection and union IoU between S Each of Whether it corresponds to foreground or background segmentation, it is defined as follows: Through the background filtering module, from the point-based mask set S Filter out the foreground object mask set M s ; Then, using the intersection-guided method, each mask in the set is evaluated against the roughly extracted object. Mp The degree of overlap between ; evaluate each mask in the set and Mp The overlap significance and prediction accuracy , to decide whether a single mask in the current evaluation set is used to enhance the coarse mask; The calculation formula is as follows: in Pij Represents a rough prediction result, Sij Indicates from M Each mask of the set s; The mask suitable for enhancement is combined with those pixels that were not segmented in the prediction by the SAM segmentation module. M n Add to Candidate Set C In the final candidate set n The masks are stacked into one n Channel Tensor T ,in T ∈R n×W×H , and then processed by a 1×1 convolution to obtain the output image.

2. The unsupervised image segmentation model for drone scenes according to claim 1, characterized in that: The self-attention feature extraction module uses the DINO model with frozen parameters. Based on ViT, the backbone network model is obtained through self-supervised training. The visual encoder of the DINO model consists of multiple layers of encoders, each of which has a feedforward network and a multi-head attention mechanism for attention. The self-attention feature map is extracted from the last layer of DINO to obtain the self-attention map.

3. The unsupervised image segmentation model for drone scenes according to claim 1, characterized in that: The method of extracting a rough mask using a lightweight segmentation head is as follows: By calculating the similarity between blocks, the block with the lowest attention in the self-attention depth map is identified and used as the background seed. Then, based on the background seed, the foreground target is extracted with the help of the standardized cutting method. The image processing edge smoothing algorithm is used for the foreground target, and the loss is calculated with the image before processing, so as to train a lightweight convolution head to achieve rough feature extraction and generate a rough mask.

4. The unsupervised image segmentation model for drone scenes according to claim 1, characterized in that: In the SAM segmentation module, a single bounding box obtained from the hint generator is input into the hint encoder of SAM, and the original image is input into the SAM image encoder, and the image segmentation result is obtained according to the bounding box, which is defined as the bounding box mask Mb; In addition, a point cue grid is generated based on the image size and input into SAM, and low-quality and duplicate masks are filtered out. This process produces an overall image segmentation, defined as the point mask Mp.

5. The unsupervised image segmentation model for drone scenes according to claim 1, characterized in that: The self-hint generator extracts key features from the last layer of self-attention on the basis of obtaining the self-attention map, constructs a scaling factor according to the image block size, extracts the feature CLS of the classification mark, and standardizes the remaining features; according to the threshold, the threshold is set to 1e-5, the similarity matrix is ​​modified, and then, the degree of each node needs to be calculated, and the angle matrix D is constructed, the generalized eigensystem (D - A)y = λDy is solved, and the eigenvector associated with the second smallest eigenvalue is obtained. The bisection of the graph is calculated by the average value of the eigenvector to obtain the foreground and background partitions, the foreground partition is selected using the maximum absolute value, and the seed point is found, the connected component is selected according to the seed point, and the minimum bounding box of the object is extracted according to the mask of the connected component. Finally, the bounding box is rescaled to the size of the original image according to the scaling ratio, and the box hint is obtained.

6. An unsupervised image segmentation method for drone scenes, characterized in that: The process includes: Acquire images through drone photography; Inputting the image into the unsupervised image segmentation model of the drone scene as described in any one of claims 1 to 5 to perform object segmentation; Output the target image after segmentation processing.

7. An unsupervised image segmentation device for drone scenes, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the unsupervised image segmentation model of the drone scene as described in any one of claims 1 to 5; when the processor executes the computer execution program stored in the memory, the processor executes an unsupervised image segmentation method for drone scenes.

8. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program or instruction of an unsupervised image segmentation model for a drone scene as described in any one of claims 1 to 5, wherein when the program or instruction is executed by a processor, the processor executes an unsupervised image segmentation method for a drone scene.

Images