A semantic guidance fuzzy point segmentation method for cross-view command of UAVs
By combining ambiguous points and semantic cues in UAV cross-view target guidance, and utilizing a conditionally deformable attention mechanism for feature-level offset correction, the problem of inaccurate target segmentation caused by cue point offset is solved, thereby improving the target segmentation accuracy and robustness of UAVs in complex scenarios.
Patent Information
- Application Number
- CN202611213696.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-11-10
AI Technical Summary
During the cross-view target guidance process of UAVs, the offset of the cue point position makes it difficult for existing interactive segmentation methods to accurately segment the target, especially in complex scenarios, which can easily lead to target ambiguity. Existing methods are unable to effectively correct the systematic offset of the cue point.
A semantically guided fuzzy point segmentation method for cross-view command of UAVs is adopted. By acquiring fuzzy point cues and semantic cues, feature-level offset correction is performed using a conditionally deformable attention mechanism. Combined with target semantic information and spatial location information, an accurate target segmentation mask is generated.
It improves the robustness and accuracy of target segmentation for UAVs in complex scenarios, enhances the model's ability to distinguish between target boundaries and background, is applicable to various human-computer interaction scenarios, and has good engineering application value.
Smart Images

Figure CN123071644A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a semantic guidance fuzzy point segmentation method for cross-view command of unmanned aerial vehicles. Background Technology
[0002] With the development of drone platforms, airborne visual perception equipment, and human-machine interaction technologies, drones have been gradually applied to scenarios such as road patrol, illegal parking evidence collection, security inspection, and emergency search and rescue. In these applications, operators typically need to quickly and accurately designate target objects to the drone so that it can subsequently perform tasks such as target confirmation, positioning, tracking, observation pose planning, or image acquisition. Compared to complex target numbering, coordinate input, or fully automatic recognition methods, the interactive method of using point selection combined with verbal description has the advantages of intuitive operation, low input cost, and clear target direction, making it suitable for drone-human collaborative target guidance.
[0003] Existing interactive object segmentation methods typically utilize point cues, bounding box cues, or mask cues to guide the model in segmenting objects within an image. Among these, point cues, being a representative interactive method, are simple to operate and have been widely applied to interactive segmentation tasks based on visual models. Existing methods generally assume that the user-inputted cues fall accurately within the target object and extract target-related features based on the location of the cues to generate a corresponding object segmentation mask. When the cues are accurately positioned, this type of method can achieve good segmentation results.
[0004] However, in drone applications, it is often difficult to maintain high positional accuracy for the designated points. On one hand, when operators directly select points within the drone's onboard view, factors such as image jitter caused by drone flight, small targets at long distances, network transmission latency, touchscreen errors, and human error can all cause the selected point to deviate from the actual target area. On the other hand, in ground-to-air collaboration, augmented reality-assisted command, or multi-drone collaboration scenarios, operators may specify targets in ground cameras, augmented reality devices, fixed monitoring equipment, other drone images, or historical images. In this case, it is necessary to map the target position from the source viewpoint to the target drone's current viewpoint. Since there are usually significant differences in scale, attitude, occlusion, and depth between different viewpoints, and the cross-viewpoint mapping process can be affected by factors such as camera calibration errors, depth estimation errors, parallax, attitude drift, target motion, time asynchrony, and sensor noise, the mapped designated point may deviate from the actual target position.
[0005] The aforementioned deviations do not necessarily manifest as simple random perturbations, but may possess a certain directionality and systematic nature. After shifting, the cue point may fall near the target boundary, or it may fall into the background area or the area of adjacent interfering targets. When vehicles are densely parked, road facilities are repeatedly distributed, or there are multiple similar targets in the scene, relying solely on the geometric position of the cue point can easily lead to target ambiguity.
[0006] Existing interactive segmentation methods based on point cues are typically sensitive to the location of the cues and lack explicit correction mechanisms for systematic shifts in the cues. When a cue deviates from the actual target and falls into the region of other objects, the model tends to prioritize responding to the location of the cue, thus segmenting incorrect objects. For example, in a drone-based scenario of inspecting illegally parked vehicles, the operator may actually want to select a specific red target vehicle, but the point cues mapped across viewpoints might fall into the region of adjacent vehicles. In this case, segmenting based solely on the point location can easily lead to the identification of adjacent vehicles as the target to be segmented.
[0007] To reduce the impact of point position errors on segmentation results, some existing methods enhance the model's adaptability to noise cues by applying random perturbations to cue points, expanding the search range, or using local windows. However, these methods typically deal primarily with independent random noise and struggle to fully reflect the structured positional deviations caused by factors such as depth, calibration, parallax, and attitude during UAV cross-view target guidance. Furthermore, when using fixed local windows for feature search, the receptive range is usually predetermined, making it difficult to adapt to cue drift caused by different flight altitudes, different viewpoints, and different mapping error magnitudes. While global feature interaction can expand the search range, it is easily affected by irrelevant areas in scenes containing many similar vehicles, road facilities, or background structures, and it also increases computational overhead.
[0008] On the other hand, textual descriptions or visual reference examples can provide semantic information such as target category, color, appearance attributes, texture features, and spatial relationships. For example, an operator can use textual descriptions such as "the red vehicle on the right side of the road" to describe the target to be segmented, or can select the target object in the source view image as a visual reference. Existing semantic segmentation or indexing expression segmentation methods can utilize the above semantic information for target localization, but if they rely entirely on semantic information to search the entire image, ambiguity can easily arise in dense scenes with multiple similar-looking or semantically identical targets. Therefore, coarse spatial location and target semantic information are complementary: spatial location can limit candidate regions, while semantic information can further distinguish between real targets and adjacent interfering targets within the candidate regions.
[0009] Therefore, how to effectively correct the prompt position and accurately obtain the target segmentation region corresponding to the operator's true intention, even when the prompt point is offset or falls into the background or adjacent target area, by simultaneously utilizing the approximate spatial position information provided by the prompt point and the target semantic information provided by the text description or visual reference, remains a technical problem that needs to be solved in the field of UAV human-machine collaborative target guidance and interactive target segmentation. Summary of the Invention
[0010] The present invention aims to at least partially solve one of the technical problems in the related art.
[0011] The main objective of this invention is to provide a semantic guidance ambiguity point segmentation method for cross-view command of unmanned aerial vehicles (UAVs).
[0012] To achieve the above objectives, embodiments of the present invention propose a semantic guidance ambiguity point segmentation method for cross-view command of unmanned aerial vehicles, comprising: Step S1: Obtain the target view image, a blurry dot cue indicating the approximate location of the target to be segmented, and a semantic cue describing the target to be segmented; Step S2: Extract features from the target view image to obtain image features; encode the location of the blurred point prompt to obtain point prompt features; encode the semantic prompt to obtain target semantic features; Step S3: Fuse the target semantic features with the point prompt features to obtain a hybrid prompt feature that simultaneously contains target semantic information and spatial location information; Step S4: Using the position corresponding to the blurred point prompt as a reference position, determine the sampling offset relative to the reference position based on the hybrid prompt features, and extract local image features from the image features according to the sampling offset, perform feature-level offset correction on the blurred point prompt, and obtain the corrected prompt features; Step S5: Perform bidirectional interactive fusion of the correction prompt features and the image features to obtain fused features; Step S6: Generate a segmentation mask for the target to be segmented that corresponds to the semantic prompt based on the fusion features.
[0013] Optionally, the fuzzy point hints include at least one hint point for indicating the approximate location of the target to be segmented; The blur point hints are obtained by the user directly selecting them in the target view image, or by obtaining the target position in the source view through cross-view mapping; the blur point hints are allowed to deviate from the actual position of the target to be segmented and fall into the target boundary, background area or adjacent interfering target area.
[0014] Optionally, the semantic prompts include at least one of textual semantic prompts and visual semantic prompts; wherein, the textual semantic prompts include at least one of target category, color, appearance attribute and spatial relationship, and the visual semantic prompts include a source viewpoint reference image and a corresponding target reference region.
[0015] Optionally, the semantic encoding of the semantic prompt in step S2 includes: When the semantic prompt is a text semantic prompt, the text description is encoded to obtain text semantic features; When the semantic cues are visual semantic cues, feature extraction is performed on the source view reference image, reference features corresponding to the target region are extracted based on the reference region, and the reference features are aggregated to obtain visual semantic features; The textual semantic features or the visual semantic features are used as the target semantic features.
[0016] Optionally, step S3 includes: The target semantic features are concatenated with the point cue features; By using a self-attention mechanism to interact with the concatenated features, a correlation between target semantic information and spatial location information is established, thereby obtaining the hybrid prompt features.
[0017] Optionally, step S4 employs conditional deformable attention for feature-level offset correction, including: Using the hybrid prompt features as query information and the location corresponding to the fuzzy point prompt as a reference location, multiple sampling offsets and corresponding attention weights relative to the reference location are predicted based on the hybrid prompt features. Based on the multiple sampling offsets, multiple sampling positions are determined in the image features, and local image features corresponding to the sampling positions are obtained; The local image features are weighted and aggregated according to the attention weights to obtain the correction prompt features.
[0018] Optionally, the image features include image features at multiple scales; In step S4, feature sampling is performed on the image features at each scale according to the sampling offset, and the local image features obtained at each scale are fused to obtain the correction prompt features.
[0019] Optionally, step S5 includes: Using the correction prompt features as query information and the image features as keys and values, the interaction between the prompt and the image is performed. The image features are used as query information, and the correction prompt features are used as keys and values to perform image-to-prompt interaction; The fusion feature is obtained based on the results of the two interactions.
[0020] Optionally, the target view image is an image captured by an airborne camera of a UAV; The blurred dot cues are derived from the target location in airborne footage, ground terminals, augmented reality devices, ground cameras, other drones, or historical images; The segmentation mask is used for subsequent target confirmation, target location, target tracking, or mission planning.
[0021] The embodiments of the present invention have the following beneficial effects: (1) In view of the problem that prompts are prone to positional deviation during the human-machine collaborative target guidance process of UAVs, the present invention combines fuzzy point prompts and semantic prompts for joint modeling. Based on the fuzzy points providing the approximate spatial position of the target, the semantic information of the target is introduced to constrain the prompt position, so that the model can still accurately determine the target to be segmented when there is a deviation in the prompt point, thereby improving the robustness of target segmentation in complex scenarios.
[0022] (2) This invention employs a conditionally deformable attention mechanism, using the location corresponding to the blurred point prompt as a reference location. It adaptively predicts the sampling offset based on the target semantic information and aggregates image features consistent with the target semantics within the target neighborhood, thus achieving feature-level offset correction for the blurred point prompt. Compared to methods relying on fixed windows or global search, this approach can more accurately recover the target actually pointed to by the operator, improving target segmentation accuracy.
[0023] (3) This invention constructs a hybrid cue feature of target semantic features and point cue features, and combines the bidirectional interactive fusion of correction cue features and image features to fully coordinate target semantic information, spatial location information and image visual information, thereby improving the target feature expression ability and enhancing the model's ability to distinguish target boundaries, target appearance and complex backgrounds.
[0024] (4) This invention supports both text semantic prompts and visual semantic prompts for target description. It can describe the target using text information such as target category, color, appearance attributes and spatial relationship, and can also provide visual examples of the target using source view reference images. It is applicable to different types of human-computer interaction scenarios, improving the applicability and interaction flexibility of the method.
[0025] (5) This invention does not require changes to the existing UAV flight control and safety decision-making module. It can be deployed as an independent target segmentation module on the UAV airborne computing platform or ground computing platform to provide target segmentation results for subsequent target confirmation, target positioning, target tracking and mission planning. It has good engineering application value and system compatibility. Attached Figure Description
[0026] The above-described and additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a semantic guidance ambiguity point segmentation method for cross-view command of unmanned aerial vehicles provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the semantic guidance adapter provided in an embodiment of the present invention; Figure 3 This is a framework diagram of a semantic guidance ambiguity point segmentation method for cross-view command of UAVs in a certain scenario, provided by an embodiment of the present invention. Detailed Implementation
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] This embodiment provides a semantic guidance ambiguity point segmentation method for cross-view command of unmanned aerial vehicles (UAVs). For example... Figure 1 As shown, the method includes the following steps: Step S1: Obtain the target view image, fuzzy dot hints indicating the approximate location of the target to be segmented, and semantic hints describing the target to be segmented.
[0030] In this embodiment, step S1 is first executed to obtain the target view image, fuzzy dot hints indicating the approximate location of the target to be segmented, and semantic hints describing the target to be segmented, providing basic input data for subsequent image feature extraction, dot hint encoding, semantic encoding, and target segmentation processing.
[0031] Specifically, this embodiment defines the semantic guidance ambiguity point segmentation task for cross-view command of UAVs as follows: given a target view image I, a spatial location cue set P, and semantic cues S, predict the segmentation mask corresponding to the target object using the joint information of the three. The target view image I can be represented as... H represents the image height, W represents the image width, and 3 represents the number of image color channels. In this embodiment, the target view image is preferably a current frame image captured in real time by the UAV's onboard camera, used to reflect the target object and its surrounding environment information in the UAV's current observation scene. Those skilled in the art will understand that in other embodiments, the target view image may also originate from historical frames in a video sequence captured by the UAV, offline acquired images, or other image data that can reflect the target scene, all of which fall within the protection scope of this invention.
[0032] After acquiring the target view image, further fuzzy point cues are obtained to represent the approximate location of the target to be segmented. Each fuzzy point cue consists of at least one cue point, and multiple cue points can form a spatial location cue set. ,in, Let N represent the coordinates of the i-th prompt point, and N represent the number of prompt points.
[0033] In this embodiment, the blurry point prompt can come from multiple sources. In one implementation, the operator can directly click on the target to be segmented in the UAV's onboard view, and the system generates a corresponding prompt point based on the click location. In another implementation, the operator can select the target location from a ground terminal, augmented reality device, ground camera, other UAVs, or historical images, and then obtain the corresponding location in the target's view image through cross-view mapping to generate the blurry point prompt. Cross-view mapping can be achieved by combining camera intrinsic and extrinsic parameters, depth estimation results, target tracking results, feature matching relationships, or other view transformation relationships; this embodiment does not specifically limit this.
[0034] Unlike traditional interactive target segmentation methods that assume the cue points are precisely located inside the target, the cue points in this embodiment are allowed to have a certain degree of positional deviation, and are therefore uniformly defined as fuzzy point cueing. Specifically, the fuzzy point cueing can deviate from the actual position of the target to be segmented; it can be located inside the target region, near the target boundary, or fall into the background region or adjacent interfering target region. The positional deviation can originate from factors such as depth estimation deviation, camera calibration error, augmented reality display parallax, device sensor offset, attitude drift, network communication latency, target motion, and cross-viewpoint mapping error. Therefore, the fuzzy point cueing in this embodiment is only used to provide an approximate spatial position of the target to be segmented, and is not the sole basis for the target's true position, providing initial spatial constraints for subsequent steps to correct the cue position using target semantic information.
[0035] After obtaining fuzzy point cues, semantic cues are further obtained to describe the target to be segmented. These semantic cues characterize the target object's category information, appearance features, or spatial relationships, providing semantic constraints for subsequent target semantic modeling and cue position correction. Specifically, the semantic cues include at least one of textual semantic cues and visual semantic cues.
[0036] When textual semantic prompts are used, the semantic prompts use natural language to describe the semantic attributes of the target object, including at least one of the following: target category, color, appearance attributes, and spatial relationships. For example, it can be described as "the red vehicle on the right side of the road," "the white vehicle in front of the shop," or "the vehicle parked next to the fire lane." Those skilled in the art will understand that any natural language content that can describe the semantic information of the target object can be used as textual semantic prompts in this embodiment.
[0037] When visual semantic cues are used, the semantic cues include the source viewpoint reference image. And the corresponding target reference region. The source viewpoint reference image can originate from various observation devices such as ground terminals, augmented reality devices, other drones, or historical images; the target reference region is used to identify the target object region in the source viewpoint image, and it can be represented by a reference mask. The target region can be represented by a bounding box or other means that can characterize the target area. In this embodiment, a reference mask is preferably used as the target reference region to facilitate the accurate extraction of the visual semantic features corresponding to the target object.
[0038] After acquiring the target view image I, the blurry point cue P, and the semantic cue S, these three types of data are used together as input to the semantically guided blurry point target segmentation model. A mapping relationship is established between the target image, the blurry point cue, and the semantic cue, and a target segmentation mask consistent with the semantic cue is predicted. The mapping relationship is as follows:
[0039] in, This represents the predicted target segmentation mask. This represents a semantic guidance fuzzy point segmentation mapping function for cross-view command of UAVs. The learnable parameters of the model are represented. Through the above mapping relationship, visual information from the target image, spatial location information provided by blurry point cues, and target semantic information provided by semantic cues are jointly modeled. This enables the model to use semantic information to correct blurry point cues with positional deviations and predict a target segmentation mask consistent with the semantics. It should be noted that the target segmentation mask obtained in this embodiment can be used as input for subsequent modules such as target confirmation, target localization, target tracking, or task planning, but the aforementioned downstream processing is not part of the target segmentation process defined in this invention.
[0040] Through the above steps, the target view image, blurry point cues, and semantic cues were acquired, and the input data and output format of the target segmentation task were established, providing a unified data input for feature extraction of the target view image, positional encoding of blurry point cues, and semantic encoding of semantic cues in step S2.
[0041] Step S2: Extract features from the target view image to obtain image features; encode the location of the blurred point prompt to obtain point prompt features; encode the semantic prompt to obtain target semantic features.
[0042] In this embodiment, after obtaining the target view image, blurry dot hints, and semantic hints in step S1, step S2 is further executed to perform feature encoding on the target view image, blurry dot hints, and semantic hints respectively, so as to obtain the image features, dot hint features, and target semantic features required for subsequent semantic guidance offset correction.
[0043] Specifically, feature extraction is first performed on the target view image to obtain the image features corresponding to the target view image. The target view image is the image acquired by the UAV's onboard camera in step S1, which contains information about the target to be segmented and its surrounding environment. In this embodiment, an image encoder is used to extract features from the target view image to obtain multi-scale image features, which are represented as follows: Where F represents the multi-scale image feature set corresponding to the target viewpoint image, Let L represent the image feature map at the scale corresponding to the l-th layer, where L represents the number of image feature layers. By extracting image features at different scales, the model can simultaneously acquire local detail information of the target region and high-level semantic information of the scene in which the target is located, providing an image feature foundation for subsequent target region localization and blur point offset correction based on semantic cues.
[0044] Furthermore, the fuzzy point hints obtained in step S1 are positionally encoded to obtain point hint features. Specifically, the fuzzy point hints are used to provide an approximate spatial location of the target to be segmented. Since the fuzzy point hints may have positional offsets, in this embodiment they are not directly used as the true location of the target, but are converted into point hint features with spatial positional information through positional encoding. Through the above processing, the two-dimensional spatial coordinate information in the fuzzy point hints is converted into a feature representation that can participate in model feature interaction, thereby providing spatial constraints for subsequent hint position correction by combining target semantic information.
[0045] Furthermore, the semantic prompts obtained in step S1 are semantically encoded to obtain target semantic features. Depending on the input format of the semantic prompts, this embodiment employs different encoding methods to extract a unified feature representation that can characterize the semantic information of the target to be segmented.
[0046] When the semantic prompt is a text-based semantic prompt, the user-inputted text description information is first obtained, and then a text encoder is used to encode the text description information to obtain text semantic features. Specifically, the text description information may include at least one of the following: target category, color, appearance attributes, and spatial relationship, such as "blue target vehicle on the right side of the road, in front of the shop". By encoding the text description, the target description in natural language form can be converted into a semantic feature representation that the model can process.
[0047] In this embodiment, the text encoder can be implemented using a pre-trained text encoding model, such as the text encoding branch in a perceptual encoder or visual language model; this embodiment does not specifically limit this. After text encoding, the text semantic embedding is obtained: ,in, C represents the semantic features of the text, and C represents the semantic feature dimension. The text semantic features are used to describe high-level semantic information such as the category, appearance, and spatial relationships of the target to be segmented.
[0048] When the semantic cues are visual semantic cues, the visual semantic cues include the source viewpoint reference image obtained in step S1 and the corresponding target reference region. In this embodiment, the target reference region is preferably represented using a reference mask.
[0049] Specifically, firstly, an image encoder is used to extract features from the source viewpoint reference image to obtain a reference feature map corresponding to the reference image: ,in, The reference image represents the visual features corresponding to the source viewpoint. Then, based on the reference mask corresponding to the target reference region, the foreground features corresponding to the target region are determined from the reference feature map.
[0050] Furthermore, to obtain a visual semantic representation that accurately characterizes the appearance information of the target to be segmented, this embodiment selects multiple feature locations from the target foreground region corresponding to the reference mask, and aggregates the feature vectors corresponding to these locations to obtain visual semantic features. Specifically, the visual semantic features are represented as follows:
[0051] in, Representing visual semantic features, Indicates the coordinate position in the reference feature map The feature vector at that location, This represents the target reference region mask in the source view reference image. This represents the set of feature locations selected from the target foreground region based on the reference mask, and Pool represents the feature aggregation operation.
[0052] Through the above processing, visual semantic information corresponding to the target to be segmented can be extracted from the source view reference image, reducing the influence of the background region on the target feature representation, so that the obtained visual semantic features can reflect the appearance, texture and structural information of the target object itself.
[0053] Furthermore, when the semantic prompt uses textual semantic prompts, the textual semantic features are used as the target semantic features; when the semantic prompt uses visual semantic prompts, the visual semantic features are used as the target semantic features. The target semantic features are denoted as: ,in, This refers to the target semantic features ultimately used to describe the high-level semantic information of the target to be segmented. These target semantic features include target category, appearance attributes, visual features, or spatial description information, and can be used in subsequent steps to guide the model in searching for regions consistent with the target semantics near ambiguous point cues.
[0054] In this embodiment, textual semantic prompts are suitable for scenarios where the target category, color, and spatial relationships can be accurately described using natural language; visual semantic prompts are suitable for scenarios where the target has a specific appearance, texture, or individual characteristics that are difficult to describe using simple text. Both semantic prompting methods can be used in conjunction with fuzzy point prompts to improve target recognition capabilities in the subsequent fuzzy point correction process through the joint constraints of spatial and semantic information.
[0055] Through the above steps, we obtain the multi-scale image features F corresponding to the target viewpoint image, the point hint features corresponding to the blurred point hints, and the target semantic features corresponding to the semantic hints, respectively. Among them, image features F provide visual information from the target's perspective image, point cue features provide approximate spatial location constraints of the target, and target semantic features provide high-level semantic information of the target object. These three types of features serve as input data for constructing the hybrid cue features in step S3.
[0056] Step S3: The target semantic features and the point prompt features are fused to obtain a hybrid prompt feature that simultaneously contains target semantic information and spatial location information.
[0057] In this embodiment, after completing step S2 and obtaining the image features corresponding to the target view image, the point prompt features corresponding to the blurred point prompts, and the target semantic features corresponding to the semantic prompts, step S3 is further executed to fuse the target semantic features and the point prompt features to obtain a hybrid prompt feature that simultaneously contains target semantic information and spatial location information.
[0058] To effectively integrate prior spatial location information with semantic information and address the "fuzzy point challenge," this application employs Conditional Deformable Attention (CDA) in the semantic guidance adapter. Traditional cross-attention often involves global computation, which is costly and easily interfered with when the drone image contains a large number of similar vehicles, building components, or natural backgrounds; simple window attention is limited by a fixed receptive field and struggles to adapt to varying degrees of drift caused by different altitudes, viewing angles, and calibration states. Therefore, this application uses fuzzy points as a reference and adaptively learns the offset and captures semantically relevant features through Conditional Deformable Attention. Figure 2 (a) in the figure gives the overall structure of the SGA. Figure 2 Figure (b) provides details of the CDA for the regression sampling offset around the blurred point. Terminology within the figure: Self-Attention; Prompt to Image; Image to Prompt; Conditional DeformableAttention; Offset Group; Attention Weights Group; Multi-Scale Image Feature.
[0059] Specifically, the target semantic features obtained in step S2 are mainly used to describe high-level semantic information such as the category, appearance attributes, and spatial relationships of the target to be segmented, while the point cue features are mainly used to represent the spatial location information corresponding to the fuzzy point cue. Both describe the target to be segmented from semantic and spatial dimensions, respectively. Therefore, this embodiment integrates the target semantic features and the point cue features, enabling subsequent modules to simultaneously utilize both target semantic constraints and spatial location constraints to further correct the region corresponding to the fuzzy point cue.
[0060] First, the target semantic features obtained in step S2 are concatenated with the point cue features to form a joint cue feature containing both semantic and spatial information. Specifically, the target semantic features are... With dot hint features The information is combined to establish a preliminary association between the target semantic information and the spatial location of the fuzzy points.
[0061] Subsequently, a self-attention mechanism is used to perform feature interaction on the concatenated joint cue features, enabling information transmission between different types of cue information in the feature space. Through this self-attention mechanism, the target semantic features can perceive the approximate spatial location provided by the ambiguous point cue, while the point cue features can fuse the semantic context information of the target to be segmented, thereby generating hybrid cue features that simultaneously possess target semantic understanding and spatial localization capabilities.
[0062] Specifically, the generation process of the hybrid prompt feature is as follows:
[0063] in, This indicates the generated hybrid cue features. This represents the target semantic features obtained in step S2. This represents the point cue features obtained in step S2, Concat represents the feature concatenation operation, and SelfAttn represents the self-attention calculation process.
[0064] In one implementation, the self-attention mechanism can dynamically adjust feature weights based on target semantic information and spatial location information by calculating the correlation between different cue features. Specifically, the attention region corresponding to the target semantic features can be constrained by the cue location, and the spatial information corresponding to the cue features can be supplemented based on the target semantic information, thereby avoiding the problem of erroneous target attention caused by relying solely on ambiguous point locations.
[0065] Furthermore, the hybrid prompting feature As input for subsequent semantic-guided offset correction, this information guides the model to search for an image region consistent with the target near the blurred point based on the target semantic information. Compared to methods that only utilize spatial location cues, this embodiment jointly encodes the target semantic information and spatial location information, enabling subsequent steps to recover the target region actually pointed to by the operator based on the target semantic constraints, even in the presence of point cue offset.
[0066] It should be noted that the hybrid cue features in this embodiment do not directly change the original coordinates of the ambiguous point cue, but rather establish the correlation between semantic information and spatial information in the feature space, providing semantic guidance information for subsequent sampling offset prediction based on conditionally deformable attention.
[0067] Through the above steps, a hybrid cue feature containing target semantic information and spatial location information is obtained. The hybrid cue feature serves as input data for determining the sampling offset and performing feature-level correction of blur points in step S4.
[0068] Step S4: Using the position corresponding to the blurred point prompt as a reference position, determine the sampling offset relative to the reference position based on the hybrid prompt features, and extract local image features from the image features according to the sampling offset to perform feature-level offset correction on the blurred point prompt to obtain the corrected prompt features.
[0069] In this embodiment, after completing step S3 and obtaining the hybrid prompt features containing target semantic information and spatial location information, step S4 is further executed to perform feature-level offset correction on the ambiguous point prompts based on the hybrid prompt features, so as to obtain corrected prompt features that match the semantics of the target to be segmented.
[0070] Specifically, since the fuzzy point cues obtained in step S1 can only provide an approximate location of the target to be segmented, their location may be offset due to factors such as cross-view mapping errors, device interaction errors, and image acquisition errors. Therefore, they cannot be directly used as accurate target locations for segmentation. To address the discrepancy between the fuzzy point cues and the actual target region, this embodiment employs conditionally deformable attention to perform feature-level offset correction on the fuzzy point cues.
[0071] In this embodiment, conditional deformable attention uses the hybrid cue features obtained in step S3. As query information, the location corresponding to the blurred point is used as a reference location, and feature sampling is performed on the multi-scale image features F obtained in step S2. The conditional deformable attention predicts the sampling offset related to the target semantics near the blurred point by utilizing the target semantic information and spatial location information contained in the hybrid cue features, enabling the model to perform feature-level search from the original offset location to the real target region.
[0072] Specifically, let p be the normalized coordinate corresponding to any cue point in the set of ambiguous cue points P. The coordinate p serves as the reference position in the conditionally deformable attention. Since ambiguous cue points may not accurately fall into the target region, this embodiment does not directly utilize the image features corresponding to p, but rather relies on hybrid cue features. Predict multiple sampling offsets relative to the reference position p and the corresponding attention weights .
[0073] Where h represents the index of the attention head, and k represents the index of the sampling position. This represents the offset of the k-th sampling point under the h-th attention head relative to the reference position p. This represents the attention weight at the corresponding sampling location. Through this method, the model can determine the degree of correlation between different regions near the ambiguous point and the target object based on the target's semantic information.
[0074] The specific process for calculating conditional deformable attention is as follows:
[0075] To elaborate further:
[0076] in, This represents the corrected semantic features obtained after conditionally deformable attention correction, where H represents the number of attention heads and K represents the number of sampling positions corresponding to each attention head. and Let F represent the learnable weights, and F represent the image features obtained in step S2. This indicates that based on the reference position p and the sampling offset Image features corresponding to the determined sampling locations.
[0077] When performing conditionally deformable attention, firstly based on the hybrid cue features Multiple sampling offsets and corresponding attention weights are predicted; then, multiple sampling positions are determined in the image features based on the sampling offsets, and local image features are extracted from the corresponding positions; finally, the multiple local image features are weighted and aggregated according to the attention weights to obtain the correction prompt features after semantic guidance and spatial correction.
[0078] Furthermore, the image feature F in this embodiment includes image features at multiple scales. Since the target object in a UAV mission may have different shooting heights, viewing angles, and target scales, using only a single-scale image feature may be insufficient to simultaneously capture both local target details and overall semantic information. Therefore, in one implementation, image features at different scales are used... The sampling positions at the corresponding scales are determined based on the predicted sampling offsets, and local image features are extracted from the image features at each scale. Subsequently, the local image features obtained at different scales are fused to obtain more complete correction prompt features.
[0079] Specifically, for the image features at the l-th scale The sampling position is determined based on the reference position p and the sampling offset at the corresponding scale, and the local features corresponding to that position are extracted. The local features obtained at different scales contain information about different levels of the target object. High-resolution scale features are used to preserve details such as target edges and textures, while low-resolution scale features are used to provide information about the target category and overall structure. By fusing multi-scale features, the accuracy of target region feature extraction during blur point correction is improved.
[0080] Through the aforementioned conditionally deformable attention mechanism, the model can perform semantic guidance correction of blurry point cues within the feature space without directly modifying the spatial coordinates of the blurry points. Even if the blurry point cues fall into the target boundary, background region, or adjacent interfering target region, the model can still use the target's semantic information to search for feature regions consistent with the target near the reference position, thereby obtaining geometrically calibrated corrected cue features.
[0081] Furthermore, the corrected cue features are used in the subsequent step S5 for bidirectional interaction and fusion of cue features and image features to further enhance the feature representation of the target-related region and provide a feature basis for the final generation of the target segmentation mask. It should be noted that the sampling offset in this embodiment is only used for feature-level correction during the segmentation process; it does not change the actual flight control coordinates of the UAV, nor does it participate in flight path planning or motion control.
[0082] Step S5: Perform bidirectional interactive fusion of the correction prompt features and the image features to obtain fused features.
[0083] In this embodiment, after completing step S4 and obtaining the correction prompt features, step S5 is further executed to perform bidirectional interactive fusion of the correction prompt features and the image features to obtain fused features, which provides a more complete target feature expression for the subsequent generation of the target segmentation mask.
[0084] Specifically, after the conditional deformable attention processing in step S4, the corrected prompt features have fused target semantic information and local image information near the blurred points after feature-level correction, enabling a more accurate representation of the target area that the operator is truly focused on. However, the corrected prompt features mainly complete the filtering of local image areas by the target prompt information, while the image features corresponding to the target viewpoint image still retain rich scene information and target details. Therefore, this embodiment further performs bidirectional interactive fusion of the corrected prompt features and image features, enabling the two types of features to update each other, thereby obtaining fused features that simultaneously contain target semantic information, spatial location information, and image visual information.
[0085] In this embodiment, the bidirectional interaction fusion is implemented using a bidirectional Transformer structure, which includes two stages: interaction from prompt to image and interaction from image to prompt.
[0086] First, the interaction process from prompt to image is executed. Specifically, the corrected prompt features obtained in step S4 are used as query information, and the image features obtained in step S2 are used as key and value. Cross-attention calculation is performed so that the corrected prompt features can further update their feature representation based on the visual information in the target viewpoint image.
[0087] In this process, the corrected prompt feature utilizes the target semantic information and spatial correction information obtained in step S4 to further focus on image regions related to the target object in the image features, thereby extracting richer target visual features and suppressing interference from background regions and irrelevant target regions. Since the corrected prompt feature has completed feature-level offset correction for blurry point prompts, the interaction with the image can more accurately locate the target region and improve the expressive power of target-related image features.
[0088] After completing the interaction from prompt to image, the interaction from image to prompt is further executed. Specifically, using image features as query information and corrected prompt features as key and value, cross-attention calculation is performed again, enabling image features to actively focus on the target semantic information contained in the corrected prompt features.
[0089] Through the above processing, the image features are updated based on the target category, appearance, and spatial location information contained in the correction cue features. This enhances the response of regions related to the target to be segmented in the image features and reduces the influence of background regions and adjacent interfering targets on the image features. In other words, the interaction between cue and image guides the image features with target cue information, while the interaction between image and cue further realizes the information feedback between image features and target cue, enabling the two types of features to be fully integrated.
[0090] Furthermore, after completing the prompt-to-image interaction and the image-to-prompt interaction, the results of the two interactions are fused to obtain the final fused feature. The fused feature simultaneously includes target semantic information, spatial location information, and target image information, which can more completely represent the target to be segmented and provide a reliable feature basis for subsequent target mask generation.
[0091] It should be noted that the bidirectional interactive fusion in this embodiment is based on the correction prompt features obtained in step S4. Therefore, the correction prompt features not only include the target semantic information and point prompt features obtained in step S2, but also incorporate the local target features extracted from the image features in step S4 based on conditional deformable attention. Through bidirectional interactive fusion, the image features can further perceive the target semantic information, and the correction prompt features can further fuse the image visual information, thereby forming a more consistent target feature representation.
[0092] Compared to a one-way interaction approach, the two-way interactive fusion in this embodiment enables bidirectional information transfer between cue features and image features. On one hand, correcting the cue features guides the model to focus more accurately on the target region; on the other hand, the image features can further update their representation based on the target cue information, resulting in fused features with stronger target discrimination capabilities and better semantic consistency.
[0093] Through the above steps, the bidirectional interactive fusion of correction prompt features and image features is completed, resulting in fused features. These fused features serve as input features for generating the segmentation mask of the target to be segmented in step S6, and are used to further predict target regions consistent with semantic prompts.
[0094] Step S6: Generate a segmentation mask for the target to be segmented that corresponds to the semantic prompt based on the fusion features.
[0095] In this embodiment, after completing step S5 and obtaining the fusion features, step S6 is further executed to generate a segmentation mask for the target to be segmented corresponding to the semantic prompt based on the fusion features, thereby completing the segmentation process from the target view image, the blur point prompt, and the semantic prompt to the target pixel region.
[0096] Specifically, the fusion features obtained in step S5 include target cue information corrected for offset and image information obtained through bidirectional interaction with the target cue information. The corrected cue features characterize the target information after semantic guidance and spatial location correction, while the image features characterize target details and surrounding scene information in the target view image. Through the bidirectional interaction in step S5, a correspondence has been established between the two types of information, thus enabling them to serve as input for generating the target segmentation mask.
[0097] In this embodiment, the fusion features obtained in step S5 are input into the mask decoder. The mask decoder performs segmentation prediction on each image region in the target view image based on the relevant information between different spatial locations in the fusion features and the target to be segmented, and obtains the target segmentation mask corresponding to the target described by the semantic prompt.
[0098] Specifically, the target segmentation mask corresponds spatially to the target view image I in step S1, and is used to represent the pixel region in the target view image that belongs to the target to be segmented. In one embodiment, the target segmentation mask is a binary segmentation mask, wherein the pixel position corresponding to the target to be segmented is divided into the target region, and the remaining pixel positions are divided into non-target regions, thereby forming the pixel-level spatial range of the target to be segmented in the target view image.
[0099] Since the fusion features used to generate the target segmentation mask have undergone the fusion of target semantic information and spatial location information in step S3, the semantic guidance feature-level offset correction in step S4, and the bidirectional interaction between the prompt features and image features in step S5, when the fuzzy point prompt obtained in step S1 does not fall accurately inside the target to be segmented, but is located on the target boundary, background area, or adjacent interfering target area, it is still possible to use the target category, color, appearance attributes, spatial relationship, or visual reference information provided by the semantic prompt to distinguish the candidate region in the target view image and generate a target segmentation mask corresponding to the semantic prompt.
[0100] For example, in a drone road patrol scenario, when multiple adjacent vehicles exist in the target view image, and the blurred point cues obtained from cross-view mapping fall into the region of another vehicle adjacent to the target vehicle, steps S3 to S5 can further combine the target vehicle's color, appearance, or spatial relationship information contained in the semantic cues to correct and fuse the image features near the blurred point. In step S6, the mask decoder determines the vehicle region consistent with the semantic cues based on the fused features obtained through the above processing, thereby outputting the corresponding target vehicle segmentation mask, rather than determining the object to be segmented solely based on the location of the blurred point.
[0101] Furthermore, the target segmentation mask output in step S6 represents the pixel-level region of the target to be segmented in the target view image and can be used as input to subsequent UAV perception or task processing modules. In one embodiment, target confirmation can be performed based on the target segmentation mask to determine the target object in the target view image that corresponds to the operator's semantic cues; in another embodiment, the target region provided by the target segmentation mask can be used for further target localization or target tracking; the target segmentation mask can also be provided to the task planning module for subsequent observation position determination or other task processing.
[0102] It should be noted that the target confirmation, target location, target tracking, or mission planning described in this embodiment are subsequent applications of the target segmentation results. Step S6 itself completes the process of generating a segmentation mask for the target to be segmented based on the fusion features, and does not limit the specific implementation methods of subsequent flight control, path planning, or safety decisions.
[0103] Through steps S1 to S6, the target view image, blurry point cues, and semantic cues are first acquired, and feature encoding is performed on each of the three input types. Then, the target semantic features and point cue features are fused to form a hybrid cue feature that simultaneously contains target semantic information and spatial location information. Further, using the location corresponding to the blurry point cues as a reference location, a sampling offset is determined through conditionally deformable attention, and local image features matching the target semantics are extracted from multi-scale image features to complete feature-level offset correction of the blurry point cues. Afterwards, the corrected cue features and image features are bidirectionally interacted and fused, and finally, a target segmentation mask corresponding to the semantic cues is generated based on the obtained fused features. Therefore, even when there is a positional deviation in the blurry point cues, the approximate spatial location and semantic information of the target can be comprehensively utilized to complete target region segmentation.
[0104] In one possible embodiment, the semantic guidance ambiguity point segmentation method for UAV cross-view command proposed in this application can be implemented on top of an existing interactive target segmentation framework, and its overall architecture is as follows: Figure 3 As shown. This embodiment uses "Segment Anything Model 2 (SAM2)" as the basic segmentation framework for illustration, but those skilled in the art will understand that the method of this application can also be deployed on other target segmentation frameworks with prompting input capabilities, and this embodiment does not limit it in this regard.
[0105] This embodiment uses the scenario of collecting evidence of illegally parked vehicles during UAV road patrol as an example. The target view image is acquired by the UAV's onboard camera, and the image contains the target vehicle to be segmented and adjacent vehicles. The ground operator provides semantic prompts to describe the target vehicle and fuzzy point prompts to indicate the approximate position of the target vehicle. The semantic prompts can be text descriptions, such as "blue vehicle," or target reference images and corresponding target reference areas from the source view. The fuzzy point prompts can be obtained by the operator clicking directly in the target view image, or they can be obtained from the target position in the source view through cross-view mapping. Therefore, the fuzzy point prompts are allowed to have a certain deviation from the actual position of the target vehicle.
[0106] Specifically, the target viewpoint image is first input into an image encoder for feature extraction. In this embodiment, the Hiera image encoder in SAM2 is used as the image encoder to encode the target viewpoint image and obtain multi-scale image features. At the same time, the location of the blurred point cues is encoded to obtain point cue features; depending on the input form of the semantic cue, the target semantic features are obtained using either a text encoder or a reference image encoding branch.
[0107] Subsequently, the target semantic features and the point cue features are fused to obtain a hybrid cue feature that simultaneously contains target semantic information and spatial location information. Further, using the hybrid cue feature as query information and the location corresponding to the blurred point cue as a reference location, conditional deformable attention is performed on the image features. A sampling offset is predicted based on the target semantic information, and local image features matching the target semantics are extracted from the neighborhood of the blurred point based on the sampling offset, thereby achieving feature-level offset correction for the blurred point cue.
[0108] After obtaining the correction prompt features, the correction prompt features and image features are input into the bidirectional interaction module for information fusion. Specifically, firstly, the correction prompt features are used as query information, and the image features are used as keys and values to perform a prompt-to-image interaction; subsequently, the image features are used as query information, and the correction prompt features are used as keys and values to perform an image-to-prompt interaction, obtaining fused features through bidirectional information transmission.
[0109] Finally, the fused features are input into the mask decoder to generate a target segmentation mask corresponding to the semantic cues. For example, when there are multiple adjacent vehicles in the target view image and the blurry dot cues fall into the region of adjacent vehicles, the model can still perform semantic matching on candidate targets in the neighborhood of the blurry dot based on the target color, appearance attributes, or visual reference information provided in the semantic cues, thereby outputting a target segmentation mask that matches the operator's true intention. Figure 1 The target vehicle segmentation mask.
[0110] The generated target segmentation mask can be used as input for subsequent modules such as target confirmation, target localization, target tracking, or mission planning. For example, in a scenario where a drone is used to inspect illegally parked vehicles, the target vehicle's location can be further determined based on the target segmentation mask, providing target area information for subsequent observation pose planning and license plate image acquisition. It should be noted that the target segmentation method disclosed in this embodiment, as a target segmentation module in the drone's perception link, can be deployed on an airborne computing platform or a ground computing platform. Its output results are used by subsequent processing modules and do not directly participate in the drone's flight control, obstacle avoidance control, and safety decision-making processes.
[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0112] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A semantic guidance fuzzy point segmentation method for cross-view command of unmanned aerial vehicles, characterized in that, Includes the following steps: Step S1: Obtain the target view image, a blurry dot cue indicating the approximate location of the target to be segmented, and a semantic cue describing the target to be segmented; Step S2: Extract features from the target view image to obtain image features; The ambiguous point prompts are positionally encoded to obtain point prompt features; The semantic prompts are semantically encoded to obtain target semantic features; Step S3: Fuse the target semantic features with the point prompt features to obtain a hybrid prompt feature that simultaneously contains target semantic information and spatial location information; Step S4: Using the position corresponding to the blurred point prompt as a reference position, determine the sampling offset relative to the reference position based on the hybrid prompt features, and extract local image features from the image features according to the sampling offset, perform feature-level offset correction on the blurred point prompt, and obtain the corrected prompt features; Step S5: Perform bidirectional interactive fusion of the correction prompt features and the image features to obtain fused features; Step S6: Generate a segmentation mask for the target to be segmented that corresponds to the semantic prompt based on the fusion features.
2. The method according to claim 1, characterized in that, The fuzzy point hints include at least one hint point used to indicate the approximate location of the target to be segmented; The blur point hints are obtained by the user directly selecting them in the target view image, or by obtaining the target position in the source view through cross-view mapping; the blur point hints are allowed to deviate from the actual position of the target to be segmented and fall into the target boundary, background area or adjacent interfering target area.
3. The method according to claim 1, characterized in that, The semantic cues include at least one of textual semantic cues and visual semantic cues; wherein, the textual semantic cues include at least one of target category, color, appearance attribute and spatial relationship, and the visual semantic cues include a source viewpoint reference image and a corresponding target reference region.
4. The method according to claim 3, characterized in that, Step S2, which involves semantically encoding the semantic prompt, includes: When the semantic prompt is a text semantic prompt, the text description is encoded to obtain text semantic features; When the semantic cues are visual semantic cues, feature extraction is performed on the source view reference image, reference features corresponding to the target region are extracted based on the reference region, and the reference features are aggregated to obtain visual semantic features; The textual semantic features or the visual semantic features are used as the target semantic features.
5. The method according to claim 1, characterized in that, Step S3 includes: The target semantic features are concatenated with the point cue features; By using a self-attention mechanism to interact with the concatenated features, a correlation between target semantic information and spatial location information is established, thereby obtaining the hybrid prompt features.
6. The method according to claim 5, characterized in that, Step S4 employs conditional deformable attention for feature-level offset correction, including: Using the hybrid prompt features as query information and the location corresponding to the fuzzy point prompt as a reference location, multiple sampling offsets and corresponding attention weights relative to the reference location are predicted based on the hybrid prompt features. Based on the multiple sampling offsets, multiple sampling positions are determined in the image features, and local image features corresponding to the sampling positions are obtained; The local image features are weighted and aggregated according to the attention weights to obtain the correction prompt features.
7. The method according to claim 6, characterized in that, The image features include image features at multiple scales; In step S4, feature sampling is performed on the image features at each scale according to the sampling offset, and the local image features obtained at each scale are fused to obtain the correction prompt features.
8. The method according to claim 1, characterized in that, Step S5 includes: Using the correction prompt features as query information and the image features as keys and values, the interaction between the prompt and the image is performed. The image features are used as query information, and the correction prompt features are used as keys and values to perform image-to-prompt interaction; The fusion feature is obtained based on the results of the two interactions.
9. The method according to claim 1, characterized in that, The target view image is an image captured by the UAV's onboard camera; The blurred dot cues are derived from the target location in airborne footage, ground terminals, augmented reality devices, ground cameras, other drones, or historical images; The segmentation mask is used for subsequent target confirmation, target location, target tracking, or mission planning.
