An unmanned aerial vehicle image segmentation method, device and medium

Abstract

The application discloses a UAV reference image segmentation method, device and medium, belongs to the technical field of computer vision, and solves the problem of how to improve the image reference segmentation precision in a complex UAV aerial scene; the UAV aerial image and the natural language description are taken as input, visual tokens and text tokens are respectively extracted through a visual encoder and a text encoder, a dense visual graph and a sparse text graph are respectively constructed, node-level matching and structure-level matching are performed on the dense visual graph and the sparse text graph, and a node-level and structure-level matching matrix is generated; a multimodal alignment score is calculated, the visual tokens related to the natural language description are selected and reserved according to the score, the irrelevant visual tokens are pruned as background tokens, the pruned visual tokens are reconstructed and spliced with the reserved visual tokens to form a new visual token sequence; and the new visual token sequence is input into a segmentation decoder to generate a reference target segmentation mask.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a method, device and medium for UAV referential image segmentation based on semantic disambiguation and information balance. Background Technology

[0002] Drone aerial imagery has wide applications in scenarios such as emergency rescue, traffic inspection, agricultural monitoring, urban governance, and low-altitude intelligent sensing. Compared with ordinary natural images, drone images typically have a large field of view, small target scale, complex backgrounds, dense clusters of similar targets, and significant variations in shooting angle. In these scenarios, it is difficult to accurately locate specific targets based solely on user-input natural language descriptions using traditional target detection or semantic segmentation methods.

[0003] In recent years, Transformer-based visual-language fusion methods have significantly improved the performance of image denotation segmentation in natural images. Some remote sensing and UAV image denotation segmentation methods have also begun to address the issues of small targets and complex backgrounds in aerial images. Existing technologies, such as the invention patent with publication number CN120340034A, disclose a free text-guided remote sensing image denotation segmentation method. This method generates multi-scale visual features by performing multi-scale perception and dynamic response enhancement on the image, and generates attribute-object information by semantic modeling of the text. It then utilizes a region relation modeling module to perform region-visual and region-language modeling interactions on the aforementioned visual and text features, respectively, and finally achieves multi-head prediction output through joint decoding.

[0004] However, existing methods still have the following significant shortcomings: (1) Most existing methods rely on implicit attention matching, which mainly fuses visual regions and text words based on similarity. They lack explicit modeling of structural relationships such as spatial adjacency and semantic similarity between targets and similar surrounding objects in aerial images. When there are a large number of targets with similar colors, shapes and categories in the image, the model is prone to mismatching the text description to nearby objects, resulting in mismatch and segmentation shift. (2) Drone aerial images usually contain a large number of background areas and redundant visual information, while the text descriptions entered by users are often relatively short, containing only a small amount of key information such as target category, color, orientation and spatial relationship. This imbalance between the high information density of the image and the low information density of the text will cause the key semantic clues in the text to be submerged by the complex visual background, making it difficult for the model to accurately focus on the target area.

[0005] In summary, existing UAV-based image segmentation methods struggle to address semantic ambiguity arising from densely similar targets within images, as well as the information asymmetry between wide-field-of-view images and concise text. Therefore, there is an urgent need for an image segmentation method capable of explicitly modeling the structural relationship between images and text, and dynamically balancing multimodal information density, to improve the accuracy and robustness of image segmentation in complex UAV aerial photography scenarios. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to improve the accuracy of image referencing and segmentation in complex UAV aerial photography scenarios.

[0007] The present invention solves the above-mentioned technical problems through the following technical solutions: A method for segmenting images of drone references includes: The input consists of drone aerial images and natural language descriptions, and visual and textual lexical units are extracted by a visual encoder and a text encoder, respectively. Using visual lexical units and textual lexical units as input, dense visual graphs and sparse text graphs are constructed respectively. Node-level matching and structure-level matching are then performed on the dense visual graphs and sparse text graphs to generate node-level matching matrices and structure-level matching matrices, thus establishing the correspondence between image structure and language structure. Using visual lexical units and textual lexical units as input, multimodal alignment scores are calculated. Based on the scores, visual lexical units that are relevant to the natural language description are selected for retention, while irrelevant visual lexical units are pruned as background lexical units. The pruned visual lexical units are reconstructed and concatenated with the retained visual lexical units to form a new visual lexical unit sequence. The new visual lexical sequence is input into the segmentation decoder along with the text lexical sequence to generate a segmentation mask that refers to the target.

[0008] Furthermore, the dense visual graph includes a sequence of visual lexical units. and image adjacency matrix ; where each word element The corresponding two-dimensional spatial coordinates on the feature map , , , The total number of visual lexical units. For the set of real numbers, The feature dimensions of each visual lexical unit; Constructing the image adjacency matrix The process is as follows: First, select each visual lexical unit based on Euclidean distance. The nearest neighbor words are used to generate a binary space mask as follows:

[0009] in, Represents the first in the binary space mask Line 1 Column elements, ; Representation and Lexical Elements The closest in spatial Euclidean distance A set of indices for each term element. and They are respectively word elements and Spatial coordinates in the feature map Indicates Euclidean distance; Next, a learnable projection matrix is ​​used. Visual lexical units are mapped to a similarity space, and the image temperature coefficient is used as a reference. Controlling the sharpness of the distribution, the semantic similarity matrix is ​​calculated as follows:

[0010] in, The semantic similarity matrix is ​​the first... Line 1 Column elements, Scaling factor For image temperature coefficient, The projected weight matrix is ​​a learnable matrix. Describes the first... Perform Softmax normalization along the column direction; Finally, the balance coefficient is introduced. By fusing the binary space mask and the semantic similarity matrix, and performing row normalization on the image adjacency matrix, the final image adjacency matrix is ​​obtained, which is represented by the following logic:

[0011] in, The first in the image adjacency matrix Line 1 The elements of the columns, after row normalization of the image adjacency matrix, are as follows: , For the image adjacency matrix, the first... The sum of the elements in the row, .

[0012] Furthermore, the sparse text graph includes a sequence of text lexical terms. and text adjacency matrix ; Constructing a text adjacency matrix The process is as follows: First, a directed dependency tree is generated for the text word sequence using a pre-trained syntactic parser, and then converted into a binary syntactic dependency mask. ,in Representing text terms and There is a direct grammatical modification relationship between them. The total number of text words; Next, prior weights are assigned based on different part-of-speech tags. ; Finally, through independent projection matrices Calculate the scaled dot product similarity between text terms and directly sum the syntactic bias terms. The text adjacency matrix is ​​as follows:

[0013] in, For text adjacency matrix The Middle Line 1 Column elements, , Text words respectively , The corresponding prior weights, Scaling factor This refers to the temperature coefficient of the text. The syntactic bias coefficient. For the syntactic dependency mask matrix, the first Line 1 Column elements, Describes the first... Perform Softmax normalization along the row and column directions.

[0014] Furthermore, the node-level matching matrix is ​​generated by calculating the dot product similarity between visual lexical units and textual lexical units, using the following logical representation:

[0015] in, For the node-level matching matrix, the first... Line 1 The elements of the column represent text terms. With visual lexical The correlation, All are learnable projection matrices. Scaling factor Describes the first... Perform Softmax normalization along the row and column directions.

[0016] Furthermore, the structure-level matching matrix is ​​generated by performing neighborhood aggregation on the adjacency matrices corresponding to the dense visual image and the sparse text image, respectively, to construct image structure features and text structure features, and then using the image structure features and text structure features to perform structure-level matching. The specific process is as follows: Using image adjacency matrix Neighborhood aggregation is obtained And construct the center-neighborhood difference feature, using the following logical representation:

[0017]

[0018] in, This is a neighborhood aggregation feature. This represents image structural features, and also center-neighborhood difference features. It is a two-layer perceptron with an input dimension of . The hidden layer dimension is The output dimension is Similarly, text structure features are constructed. Next, structural-level matching is performed in the enhanced feature space, and the structural-level matching matrix is ​​calculated:

[0019] in, For the structure-level matching matrix, the first... Line 1 Column elements, Each is an independent projection matrix. , This is the scaling factor.

[0020] Furthermore, the multimodal alignment score consists of a visual-text alignment score, a structure centrality score, and an autosalience score. The calculation process for the multimodal alignment score is as follows: Based on node-level matching matrix For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the visual-text alignment score. ,in For the node-level matching matrix, the first... Line 1 Column elements, This indicates that for the node-level matching matrix The Middle The set of column indices corresponding to the k largest elements of a row, sorted from largest to smallest. Based on structural matching matrix For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the structure centrality score. ,in For the structure-level matching matrix, the first... Line 1 Column elements, This indicates that for the structural matching matrix The Middle The set of column indices corresponding to the k largest elements of a row, sorted from largest to smallest. Utilizing visual lexical elements With global text representation Calculate the self-significance score ,in Visual lexical elements The corresponding self-significance score; It is represented as a two-layer perceptron, with an input dimension of 2d, a hidden layer dimension of d, and an output dimension of 1; Indicates visual lexical elements With global text representation The components are stitched together along the channel dimension. The global text representation is obtained by calculating the mean of the text words in the text word sequence; The visual-text alignment score, structure centrality score, and self-salience score were normalized using Min-Max and compared with visual lexical units. splicing, inputting into a multilayer perceptron The multimodal alignment score is obtained by activating the Sigmoid function as follows:

[0021] in, Represents primitive visual lexical units The corresponding multimodal alignment score, ; It is a two-layer perceptron with an input layer dimension of . The hidden layer dimension is The output layer dimension is , This is the Sigmoid function.

[0022] Furthermore, Top-K selection is performed based on the multimodal alignment score to generate a binary pruning mask; wherein visual words with a mask of 1 are retained, and visual words with a mask of 0 are pruned.

[0023] Furthermore, for visual lexical units with a mask of 0, background lexical units are first generated through global average pooling. And concatenate it with the visual lexical with a mask of 1 to form a new visual lexical sequence. ,in This indicates that visual lexical units with a mask of 1 are retained.

[0024] An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the aforementioned UAV-referencing image segmentation method, the processor being configured to execute the program stored in the memory.

[0025] A storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described UAV-represented image segmentation method.

[0026] The advantages of this invention are: (1) This invention proposes an image referencing segmentation method for complex UAV aerial photography scenarios. The UAV aerial photography image and natural language description are taken as input. Visual words and text words are extracted by a visual encoder and a text encoder, respectively. Then, a dense visual graph and a sparse text graph are constructed by a dual-graph semantic disambiguation module. Node-level matching and structure-level matching are performed on the above graph structures to generate node-level and structure-level matching matrices, and the correspondence between the image structure and the language structure is established. Then, the adjacency matrix corresponding to the visual words, text words and dense visual graph are taken as input. The multimodal alignment score is calculated by an alignment-guided word pruning module. Based on the score, visual words related to the text description are selected to be retained, and irrelevant words are pruned as background words. The pruned background words are reconstructed to form a new visual word sequence. After being input into the segmentation decoder, the target segmentation mask is output.

[0027] This invention leverages the modeling advantages of graph structures in spatial adjacency and textual grammatical dependencies to enhance the ability to distinguish densely similar targets. Simultaneously, it utilizes alignment scores to adaptively filter redundant background words, thereby balancing the information density of images and text. This enables high-precision image referencing segmentation in complex UAV aerial photography scenarios, improving the accuracy of image referencing segmentation in such scenarios.

[0028] (2) This invention proposes a reference image segmentation framework specifically for UAV aerial photography scenarios, filling the gap in the adaptability of existing methods in wide-field-of-view, dense target scenarios. This invention addresses the characteristics of UAV aerial images, such as large field of view, small target scale, complex background, and dense arrangement of similar targets. It designs a reference image segmentation network framework that includes a dual-image semantic disambiguation module and an alignment-guided lexical pruning module. This network framework can accurately locate and segment specified targets pixel-level in complex aerial images in an end-to-end manner based on user-input natural language descriptions. It can be widely applied to practical low-altitude perception scenarios such as emergency rescue, traffic inspection, agricultural monitoring, urban governance, power line inspection, and security monitoring, and has good engineering application prospects.

[0029] This invention effectively utilizes the complementary but asymmetric multimodal information between UAV aerial images and natural language descriptions. It enhances target discrimination capabilities through dual-image semantic disambiguation, solving the matching confusion problem caused by densely similar targets. By using alignment-guided lexical pruning to achieve visual information filtering, it alleviates the information density imbalance problem between wide-field aerial images and short text descriptions, making the model more suitable for UAV aerial image reference segmentation scenarios with wide field of view, dense targets, and complex backgrounds.

[0030] (3) This invention achieves explicit modeling of image structure and text structure through a dual-image semantic disambiguation module, effectively solving the semantic ambiguity problem caused by densely similar targets: This invention constructs two heterogeneous graph structures: a dense visual graph and a sparse text graph. The dense visual graph encodes both local spatial topological relationships and global semantic consistency in the image through spatial adjacency and feature semantic similarity. The sparse text graph, on the other hand, accurately extracts key modifications and main structures in the text through syntactic dependency priors and part-of-speech importance weights. Based on this, this invention further establishes a basic correspondence between visual regions and text words through node-level matching, and then verifies whether the local structure of the image is consistent with the semantic structure of the text in the enhanced feature space through structure-level matching. This enables the model to distinguish interference objects that are similar in appearance but different in context, and in principle, significantly reduces the mismatch rate in densely similar target scenes.

[0031] (4) This invention achieves adaptive filtering of visual information guided by text through the alignment-guided lexical pruning module, effectively alleviating the problem of information density imbalance between aerial images and short text: The multimodal alignment score calculated in this invention consists of a visual-text alignment score, a structural centrality score, and an autosalience score. The visual-text alignment score determines the basic semantic relevance between the visual region and the text description; the structural centrality score determines the contextual importance of the region within the visual graph structure; and the autosalience score determines whether the region itself possesses prominent visual features. These scores complement each other across three dimensions to form the multimodal alignment score, providing a basis for subsequent pruning decisions. This ensures that pruning decisions consider not only "what the region resembles," but also "whether the region's position in the graph is important" and "whether the region itself is salient." Because this invention uses structural relationships and textual guidance to jointly filter visual information, it reduces the risk of small targets being obscured by the background or incorrectly pruned, improves the segmentation effect of small targets in UAV aerial photography, and enhances the robustness of the model under conditions of small targets and complex backgrounds.

[0032] Furthermore, this invention does not directly discard pruned background lexical units. Instead, it reconstructs them into a compact background representation through background-aware lexical unit reconstruction, which is then concatenated with the retained target-related lexical units and input into the segmentation decoder. This approach significantly reduces the interference of background lexical units on text semantics, making key referential information more concentrated, while preserving necessary global scene context and avoiding excessive information loss. In principle, it achieves an adaptive balance between image and text information density, preserving target-related regions and suppressing irrelevant background regions, thereby alleviating the information density imbalance problem between aerial images and short texts. Attached Figure Description

[0033] Figure 1 This is an overall structural diagram of the denotation segmentation network framework in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the workflow of the dual-image semantic disambiguation module in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the alignment guide word pruning module of Embodiment 1 of the present invention; Figure 4 This is a visualization result of Embodiment 1 of the present invention; Figure 5 This is a comparative experimental result diagram of Embodiment 1 of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: Example 1 In real-world drone aerial photography scenarios, image referential segmentation faces two core challenges: first, the severe visual ambiguity caused by densely distributed similar objects; and second, the inherent information density asymmetry between the information-rich (high-entropy) wide-field image and the extremely brief (low-entropy) referential text, which makes key textual clues easily submerged by background visual noise.

[0036] To overcome the above problems, such as Figure 1 As shown, this invention proposes a semantic disambiguation and information balance-based subspecies segmentation network framework (SDIE). The subspecies segmentation network framework comprises an input layer, a dual-graph semantic disambiguation module (DGSD), an alignment-guided lexical pruning module (AGTP), and an output layer connected sequentially. The subspecies segmentation network framework takes UAV aerial images and natural language descriptions as input. Visual and text subspecies are extracted using a visual encoder and a text encoder, respectively. The dual-graph semantic disambiguation module then constructs dense visual and sparse text graphs. Node-level and structure-level matching matrices are generated from these graph structures to establish the correspondence between image and language structures. Using visual subspecies, text subspecies, and the adjacency matrix corresponding to the dense visual graph as input, the alignment-guided lexical pruning module calculates multimodal alignment scores. Based on these scores, visual subspecies relevant to the text description are retained, while irrelevant subspecies are pruned as background subspecies. The pruned background subspecies are reconstructed to form a new visual subspecies sequence, which is then input into the segmentation decoder to output a target segmentation mask. This invention leverages the modeling advantages of graph structures in spatial adjacency and textual grammatical dependencies to enhance the ability to distinguish densely similar targets. Simultaneously, it utilizes alignment scores to adaptively filter redundant background words, thereby balancing the information density of images and text. This enables high-precision image referencing segmentation in complex UAV aerial photography scenarios, improving the accuracy of image referencing segmentation in such scenarios.

[0037] like Figure 1 As shown, based on the overall structure of the aforementioned denotation segmentation network framework, this invention provides a UAV denotation image segmentation method based on semantic disambiguation and information balance, comprising the following steps: S1 takes drone aerial images and natural language descriptions as input, and extracts visual and text lexical units through a visual encoder and a text encoder, respectively.

[0038] like Figure 1As shown, the input layer deploys a visual encoder and a text encoder; the visual encoder receives aerial images taken by the UAV and encodes them into a set of visual words V; the text encoder receives natural language descriptions and encodes them into a set of text words T.

[0039] S2, the dual-graph semantic disambiguation module takes visual lexical V and text lexical T as input, constructs a dense visual graph and a sparse text graph respectively, and performs node-level matching and structure-level matching on the dense visual graph and the sparse text graph to generate a node-level matching matrix and a structure-level matching matrix, and establishes the correspondence between the image structure and the language structure.

[0040] like Figures 1-2 As shown, visual lexical V and textual lexical T are fed into a dual-graph semantic disambiguation module. Visual lexical V is constructed into a dense visual graph to express the spatial adjacency and semantic similarity relationships between different regions in the aerial image; textual lexical T is constructed into a sparse text graph to express the syntactic dependencies and semantic associations between different words in natural language descriptions. The node-level matching matrix and structure-level matching matrix generated based on the above two graph structures are used to enable the referencing segmentation network framework to better distinguish between the target and surrounding similar interference objects, thereby enhancing the model's ability to distinguish between the referred target and surrounding similar interference objects.

[0041] Graph modeling, as a relational modeling method, can represent image regions or text words as nodes and describe the spatial, semantic, or syntactic relationships between nodes through edges. In visual language tasks, graph modeling can further characterize the adjacency relationships between image regions, the contextual relationships between the target and the background, and the dependencies between category words, attribute words, and locative words in text descriptions. Compared to ordinary attention mechanisms, graph modeling is more suitable for expressing the structural information between "target-neighborhood-reference point". Specifically, S2 in this embodiment includes the following: S2.1 Construct a dense visual graph based on the positional relationships and semantic similarities between visual lexical units V. ,in It is a visual word sequence. This is the image adjacency matrix.

[0042] Specifically, given the visual word sequence output by the visual encoder Each word element It corresponds to a local region in the input drone aerial image and has its two-dimensional spatial coordinates on the feature map. , , The total number of visual lexical units. For the set of real numbers, For each lexical feature dimension, the entire lexical sequence The visual lexical sequence is obtained by the visual encoder after segmenting and embedding the input image into blocks. Each visual lexical represents the appearance semantic information of a local region in the image.

[0043] To explicitly model the structural dependencies between local regions, the image adjacency matrix... The image adjacency matrix is ​​constructed by jointly driving spatial adjacency and semantic similarity. The process is as follows: (1) Spatial adjacency: In order to preserve the spatial context of local regions, each visual lexical unit is selected based on Euclidean distance. The nearest neighbor words are used to generate a binary space mask as follows:

[0044] in, Represents the first in the binary space mask Line 1 Column elements, ; Representation and Lexical Elements The closest in spatial Euclidean distance A set of indices for each term element. express Belongs to set (i.e., word elements) Spatially with lexical units (neighborhood) and the above relationship is expressed by Euclidean distance. Verify to ensure the tokens are valid. It is indeed a distance word. Recent One of the word elements, at this time Take 1; and They are respectively word elements and Spatial coordinates in the feature map Represents Euclidean distance. Binary space mask. This approach forces each node to establish basic connections only with its spatially adjacent nodes, effectively capturing local layout relationships such as "vehicle-road" and "vehicle-adjacent vehicle"; this embodiment uses a dense visual graph. Each node is a visual lexical unit. That is, a local region (Patch) in an image, the total number of visual units. That is, the total number of nodes; similarly, the subsequent sparse text graph Each node corresponds to a text word. Preferably, in this embodiment, K is 9.

[0045] (2) Feature semantic similarity: In order to capture long-range dependencies and global semantic consistency, a learnable projection matrix is ​​adopted. Visual lexical units are mapped to a similarity space, and the image temperature coefficient is used as a reference. To control the sharpness of the distribution, the dense semantic similarity matrix is ​​calculated as follows:

[0046] in, The semantic similarity matrix is ​​the first... Line 1 Column elements, The projected weight matrix is ​​a learnable matrix. This is used to transform the original word features into a feature space that is more suitable for calculating similarity. Describes the first... The rows undergo Softmax normalization, where normalization is performed along the column direction (i.e., ...). (dimension) to perform, so that for a fixed number of rows The sum of the output values ​​in all columns is 1, that is... This operation converts the original similarity score of each line into a probability distribution, reflecting the relative matching strength of each visual word to all text words. This is the image temperature coefficient, used to adjust the sharpness of the Softmax distribution. , The smaller the value, the more concentrated the similarity distribution is on the most similar word groups; This is a scaling factor used to prevent the dot product result from becoming too large and causing the gradient to enter the saturation region. Semantic similarity matrix. This enables the establishment of soft connections between targets that are spatially far apart but have similar appearances, such as connecting vehicles of the same model that are scattered in a parking lot but have the same appearance.

[0047] (3) Spatial location and feature semantic fusion: Finally, weighted parameters are introduced. By integrating the spatial adjacency relationships and semantic similarities mentioned above, and performing row normalization on the image adjacency matrix to ensure the numerical stability of graph propagation, the final dense visual graph's image adjacency matrix is ​​obtained, represented by the following logic:

[0048]

[0049] in, The first in the image adjacency matrix Line 1 Column elements; This is a balance coefficient between spatial adjacency and semantic similarity, used to adjust the relative contributions of these two relationships in graph construction. In this embodiment, 0.6 is preferred; For the image adjacency matrix, the first... The sum of the elements in the row, The row normalization operation ensures that the sum of the adjacency weights of each node is 1, thus guaranteeing the numerical stability during graph convolution propagation. This indicates row normalization of the image adjacency matrix.

[0050] After the above construction process, the image adjacency matrix It not only characterizes the local spatial structure to distinguish the target from the background region, but also encodes global semantic similarity to distinguish the target from similar-looking distractors, providing rich neighborhood aggregation weights for subsequent structured feature enhancement.

[0051] S2.2, Constructing a sparse text graph ,in For text word sequence, This is the text adjacency matrix.

[0052] In this embodiment, a sparse text graph is constructed. To explicitly highlight key semantic modifications in the text. Specifically, given a sequence of words in the text... ,in This represents the total number of text terms. A text adjacency matrix is ​​constructed by comprehensively considering syntactic prior dependencies and self-attention semantic affinity. The process is as follows: (1) Generate prior syntactic dependency mask: By introducing a pre-trained syntactic parser, a directed dependency tree is generated for the text lexical sequence and converted into a binary syntactic dependency mask. ,in Representing text terms and There are direct syntactic modification relationships between them, for example, "dark" modifies "car" and "bottomright" modifies "corner". In this embodiment, the generated prior syntactic dependency mask provides explicit syntactic skeleton constraints for the sparse text graph.

[0053] (2) Assigning part-of-speech importance weights: In order to highlight nouns and adjectives that carry core semantics, this embodiment assigns corresponding prior weights based on different part-of-speech tags. This is used to amplify the role of key semantic nodes in adjacency matrix calculation, enabling nouns (such as "car") and modifiers (such as "dark") to obtain higher connection strength in sparse text graphs, thereby better preserving the core directional information in the text.

[0054] In a preferred embodiment, the prior weights for noun tags are assigned as follows: The prior weights of adjectives or modifiers are assigned as follows: The prior weights of the remaining part-of-speech tags are assigned as follows: .

[0055] (3) Constructing a text adjacency matrix that integrates syntactic dependency and semantic affinity: In order to preserve the syntactic skeleton while taking into account the contextual semantic affinity, two independent projection matrices are introduced. Calculate the scaled dot product similarity between text terms, and then directly add syntactic bias terms on top of this. The final text adjacency matrix is ​​as follows:

[0056] in, For text adjacency matrix The Middle Line 1 Column elements, Product term This is a part-of-speech importance weight matrix, used to amplify the connection strength of keyword elements in the final adjacency; Describes the first... Softmax normalization is performed on the rows, with normalization along the column direction (i.e., ... (dimension) to perform, so that for a fixed number of rows The sum of the output values ​​in all columns is 1, that is... This operation converts the original similarity score of each line into a probability distribution, reflecting the relative semantic affinity strength between each text word and all other text words. These are the query projection matrix and key projection matrix for the text modality, respectively, used to map lexical features to the attention computation space. ; This is a text temperature coefficient used to control the sharpness of the semantic similarity distribution; The syntactic bias coefficient is used to control the strength of the influence of the syntactic prior in the Softmax logarithmic space, and is preferably... ; For the syntactic dependency mask matrix, the first Line 1 The elements of the column.

[0057] Through the above construction process, the sparse text graph can effectively filter out redundant information such as interjections, and accurately focus on key directional semantic structures such as "dark color (attribute) + car (target)" and "bottom right corner (location)," providing clear text structure anchors for subsequent cross-modal structure-level matching.

[0058] S2.3, perform node-level matching between candidate regions in the dense visual image and keywords in the sparse text image. By calculating the dot product similarity between visual and text words, establish the basic correspondence between visual and text words in the original feature space and generate a node-level matching matrix.

[0059] in, For the node-level matching matrix, the first... Line 1 The elements of the column represent text terms. With visual lexical The correlation; For; among them, Describes the first... Row along column direction (i.e. Normalize the dimensions to make the visual lexical units more uniform. The sum of its matching scores with all text tokens is 1. This ensures the node-level matching matrix... Each row constitutes an effective probability distribution, which facilitates subsequent weighted fusion with the structural-level matching score; All are learnable projection matrices. The above node-level matching yields basic visual-text alignment results, such as matching vehicle regions in an image with "car", dark regions with "dark", and the lower right region with "bottomright". Based on this, the referential segmentation network framework further performs structural-level matching.

[0060] S2.4, perform neighborhood aggregation on the adjacency matrices corresponding to the dense visual image and the sparse text image respectively, construct image structure features and text structure features respectively, and use the above structure features to perform structure-level matching to generate a structure-level matching matrix.

[0061] In this embodiment, structure-level matching does not only determine whether a single local image region resembles the target, but also whether the local structure in the image is consistent with the semantic structure in the text description. Specifically, it first utilizes the image adjacency matrix. Neighborhood aggregation is obtained Furthermore, center-neighborhood difference features are constructed to enhance structure identification, using the following logical representation:

[0062]

[0063] in, This is a neighborhood aggregation feature. This represents image structural features, and also center-neighborhood difference features. It is a two-layer perceptron with an input dimension of . (Corresponding to the concatenated original features and difference features), the hidden layer dimension is... The output dimension is The middle layer uses the GELU activation function, and each layer is followed by LayerNorm to map the center-neighborhood difference information into enhanced structural features; similarly, text structural features are constructed. .

[0064] It should be noted that the MLPs used in this embodiment are all basic general-purpose modules in the field of deep learning, and the standard structure of the perceptron is directly used as the network architecture of the corresponding module.

[0065] This embodiment utilizes an image adjacency matrix. Neighborhood aggregation is obtained The process is implemented autonomously within the Transformer framework.

[0066] In this embodiment, the process of obtaining the neighborhood aggregation using the image adjacency matrix is ​​described. This is the core operation of Graph Neural Networks (GNNs)—neighborhood message passing. This process is a lightweight graph message passing mechanism implemented autonomously within the Transformer framework. Its neighborhood aggregation is equivalent to the aggregation step of a single-layer graph convolution, but the aggregation weights come from a custom fusion adjacency matrix defined in S2.1. After aggregation, center-neighborhood difference features are further constructed. By explicitly encoding the uniqueness of a node relative to its context, this differential feature is concatenated with the original feature and input into a two-layer perceptron. The final output image structure features Used for subsequent structural-level matching. The entire GNN message passing and difference mapping process is autonomously completed within the Transformer backbone network using matrix operations and standard MLP methods.

[0067] Furthermore, structural-level matching is then performed in the enhanced feature space, and the structural-level matching matrix is ​​calculated:

[0068] in, For the structure-level matching matrix, the first... Line 1 Column elements, Each is an independent projection matrix. The structural matching matrix obtained from the above formula It can effectively distinguish distractors that are similar in appearance but different in context. For example, only areas that simultaneously satisfy structural relationships such as "car", "dark color", "on the road" and "near the lower right corner" are more likely to be the true target. Structural matching can effectively reduce false matching caused by densely similar targets.

[0069] S3, the alignment-guided lexical pruning module takes visual lexical units and textual lexical units as input, calculates the multimodal alignment score through the alignment-guided lexical pruning module, selects visual lexical units related to the text description to retain based on the score, and prunes irrelevant ones as background lexical units, reconstructs the pruned background lexical units, and forms a new visual lexical unit sequence.

[0070] like Figure 1 and Figure 3 As shown, the alignment information obtained from the two-image semantic disambiguation is sent to the alignment-guided lexical pruning module. The alignment-guided lexical pruning module calculates a multimodal alignment score based on visual-text alignment, structural centrality, and self-salience. Based on the multimodal alignment score, it selects key visual lexical units that are highly relevant to the text description. Irrelevant visual lexical units are pruned as background lexical units, thereby suppressing a large number of irrelevant background lexical units and achieving visual information filtering.

[0071] Furthermore, for the pruned background lexical units, the alignment-guided lexical pruning module further compresses and preserves them through a background-aware lexical reconstruction mechanism, and splices them with the preserved useful visual lexical units to form a new visual lexical sequence. This sequence is used to reduce background noise while preserving necessary scene context information, resulting in a refined visual representation.

[0072] Lexical pruning is primarily used to reduce redundant information in visual models. For Transformer-type models, images are typically divided into a large number of visual lexical units, with different units corresponding to different image regions. Pruning can filter out visual lexical units that are irrelevant to the task or contribute little, thereby reducing background noise and computational burden. Existing visual compression methods typically reduce redundant visual information through downsampling, hierarchical representation, or dynamic token selection. However, if pruning is performed solely based on visual saliency, small target regions in drone images may be mistakenly removed. Specifically, S3 in this embodiment includes the following: S3.1, Construction of multimodal aligned fractions: like Figure 1 and Figure 3 As shown, the referential segmentation network framework calculates the multimodal alignment score (MAS) based on visual lexicon V and text lexicon T. The MAS consists of three parts: visual-text alignment score, structural centrality score, and self-salience score.

[0073] The visual-text alignment score is used to determine whether a visual region is related to a text description. Specifically, it is based on the node-level matching matrix generated in the dual-image semantic disambiguation module. For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the visual-text alignment score. As shown in the following formula:

[0074] in, This indicates that for the node-level matching matrix The Middle After sorting all elements of a row in descending order of value, select the column indices (i.e., text term indices) corresponding to the k largest elements. The set of ), k is preferably 3; the visual-text alignment score calculated by the above formula can measure the basic semantic relevance between the visual region and the individual word in the text description.

[0075] The structure centrality score is used to determine the importance of the visual region within the graph structure. Specifically, it is based on the structure-level matching matrix generated in the dual-graph semantic disambiguation module. For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the structure centrality score. As shown in the following formula:

[0076] in, This indicates that for the structural matching matrix The Middle After sorting all elements of a row in descending order of value, select the column indices (i.e., text term indices) corresponding to the k largest elements. A set of ).

[0077] In this embodiment, due to the structural-level matching matrix Based on graph-enhanced features (i.e., center-neighborhood difference features) Therefore, the structural centrality score calculated based on the structural matching matrix not only reflects the degree of visual-text alignment, but also implicitly encodes the contextual importance of nodes in the graph structure, and can be regarded as a measure of structural centrality.

[0078] The self-salience score is used to determine whether the visual region itself has obvious visual features, using only the original visual lexical units. With global text representation Calculating cross-modal significance, independent of graph structure, can be achieved using the following logical representation:

[0079] in, Visual lexical elements The corresponding self-significance score; It is represented as a two-layer perceptron with an input dimension of 2d, a hidden layer dimension of d, and an output dimension of 1. The GELU activation function is used in between, and each layer is followed by LayerNorm. Indicates visual lexical elements With global text representation Concatenation is performed along the channel dimension to obtain the dimension as follows. The combined feature vectors, after concatenation, are used as... The input is used to calculate the self-salience score of visual lexical units; The global text representation is obtained by calculating the mean of the text words in the text word sequence, using the following logical representation:

[0080] In this embodiment, the self-salience score can capture whether the visual region itself has prominent visual features (such as obvious color, edge intensity, etc.) and serves as a supplement to the first two scores.

[0081] Furthermore, the three scores are subjected to Min-Max normalization to normalize the scores to the nearest integer. Then, with the original visual lexical units splicing, inputting into a multilayer perceptron The final multimodal alignment score (MAS) is obtained by activating the Sigmoid function, as shown in the following equation:

[0082] in, Represents primitive visual lexical units The corresponding multimodal alignment score, ; It is a two-layer perceptron with an input layer dimension of . The hidden layer dimension is The output layer dimension is The intermediate layer uses the GELU activation function; The Sigmoid activation function maps the output to... Interval.

[0083] In this embodiment, by concatenating the three scores and the original visual lexical units, the model adaptively integrates three complementary cues, providing a joint discrimination criterion for subsequent lexical unit pruning. A higher multimodal alignment score (MAS) indicates a stronger corresponding visual lexical unit. The more likely it is to be related to the target it refers to.

[0084] S3.2, Alignment-guided lexical selection: like Figure 3 As shown, the referential segmentation network framework performs Top-K selection based on the multimodal alignment score (MAS) to generate a binary pruning mask. Visual terms with a mask value of 1 are retained, while visual terms with a mask value of 0 are considered background or low-relevance regions and are pruned.

[0085] Specifically, firstly, the multimodal alignment scores corresponding to all visual morphemes are sorted in descending order from high to low, and a retention ratio is set to [value missing]. The number of visual lexical units retained is , The total number of visual lexical units; Then, select the previous One visual terminology is selected as the retained terminology, and the remaining visual terms are selected as the pruned terms. A binary pruning mask is generated based on the selection results. ,like This indicates that the visual lexical units corresponding to the mask are preserved. This indicates that the visual lexical units corresponding to the mask are treated as background or low-relevance regions and are pruned.

[0086] Then, the binary pruning mask With the original visual lexical sequence Element-wise multiplication is performed to solve for the Hadamard product, thus completing the lexical pruning operation, represented as follows: ,in This represents the Hadamard product. In this embodiment, a binary pruning mask is first generated, and then pruning is completed through the Hadamard product. Through the above-mentioned lexical selection method, the model can reduce the interference of a large number of irrelevant background lexical terms on the semantics of the text, making the key referential information more concentrated.

[0087] It should be noted that in the fields of natural language processing and computer vision, Top-K lexical selection based on importance score ranking is a widely used pruning paradigm. For example, methods such as DynamicViT and A-ViT employ similar strategies to dynamically prune lexical units in the visual Transformer. The core of S3.2 in this embodiment lies in the fact that the Multimodal Alignment Score (MAS) does not rely solely on visual appearance information, but rather integrates the node-level matching score and structure-level matching score output by the dual-graph semantic disambiguation module. This allows the pruning decision to be guided by both cross-modal semantic alignment results and graph structural context information, rather than solely relying on visual saliency.

[0088] S3.3, Background-Aware Lexical Reconstruction: like Figure 3As shown, the proposed denotation segmentation network framework does not directly discard pruned visual lexical units, but instead uses a background-aware lexical unit reconstruction method for aggregation: Specifically, for visual lexical units with a binary pruning mask of 0, background lexical units are first generated using Global Average Pooling (GAP). Visual lexical units with a mask of 1 are retained as useful visual lexical units. The two are then combined to form a new visual word sequence. , used for subsequent segmentation and decoding; among which, This represents the reconstructed visual word sequence. Representing background-aware lexical units, This indicates that visual terms related to the target should be retained.

[0089] By using the above-mentioned word aggregation method, the model reduces background redundancy while retaining the necessary global contextual information.

[0090] S4 inputs the new visual word sequence and the original text word sequence into the segmentation decoder to generate a segmentation mask for the referential target.

[0091] like Figure 1 As shown, the output layer deploys a segmentation decoder; the input of the segmentation decoder contains a new sequence of visual lexical terms. and text word sequence The segmentation decoder uses text word sequences. For querying, a new visual lexical sequence Cross-modal interaction and feature decoding are performed, and textual semantics and visual spatial information are fused to generate the final target segmentation mask.

[0092] In this embodiment, the decoder uses the newly reconstructed visual lexical sequence as an example. As input, the sequence retains both target-related discriminative visual terms. Furthermore, it perceives word units through background information. Global contextual information is preserved; on the other hand, the decoder uses the text word sequence extracted by the text encoder. As a cross-modal conditionally guided information decoder, it can make full use of the simplified visual features and complete textual semantic description to achieve accurate target segmentation in aerial photography scenes with dense targets and complex backgrounds.

[0093] In a preferred embodiment, the model training process for the denotation segmentation network framework described in this embodiment is as follows: (1) Training dataset and preprocessing: This embodiment uses publicly available UAV referencing and segmentation datasets, such as the RIS-LAD dataset, for model training and validation. During model training, the input UAV aerial images are uniformly adjusted to a fixed spatial resolution, such as 608×608 pixels. The input images are divided into multiple fixed-size square regions according to a grid. In this embodiment, the image patch size is set to 16×16 pixels. Each image patch is processed by a visual encoder and mapped to a corresponding visual word, which serves as the basic unit for subsequent network processing.

[0094] For the input natural language description, this embodiment sets the maximum text token length to 20; if the number of text tokens is less than 20, padding is used to complete the length; if it exceeds 20, truncation is performed.

[0095] (2) Network initialization and training hyperparameters: In this embodiment, the visual encoder is loaded onto ImageNet. Swin pre-trained on 21K S-vision backbone network; the text encoder adopts BEiT3. The B model has a built-in language encoder, and its pre-trained weights for multimodal tasks are loaded.

[0096] The model hyperparameters are configured as follows: This embodiment uses the Adam optimizer, with a training period of 20 epochs, and the initial learning rate of the overall framework is set to... The pre-trained BEiT3-B language encoder uses The learning rate is fine-tuned; the output layer parameters of the visual encoder (each layer of the Swing-S) and the language encoder are used... Fine-grained updates are performed using a smaller learning rate; a multi-step learning rate decay strategy is adopted, with decay occurring in the 15th and 20th epochs, weight decay set to 0, and gradient clipping norm set to 0.15.

[0097] (3) Design the loss function: This embodiment uses a multi-task joint loss function for end-to-end training, and the model's total loss function is... It consists of a weighted sum of the segmentation loss and two auxiliary ranking losses, and can be represented logically as follows:

[0098] in, The segmentation loss is used to constrain the consistency between the predicted segmentation mask and the manually labeled ground truth mask. In this embodiment, the segmentation loss is... A combination of binary cross-entropy loss and Dice loss is used; The saliency ranking loss is used to drive the model during training to improve the multimodal alignment score (MAS) of visual words related to the target, while reducing the MAS score of background words unrelated to the target. As shown in the following formula:

[0099] in, For target word MAS score, background word MAS score, For sorting interval, Given a randomly sampled set of target-background word pairs, the loss function is based on the general Margin Ranking Loss design. By pushing the MAS score of target words higher than that of background words, the model gradually learns to distinguish between the target and the background during training. The interference ranking loss is used to emphasize difficult-to-distinguish interference objects that are spatially adjacent to the target. As shown in the following formula:

[0100] in, This is a set of target-difficult distractor word pairs sampled using the Online Hard Example Mining (OHEM) strategy. These distractors are spatially adjacent to the target and highly similar to the target in terms of feature appearance. For the corresponding sorting interval, this loss function in Building upon this foundation, we further focus on spatially adjacent, difficult-to-distinguish interfering objects, explicitly enhancing the model's ability to distinguish densely similar targets; , These are the balance coefficients corresponding to the auxiliary sorting loss.

[0101] (4) Training execution process: In this embodiment, the entire referential segmentation network framework is jointly optimized as a whole. The specific process is as follows: Forward propagation: The drone aerial images and natural language descriptions are input to the referential segmentation network framework. The original visual and textual words are extracted by the visual encoder and text encoder, respectively. The dual-graph semantic disambiguation module performs dual-graph construction, node-level matching, and structure-level matching. Then, the alignment-guided word pruning module calculates the multimodal alignment score (MAS) and completes word pruning and word sequence reconstruction. Finally, the segmentation decoder outputs the predicted referential target segmentation mask.

[0102] Loss calculation: Calculate the main segmentation loss based on the predicted mask and the ground truth label. Simultaneously, the significance ranking loss is calculated based on the fractional MAS. And interference sorting loss The weighted average yields the total loss. .

[0103] Backpropagation and parameter update: The gradient of the total loss with respect to each network parameter is calculated through backpropagation, and the network weights are updated using the Adam optimizer. The learning rate is adjusted according to a multi-step decay strategy at the 15th and 20th epochs.

[0104] Model saving: After a preset 20 training cycles, the weights of the network model that performs best on the validation set are saved as the final model for testing.

[0105] Furthermore, during the testing phase, only the forward inference process of the aforementioned network is executed, without involving gradient calculation and parameter updates. This enables accurate segmentation of specified targets in UAV aerial images based on natural language descriptions, making it particularly suitable for low-altitude aerial photography scenarios with dense targets, complex backgrounds, and obvious small targets.

[0106] like Figure 4 As shown, this embodiment selects four representative drone aerial photography scenarios to visualize the intermediate outputs and final results of the denotation segmentation network framework: The referential text specifically refers to the natural language description of the input, which specifies the target object to be segmented and its attributes and spatial location relationships. For example, "the dark colored car is driven on the road positioned near the bottom right corner of the image" contains category words (car), attribute words (darkcolored), and location words (bottom right corner), providing cross-modal semantic guidance for the model.

[0107] Aerial images specifically refer to images taken by low-altitude drones, characterized by small target scale, wide field of view, complex background, and dense target distribution. They correspond to the basic input for the aerial image segmentation task targeted in this embodiment.

[0108] The true mask specifically refers to a manually annotated pixel-level segmentation mask, representing the true target region corresponding to the text being referred to, and is used as the segmentation loss during model training. The supervisory signal is used to constrain the consistency between the predicted mask and the true mask.

[0109] Attention heatmaps specifically refer to the visualization results of MAS scores upsampled to the original image resolution via bilinear interpolation. Regions with darker / warmer color responses indicate higher MAS scores assigned to them by the model, meaning a stronger semantic relevance between the visual term and the referential text. Attention heatmaps accurately focus on the referential target (such as vehicles or pedestrians) while maintaining low responses to background regions and similar distractions, indicating that MAS scores effectively capture cross-modal alignment cues.

[0110] The pruned lexical graph specifically refers to the visualization of the spatial distribution of retained and pruned lexical units after Top-K selection by the AGTP module. Green dots represent retained visual lexical units (mask of 1), mainly distributed in the target referent and its key context region; red dots represent pruned visual lexical units (mask of 0), mainly distributed in background redundant regions. After pruning, the model retains only approximately... The visual lexical units are such that a large number of irrelevant background lexical units are effectively suppressed, while target-related lexical units are completely preserved, indicating that the alignment-guided lexical unit selection strategy can effectively compress visual information redundancy while maintaining segmentation accuracy.

[0111] Figure 4 The four examples shown cover four typical challenging scenarios: dense parking lots, small-scale traffic participants, complex road backgrounds, and nighttime roads. The results demonstrate that the proposed method can accurately locate the referential target and retain key visual terms under conditions of varying lighting, dense targets, small target scale, and complex backgrounds, verifying the effectiveness and robustness of the DGSD and AGTP modules in different aerial photography scenarios.

[0112] In a preferred embodiment, to verify the effectiveness of the UAV-based image segmentation method described in this embodiment, a comprehensive comparative experiment was conducted on the publicly available UAV aerial photography segmentation dataset RIS-LAD. The RIS-LAD dataset is specifically designed for low-altitude UAV aerial photography scenarios, and its image samples have characteristics such as wide field of view, small target scale, dense arrangement of similar targets, and complex and diverse backgrounds, which can fully reflect the visual challenges in real low-altitude aerial photography tasks.

[0113] This embodiment is compared with existing mainstream indexing image segmentation methods. The selected comparison methods cover different types of technical approaches, including indexing segmentation methods for natural scenes such as LAVT, ASDA, and VATEX, indexing segmentation methods for remote sensing images such as LGCE, FIANet, RMSIN, RSRefSeg, and CADFormer, and the dedicated method SAARN for low-altitude UAV scenarios. Among them, SAARN is currently recognized as one of the best performing methods in this field.

[0114] The comparative experiments adopted a commonly used evaluation index system in the field of image segmentation, including P@0.5, P@0.6, P@0.7, P@0.8, P@0.9, oIoU, and mIoU. P@X represents the proportion of correctly predicted samples out of all samples when the IoU threshold is greater than or equal to X. In this embodiment, 0.5, 0.6, 0.7, 0.8, and 0.9 are used as evaluation indicators, respectively. oIoU represents the overall IoU, calculated by summing the predicted results and the actual results of all test samples to obtain the global union-intersection ratio, reflecting the overall segmentation accuracy of the model on the entire dataset. mIoU represents the average IoU, calculated by first calculating the IoU of each test sample and then averaging it across all samples, reflecting the average segmentation performance of the model on a single sample.

[0115] like Figure 5 As shown, Val (Validation Set) is used for hyperparameter tuning and intermediate result validation during model training, while Test (Test Set) is used to finally evaluate the model's generalization performance. The method described in this embodiment (corresponding to...) Figure 5 Ours achieved the best results on all the above evaluation metrics, significantly surpassing the current best UAV-specific method SAARN. This fully demonstrates that the method described in this embodiment effectively solves the ambiguity problem of densely similar targets through the dual-graph semantic disambiguation module, and at the same time alleviates the interference caused by the mismatch between image and text information density through the alignment-guided lexical pruning module, significantly improving the accuracy and robustness of UAV aerial image referencing segmentation.

[0116] Example 2 An apparatus includes a memory and a processor, the memory being used to store a program that supports the processor in executing the UAV-referencing image segmentation method of Embodiment 1, the processor being configured to execute the program stored in the memory.

[0117] Example 3 A storage medium storing a computer program, which, when executed by a processor, performs the steps of the UAV-referencing image segmentation method in Embodiment 1.

[0118] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-based system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0119] For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit a program for use in or in conjunction with an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CD-ROM). Additionally, a computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0120] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] In the description of this specification, references to terms such as "in a preferred embodiment," "preferred," "in this embodiment," "specific," or "furthermore," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting images of unmanned aerial vehicles (UAVs), characterized in that, include: The input consists of drone aerial images and natural language descriptions, and visual and textual lexical units are extracted by a visual encoder and a text encoder, respectively. Using visual lexical units and textual lexical units as input, dense visual graphs and sparse text graphs are constructed respectively. Node-level matching and structure-level matching are then performed on the dense visual graphs and sparse text graphs to generate node-level matching matrices and structure-level matching matrices, thus establishing the correspondence between image structure and language structure. Using visual lexical units and textual lexical units as input, multimodal alignment scores are calculated. Based on the scores, visual lexical units that are relevant to the natural language description are selected for retention, while irrelevant visual lexical units are pruned as background lexical units. The pruned visual lexical units are reconstructed and concatenated with the retained visual lexical units to form a new visual lexical unit sequence. The new visual lexical sequence is input into the segmentation decoder along with the text lexical sequence to generate a segmentation mask that refers to the target.

2. The UAV-based image segmentation method according to claim 1, characterized in that, The dense visual graph includes visual word sequences. and image adjacency matrix ; where each word element The corresponding two-dimensional spatial coordinates on the feature map , , , The total number of visual lexical units. For the set of real numbers, The feature dimensions of each visual lexical unit; Constructing the image adjacency matrix The process is as follows: First, select each visual lexical unit based on Euclidean distance. The nearest neighbor words are used to generate a binary space mask as follows: in, Represents the first in the binary space mask Line 1 Column elements, ; Representation and Lexical Elements The closest in spatial Euclidean distance A set of indices for each term element. and They are respectively word elements and Spatial coordinates in the feature map Indicates Euclidean distance; Next, a learnable projection matrix is ​​used. Visual lexical units are mapped to a similarity space, and the image temperature coefficient is used as a reference. Controlling the sharpness of the distribution, the semantic similarity matrix is ​​calculated as follows: in, The semantic similarity matrix is ​​the first... Line 1 Column elements, Scaling factor For image temperature coefficient, The projected weight matrix is ​​a learnable matrix. Describes the first... Perform Softmax normalization along the column direction; Finally, the balance coefficient is introduced. By fusing the binary space mask and the semantic similarity matrix, and performing row normalization on the image adjacency matrix, the final image adjacency matrix is ​​obtained, which is represented by the following logic: in, The first in the image adjacency matrix Line 1 The elements of the columns, after row normalization of the image adjacency matrix, are as follows: , For the image adjacency matrix, the first... The sum of the elements in the row, .

3. The UAV-based image segmentation method according to claim 1, characterized in that, The sparse text graph includes a sequence of text terms. and text adjacency matrix ; Constructing a text adjacency matrix The process is as follows: First, a directed dependency tree is generated for the text word sequence using a pre-trained syntactic parser, and then converted into a binary syntactic dependency mask. ,in Representing text terms and There is a direct grammatical modification relationship between them. The total number of text words; Next, prior weights are assigned based on different part-of-speech tags. ; Finally, through independent projection matrices Calculate the scaled dot product similarity between text terms and directly sum the syntactic bias terms. The text adjacency matrix is ​​as follows: in, For text adjacency matrix The Middle Line 1 Column elements, , Text words respectively , The corresponding prior weights, Scaling factor This refers to the temperature coefficient of the text. The syntactic bias coefficient. For the syntactic dependency mask matrix, the first Line 1 Column elements, Describes the first... Perform Softmax normalization along the row and column directions.

4. The UAV-based image segmentation method according to claim 1, characterized in that, The node-level matching matrix is ​​generated by calculating the dot product similarity between visual and textual words, and is represented by the following logic: in, For the node-level matching matrix, the first... Line 1 The elements of the column represent text terms. With visual lexical The correlation, All are learnable projection matrices. Scaling factor Describes the first... Perform Softmax normalization along the row and column directions.

5. The UAV-based image segmentation method according to claim 1, characterized in that, The structure-level matching matrix is ​​generated by performing neighborhood aggregation on the adjacency matrices corresponding to the dense visual image and the sparse text image, respectively, to construct image structure features and text structure features. Structure-level matching is then performed using these image structure features and text structure features. The specific process is as follows: Using image adjacency matrix Neighborhood aggregation is obtained And construct the center-neighborhood difference feature, using the following logical representation: in, This is a neighborhood aggregation feature. This represents image structural features, and also center-neighborhood difference features. It is a two-layer perceptron with an input dimension of . The hidden layer dimension is The output dimension is Similarly, text structure features are constructed. Next, structural-level matching is performed in the enhanced feature space, and the structural-level matching matrix is ​​calculated: in, For the structure-level matching matrix, the first... Line 1 Column elements, Each is an independent projection matrix. , This is the scaling factor.

6. The UAV-based image segmentation method according to claim 1, characterized in that, The multimodal alignment score consists of a visual-text alignment score, a structure centrality score, and an autosalience score. The calculation process for the multimodal alignment score is as follows: Based on node-level matching matrix For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the visual-text alignment score. ,in For the node-level matching matrix, the first... Line 1 Column elements, This indicates that for the node-level matching matrix The Middle The set of column indices corresponding to the k largest elements of a row, sorted from largest to smallest. Based on structural matching matrix For each visual lexical unit Take the one with the highest similarity to the text word units. The mean of the values ​​is used as the structure centrality score. ,in For the structure-level matching matrix, the first... Line 1 Column elements, This indicates that for the structural matching matrix The Middle The set of column indices corresponding to the k largest elements of a row, sorted from largest to smallest. Utilizing visual lexical elements With global text representation Calculate the self-significance score ,in Visual lexical elements The corresponding self-significance score; It is represented as a two-layer perceptron, with an input dimension of 2d, a hidden layer dimension of d, and an output dimension of 1; Indicates visual lexical elements With global text representation The components are stitched together along the channel dimension. The global text representation is obtained by calculating the mean of the text words in the text word sequence; The visual-text alignment score, structure centrality score, and self-salience score were normalized using Min-Max and compared with visual lexical units. splicing, inputting into a multilayer perceptron The multimodal alignment score is obtained by activating the Sigmoid function as follows: in, Represents primitive visual lexical units The corresponding multimodal alignment score, ; It is a two-layer perceptron with an input layer dimension of . The hidden layer dimension is The output layer dimension is , This is the Sigmoid function.

7. The UAV-based image segmentation method according to claim 6, characterized in that, Top-K selection is performed based on the multimodal alignment score to generate a binary pruning mask; wherein visual words with a mask of 1 are retained, and visual words with a mask of 0 are pruned.

8. The UAV-based image segmentation method according to claim 7, characterized in that, For visual words with a mask of 0, background words are first generated through global average pooling. And concatenate it with the visual lexical with a mask of 1 to form a new visual lexical sequence. ,in This indicates that visual lexical units with a mask of 1 are retained.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the UAV-referencing image segmentation method according to any one of claims 1 to 8, wherein the processor is configured to execute the program stored in the memory.

10. A storage medium storing a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the UAV-referencing image segmentation method according to any one of claims 1 to 8.

Images