Multi-type audio guided unmanned aerial vehicle remote sensing image segmentation method and system based on fine semantic embedding
By constructing a multi-type audio-guided UAV remote sensing image segmentation model with refined semantic embedding, the problem of insufficient semantic alignment between audio and image in remote sensing scenarios is solved, achieving high-precision voice command-guided UAV remote sensing image segmentation and improving intelligent interactive processing capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV OF TECH
- Filing Date
- 2025-11-25
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies lack audio and image semantic alignment mechanisms in remote sensing scenarios, resulting in insufficient segmentation accuracy, inadequate modeling of audio command semantic information, lack of fine-grained semantic embedding, and the absence of a complete speech or audio signal-guided UAV remote sensing image segmentation technology system.
A multi-type audio-guided UAV remote sensing image segmentation model based on fine semantic embedding is constructed, including image feature encoding, audio feature encoding, fine semantic embedding, audio semantic request, and mask feature decoding modules. Through multi-scale sampling, cross-modal fusion, and attention mechanisms, fine alignment and semantic embedding of audio and image features are achieved, and the model is trained to output a segmentation mask.
It enhances the intelligent interactive processing capabilities of UAV remote sensing images, improves segmentation accuracy and semantic overlap, and enables efficient and intelligent operation via voice commands.
Smart Images
Figure CN121640053B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of intelligent remote sensing image processing and multimodal artificial intelligence, and in particular relates to a method and system for segmenting UAV remote sensing images based on fine semantic embedding and multi-type audio guidance. Background Technology
[0002] With the rapid development of multimodal learning, visual tasks based on speech or natural audio have gradually attracted attention. Existing research mainly focuses on audio-guided image segmentation in natural scenes, achieving semantic segmentation of target regions in images by aligning natural audio with images. However, in remote sensing applications, due to long imaging distances and complex environments, traditional natural sound cues relying on near-field sound sources are difficult to apply directly. Furthermore, in the actual operation of remote sensing platforms such as drones, operators often need to use both hands to control the flight equipment or other mission devices, making interaction via keyboard or text input impossible. In contrast, voice commands, as a natural and efficient input method, are more direct and convenient.
[0003] While existing methods have made progress in cross-modal alignment and semantic segmentation, they still face the following shortcomings in remote sensing scenarios: First, the lack of an audio-image semantic alignment mechanism tailored to the characteristics of remote sensing images leads to insufficient segmentation accuracy; second, the semantic information in audio commands is not fully modeled in existing models, lacking fine-grained semantic embedding representation; and third, a complete technical system capable of directly using speech or audio signals to guide UAV remote sensing image segmentation has not yet been established. Therefore, developing a multi-type audio-guided UAV remote sensing image segmentation method based on fine semantic embedding is of great significance for improving the intelligent interactive processing capabilities and application value of UAV remote sensing images. Summary of the Invention
[0004] This invention provides a multi-type audio-guided UAV remote sensing image segmentation method and system based on refined semantic embedding, which enhances the intelligent interactive processing capabilities of UAV remote sensing images.
[0005] This invention provides a multi-type audio-guided UAV remote sensing image segmentation method based on refined semantic embedding. The method includes: collecting UAV remote sensing images and related audio and speech data to construct a multi-type audio UAV remote sensing image segmentation dataset; and constructing a multi-type audio-guided UAV remote sensing image segmentation model based on refined semantic embedding. The model includes: a UAV remote sensing image feature encoding module, an audio feature encoding module, a refined semantic embedding module, an audio semantic request module, and a mask feature decoding module. The image feature encoding module converts the remote sensing image into image features, the audio feature encoding module extracts original audio features, and the refined semantic embedding module converts the audio into image features. The original features and the image features are fused across modally to obtain cross-modal mask features; the audio semantic request is used to convert the original audio features into audio requests through an attention mechanism and mix them with learnable requests to obtain a hybrid request; the mask feature decoding module is used to decode the hybrid request into an output request and fuse the output request with the cross-modal mask features to obtain a segmentation mask; the multi-type audio-guided UAV remote sensing image segmentation model is trained on the multi-type audio UAV remote sensing image segmentation dataset until the training completion condition is met. The trained multi-type audio-guided UAV remote sensing image segmentation model can output a segmentation mask for ambient sound or speech based on the input UAV remote sensing image with ambient sound or speech.
[0006] In some implementations, converting the remote sensing image into image features includes: performing multi-scale sampling on the input UAV remote sensing image to obtain multi-scale image features.
[0007] In some implementations, the fine semantic embedding module includes an adaptive modal fusion module, a multi-scale attention module, and an audio-visual hybridization module. The adaptive modal fusion module is used to fuse the original audio features with image features at various scales to obtain fused features. The multi-scale attention module is used to convert the fused features into mask features through an attention mechanism. The audio-visual hybridization module is used to perform cross-modal attention calculation on the original audio features to obtain the cross-modal mask features.
[0008] In some embodiments, the step of multi-scale fusion of the original audio features with image features of various sizes to obtain fused features includes: linearly mapping the original audio features to obtain compressed audio features; mapping the compressed audio features to image feature spaces sampled at different scales to obtain compressed projection features at different scales; calculating the average value of the original audio features in the time dimension to obtain global audio features; linearly transforming the global audio features to obtain global projection features; processing the compressed projection audio features using the Sigmoid function to obtain a gating graph; and fusing the compressed projection features, the image features, and the global projection features weighted by the gating graph to obtain fused features.
[0009] In some implementations, the step of converting fused features into mask features through an attention mechanism includes: extracting sub-fused features of multiple resolutions from the fused features; flattening and concatenating each of the sub-fused features as input queries for the Transformer encoder; restoring the features output by the Transformer encoder to their original shape at the corresponding scale; extracting features with a resolution of 1 separately and upsampling them by 2 times to obtain sampled features; and fusing the sampled features with the original UAV remote sensing image to obtain mask features.
[0010] In some implementations, the step of performing cross-modal attention calculation on the original audio features to obtain the cross-modal mask features includes: performing cross-modal attention calculation on the original audio features and the mask features through a channel attention-based design, learning a set of weights, and thus obtaining the cross-modal mask features.
[0011] In some implementations, the step of converting the original audio features into an audio request and mixing it with a learnable request to obtain a hybrid request via an attention mechanism includes: constructing an initial request, using the initial request as a query, inputting the original audio features as keys and values into a request generator in the attention model to obtain an audio request; and fusing the audio request with the learnable request to obtain the hybrid request.
[0012] In some implementations, decoding the hybrid request into an output request and fusing the output request with the cross-modal mask features to obtain a segmentation mask includes: using the hybrid request as an input query, using multi-scale image features as key-value pairs, and obtaining an output request through a decoding process; fusing the output request with the cross-modal mask features, integrating channel information through a multi-layer perceptron mechanism, and finally obtaining a segmentation mask through a fully connected layer.
[0013] In some implementations, training the multi-type audio-guided UAV remote sensing image segmentation model using the multi-type audio UAV remote sensing image guided segmentation dataset includes: sequentially inputting the training data from the multi-type audio-guided UAV remote sensing image segmentation dataset into the multi-type audio-guided UAV remote sensing image segmentation model; calculating a loss value based on the output of the multi-type audio-guided UAV remote sensing image segmentation model; calculating the loss value using a loss function that is a weighted sum of segmentation loss and mask request semantic consistency loss; backpropagating the loss value and adjusting the weights of the multi-type audio-guided UAV remote sensing image segmentation model until the loss value of the multi-type audio-guided UAV remote sensing image segmentation model reaches its minimum; repeating the above process until the loss value continuously converges to a range less than a preset threshold.
[0014] A second aspect of this invention provides a multi-type audio-guided UAV remote sensing image segmentation system based on refined semantic embedding. This system includes: a data construction module for collecting UAV remote sensing images and related audio and speech data to construct a multi-type audio-guided UAV remote sensing image segmentation dataset; and a model construction module for constructing a multi-type audio-guided UAV remote sensing image segmentation model based on refined semantic embedding. The image feature encoding module converts the remote sensing image into image features, the audio feature encoding module extracts original audio features, and the refined semantic embedding module performs cross-modal fusion of the original audio features and the image features. The system obtains cross-modal mask features; the fine semantic embedding module is used to convert the original audio features into audio requests through an attention mechanism and mix them with learnable requests to obtain a hybrid request; the mask feature decoding module is used to decode the hybrid request into an output request and fuse the output request with the cross-modal mask features to obtain a segmentation mask; the training module is used to train the multi-type audio-guided UAV remote sensing image segmentation model using the multi-type audio UAV remote sensing image guided segmentation dataset. The trained multi-type audio-guided UAV remote sensing image segmentation model can output a segmentation mask for ambient sound or speech based on the input UAV remote sensing image with ambient sound or speech.
[0015] By training the segmentation model, the trained segmentation model can intelligently segment and manipulate UAV remote sensing images based on various types of input audio or input ambient sound, thereby improving the intelligent interactive processing capability of UAV remote sensing images. Moreover, by aligning audio with image features at different scales through fine semantic embedding, and by fully modeling the audio semantics and fine semantic embedding, the overlap between UAV remote sensing images and audio semantics is improved. Attached Figure Description
[0016] Figure 1A flowchart illustrating a multi-type audio-guided UAV remote sensing image segmentation method based on fine semantic embedding, provided in an embodiment of the present invention;
[0017] Figure 2 This is a structural diagram of a multi-type audio-guided UAV remote sensing image segmentation model provided in an embodiment of the present invention;
[0018] Figure 3 This is a schematic diagram of the structure of a multi-type audio-guided UAV remote sensing image segmentation system based on fine semantic embedding, provided in an embodiment of the present invention. Detailed Implementation
[0019] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this invention will not be described separately.
[0020] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.
[0021] Additionally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the following description, the terms "first," "second," etc., are used merely to distinguish different objects and do not indicate any similarity or connection between them. It should be understood that the directional descriptions such as "above," "below," "inside," and "outside" refer to the orientation under normal use conditions.
[0022] In the following specific implementation, the multi-type audio-guided UAV remote sensing image segmentation method based on fine semantic embedding is used to output a segmented image corresponding to the semantics of the input audio, thereby enabling more convenient and intelligent operation of UAV remote sensing images through voice. This segmentation method relies on a multi-type audio-guided UAV remote sensing image segmentation model based on fine semantic embedding. The structure, function and training method of this model are illustrated below with reference to various embodiments.
[0023] In some embodiments, please refer to Figure 1 , Figure 1This is a flowchart illustrating an audio-guided UAV remote sensing image segmentation method based on fine semantic embedding, as provided in an embodiment of the present invention. Figure 1 As shown, the audio-guided UAV remote sensing image segmentation method includes:
[0024] Step S101: Collect UAV remote sensing images and remote sensing-related audio and speech data to construct a multi-type audio UAV remote sensing image-guided segmentation dataset.
[0025] A multi-type audio UAV remote sensing image-guided segmentation dataset is constructed using labeled UAV remote sensing images and remote sensing-related audio and speech data. This dataset also includes semantically related masks of the remote sensing images to the audio or speech.
[0026] Step S102: Construct a multi-type audio-guided UAV remote sensing image segmentation model based on refined semantic embedding.
[0027] Among them, such as Figure 2 As shown, the multi-type audio-guided UAV remote sensing image segmentation model includes: a UAV remote sensing image feature encoding module, an audio feature encoding module, a fine semantic embedding module, an audio semantic request module, and a mask feature decoding module. The functions of these modules are explained in detail below.
[0028] The UAV remote sensing image feature encoding module is used to convert remote sensing images into image features. Specifically, the module performs multi-scale sampling on the input UAV remote sensing image and uses a pre-trained visual backbone model to obtain multi-scale image features. The elements in the multi-scale image features are Where i represents i times downsampling, for example, By obtaining image features at different scales through multi-scale sampling, we can take into account both the visual field and semantic content of the image, thereby extracting more information from the image.
[0029] The audio feature encoding module is used to extract the original audio features. Specifically, the audio feature encoding module uses the VGGish model to obtain the original audio features through audio feature sampling. For example, the VGGish model undergoes the following audio feature sampling process:
[0030] The first step is to use the short-time Fourier transform to divide the waveform in the time domain into short time frames and convert them into frequency domain representation.
[0031] The second step is to pass the obtained spectrum through a Log-Mel filter bank to obtain a two-dimensional feature map.
[0032] The third step is to segment the audio feature map according to a fixed duration of 0.96 seconds to obtain multiple temporal segments, and finally obtain the audio tensor.
[0033] Step 4: Input the data into the VGGish model to obtain the original audio features. .
[0034] The fine semantic embedding module is used to fuse audio features and image features across modalities to obtain cross-modal mask features. Since image features have different scales, semantic information also needs to be aligned and semantic information of different granularities needs to be embedded into the image information to improve the segmentation accuracy of semantically guided image segmentation. The specific process of fine semantic embedding is illustrated below with reference to the specific structure of the fine semantic embedding module.
[0035] The fine semantic embedding module includes an adaptive modality fusion module, a multi-scale attention module, and an audio-visual fusion module. The functions of each module are described below with examples. The adaptive modality fusion module is used to fuse the original audio features with image features at various scales to obtain fused features, thereby aligning the original audio features and the visual spaces of the image features at various scales. The specific process is as follows:
[0036] Step 1: Extract original audio features Compressed audio features are obtained by performing linear mapping. The formula can be expressed as follows: ,in It is a linear transformation.
[0037] Step 2: Extract the original audio features Compressed projection features of different scales are obtained by projecting them onto the image feature space sampled at different scales. To compress projection features Capable of using image features of various orders Alignment; simultaneously, calculate original audio features. The global audio features are obtained by averaging over the time dimension. Global audio features Perform a linear transformation to obtain global projection features .
[0038] Step 3: Apply the Sigmoid function to the global projection features. The gating graph G is obtained after processing, and the calculation formula is:
[0039]
[0040] Among them, Represents audio length;
[0041] The fused feature is obtained by fusing compressed projection features, image features, and global projection features weighted by a gated graph. The specific calculation formula is as follows:
[0042]
[0043] in This represents element-wise matrix multiplication.
[0044] The multi-scale attention module is used to transform fused features into mask features through an attention mechanism, thereby converting aligned audio features into semantically relevant mask features. The specific process is as follows:
[0045] The first step is to extract sub-fusion features of multiple resolutions from the fusion features. For example, extract features of three resolutions (i=1,2,3) from the fusion features, flatten and concatenate the respective fusion features as input queries for the Transformer encoder.
[0046] The second step is to restore the features output by the Transformer encoder to the original shape of the corresponding scale, and extract the features of the scale with a resolution of 1 separately and perform upsampling of the features by 2 times.
[0047] The third step is to fuse the upsampled features with the original UAV remote sensing image (the UAV remote sensing image from the visual backbone network at i=0) to obtain the mask features. .
[0048] It should be noted that the masking features mentioned above are only masking features based on semantic segmentation. It is also necessary to perform cross-modal fusion of semantics and UAV remote sensing images to obtain cross-modal masking features. This step needs to be implemented through an audio-visual fusion module. Specifically, the audio-visual fusion module adopts a channel attention-based design, using original audio features... With mask features Perform cross-modal attention calculations to learn a set of weights. The formula for calculating this weight is as follows:
[0049]
[0050] in Represents the dimension of feature encoding. This represents the number of heads receiving multi-head attention. The final fused mask features are obtained. .
[0051] The above provides an introduction to the structure and function of the fine semantic embedding module. It achieves the alignment of different compressed audio features and image features at different scales through projection, and also embeds semantics of different fine granularities into image features through modeling, thereby enabling mask features to perform more accurate segmentation of UAV remote sensing images.
[0052] The audio semantic request module uses an attention mechanism to transform the original audio features into audio requests and mix them with learnable requests to obtain a hybrid request. Specifically, it first constructs an initial request. and original audio features Using Q as the query, The audio request is input as a key and value into the request generator. Then, the audio request will be sent. With Learnable Requests The process involves merging the data to obtain a mixed request. This serves as the input to the Transformer decoder. The learning request is... This is a parameter matrix that can be automatically learned during model training; in this module, it is used to balance audio requests. A mixed request is obtained by concatenation. The specific formula can be expressed as:
[0053]
[0054] How to obtain it: First, set the number of learnable requests as query_num and the dimension as embed_dim. Then, construct a learnable matrix of size query_num * embed_dim. .
[0055] The mask feature decoding module is used to decode the mixed request into an output request and fuse the output request with cross-modal mask features to obtain a segmentation mask. Specifically, the mask feature decoding module uses the mixed request... The input query uses multi-scale image features as key-value pairs, and the output request is generated after a decoding process. , with mask features from the audio-visual mixing module Feature fusion is performed, and a multilayer perceptron is used. The channel information is integrated and finally passed through the fully connected layer. Obtain the segmentation mask The specific formula can be expressed as:
[0056]
[0057] Step S103: Train the multi-type audio-guided UAV remote sensing image segmentation model using a multi-type audio UAV remote sensing image guided segmentation dataset until the training completion conditions are met.
[0058] Specifically, labeled data from the multi-type audio UAV remote sensing image guided segmentation dataset are sequentially input into the audio-guided UAV remote sensing image segmentation model. After each input, an output value is obtained. The loss value for this training is calculated based on the output value and the constructed loss function. After obtaining the loss value, the loss value is backpropagated and the predetermined weight parameters in the model are adjusted (e.g., by searching using gradient descent) until the loss value for this training reaches the minimum value. New labeled data is input into the model after this training and the above process is repeated for iterative training until the loss value after multiple training iterations continuously converges to a range less than a preset threshold. The training is then considered to be complete, and the model training is considered to be satisfied.
[0059] The following is an illustrative example of the loss function used to calculate the total loss value of the model. The total loss function value is... Including segmentation loss Mask request semantic consistency loss The total loss L of the model is calculated in two parts using the following formula:
[0060]
[0061] Where λ represents the hyperparameter value that controls the proportion of semantic consistency loss in the mask request.
[0062] The following discusses the segmentation loss. and mask request semantic consistency loss The calculation method is explained below:
[0063] Segmentation loss It uses Dice loss as the segmentation loss representation, and the specific calculation formula is as follows:
[0064]
[0065] Where P represents the predicted mask and G represents the pre-labeled true mask.
[0066] Mask request semantic consistency loss Mixed requests from the mask feature decoding module Request for prediction mask generation The cosine similarity is calculated using the following formula:
[0067]
[0068] in This represents the number of requests.
[0069] Specifically, predictive mask generation The specific steps for making a request are as follows:
[0070] Step 1: Process the input mask tensor Perform convolution operations and obtain the first layer feature map using the ReLU activation function. ;
[0071] The second step is to map the features of the first layer. The input is fed into the second convolutional layer and passed through the ReLU activation function to obtain the deep feature representation of the mask.
[0072] Step 3: Feature mapping Adaptive average pooling is performed to compress the spatial dimension to the number of queries, and the final result is obtained by flattening. .
[0073] This invention also provides a multi-type audio-guided UAV remote sensing image segmentation system based on fine semantic embedding, such as... Figure 3 As shown, the multi-type audio-guided UAV remote sensing image segmentation system based on fine semantic embedding includes: a data construction module 100, a model construction module 200, and a training module 300. The data construction module 100 is used to collect UAV remote sensing images and remote sensing-related audio and speech data to construct a multi-type audio UAV remote sensing image guided segmentation dataset. The model construction module 200 is used to construct a multi-type audio guided UAV remote sensing image segmentation model based on fine semantic embedding. The image feature encoding module is used to convert remote sensing images into image features, the audio feature encoding module is used to extract original audio features, and the fine semantic embedding module is used to perform cross-modal fusion of original audio features and image features to obtain cross-modal mask features. The fine semantic embedding module is used to convert original audio features into audio requests through an attention mechanism and mix them with learnable requests to obtain mixed requests. The mask feature decoding module is used to decode mixed requests into output requests and fuse the output requests with cross-modal mask features to obtain a segmentation mask. The training module 300 is used to train the multi-type audio guided UAV remote sensing image segmentation model using the multi-type audio UAV remote sensing image guided segmentation dataset. The trained multi-type audio guided UAV remote sensing image segmentation model can output a segmentation mask with ambient sound or speech references based on the input UAV remote sensing image with ambient sound or speech.
[0074] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A multi-type audio-guided UAV remote sensing image segmentation method based on refined semantic embedding, characterized in that, The multi-type audio-guided UAV remote sensing image segmentation method includes: Collect UAV remote sensing images and remote sensing-related audio and speech data, and construct a multi-type audio UAV remote sensing image-guided segmentation dataset; A multi-type audio-guided UAV remote sensing image segmentation model based on refined semantic embedding is constructed. The model includes: a UAV remote sensing image feature encoding module, an audio feature encoding module, a refined semantic embedding module, an audio semantic request module, and a mask feature decoding module. The image feature encoding module converts the remote sensing image into image features; the audio feature encoding module extracts original audio features; the refined semantic embedding module performs cross-modal fusion of the original audio features and the image features to obtain cross-modal mask features; and the audio semantic request module uses an attention mechanism to convert the original audio features into audio requests and mixes them with learnable requests to obtain... The hybrid request; the mask feature decoding module is used to decode the hybrid request into an output request and fuse the output request with the cross-modal mask feature to obtain a segmentation mask; the fine semantic embedding module includes an adaptive modal fusion module, a multi-scale attention module and an audio-visual hybrid module, wherein the adaptive modal fusion module is used to perform multi-scale fusion of the original audio features with image features at various scales to obtain fused features, the multi-scale attention module is used to convert the fused features into mask features through an attention mechanism, and the audio-visual hybrid module is used to perform cross-modal attention calculation on the original audio features to obtain the cross-modal mask feature; The multi-type audio-guided UAV remote sensing image segmentation model is trained using the multi-type audio UAV remote sensing image guided segmentation dataset until the training completion condition is met. The trained multi-type audio-guided UAV remote sensing image segmentation model can output a segmentation mask for ambient sound or speech based on the input UAV remote sensing image containing ambient sound or speech.
2. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 1, characterized in that, The process of converting remote sensing images into image features includes: Multi-scale sampling is performed on the input UAV remote sensing images to obtain multi-scale image features.
3. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 2, characterized in that, The step of fusing the original audio features with image features of various sizes at multiple scales to obtain fused features includes: Compressed audio features are obtained by linearly mapping the original audio features. The compressed audio features are mapped to image feature spaces at different sampling scales to obtain compressed projection features at different scales. The average value of the original audio features in the time dimension is calculated to obtain global audio features. The global audio features are then linearly transformed to obtain global projection features. The compressed projected audio features are processed by the Sigmoid function to obtain a gated graph. The compressed projected features, the image features, and the global projected features weighted by the gated graph are then fused to obtain a fused feature.
4. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 3, characterized in that, The process of converting fused features into masked features through an attention mechanism includes: Sub-fusion features of multiple resolutions are extracted from the fusion features, and the sub-fusion features are flattened and concatenated as input queries for the Transformer encoder. The features output by the Transformer encoder are restored to their original shape at the corresponding scale. The features with a resolution of 1 are extracted separately and upsampled by 2 times to obtain the sampled features. The sampled features are fused with the original UAV remote sensing image to obtain the mask features.
5. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 4, characterized in that, The step of performing cross-modal attention calculation on the original audio features to obtain the cross-modal mask features includes: By using a channel attention-based design, cross-modal attention calculations are performed on the original audio features and the mask features to learn a set of weights, thereby obtaining the cross-modal mask features.
6. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 1, characterized in that, The step of converting the original audio features into an audio request through an attention mechanism and mixing it with a learnable request to obtain a hybrid request includes: An initial request is constructed, which is used as a query. The original audio features are used as keys and values and input into the request generator in the attention model to obtain the audio request. The audio request is fused with the learnable request to obtain the hybrid request.
7. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 1, characterized in that, The step of decoding the hybrid request into an output request and fusing the output request with the cross-modal mask features to obtain a segmentation mask includes: The hybrid request is used as the input query, and multi-scale image features are used as key-value pairs. The output request is obtained through a decoding process. The output request is fused with the cross-modal mask features, and the channel information is integrated through a multi-layer sensing mechanism. Finally, a segmentation mask is obtained through a fully connected layer.
8. The multi-type audio-guided UAV remote sensing image segmentation method according to claim 1, characterized in that, The step of training the multi-type audio-guided UAV remote sensing image segmentation model using the multi-type audio UAV remote sensing image guided segmentation dataset includes: The training data in the multi-type audio UAV remote sensing image guided segmentation dataset are sequentially input into the multi-type audio guided UAV remote sensing image segmentation model. The loss value is calculated based on the output of the multi-type audio guided UAV remote sensing image segmentation model. The loss function for calculating the loss value is a weighted sum of the segmentation loss and the mask request semantic consistency loss. The loss value is backpropagated and the weights of the multi-type audio-guided UAV remote sensing image segmentation model are adjusted until the loss value of the multi-type audio-guided UAV remote sensing image segmentation model reaches its minimum. The above process is repeated until the loss value continuously converges to a range less than a preset threshold.
9. A multi-type audio-guided UAV remote sensing image segmentation system based on refined semantic embedding, characterized in that, The multi-type audio-guided UAV remote sensing image segmentation system includes: The data construction module is used to collect UAV remote sensing images and remote sensing-related audio and speech data, and to construct a multi-type audio UAV remote sensing image-guided segmentation dataset. The model building module is used to construct a multi-type audio-guided UAV remote sensing image segmentation model based on fine semantic embedding. The model includes: a UAV remote sensing image feature encoding module, an audio feature encoding module, a fine semantic embedding module, an audio semantic request module, and a mask feature decoding module. The image feature encoding module converts the remote sensing image into image features; the audio feature encoding module extracts the original audio features; the fine semantic embedding module performs cross-modal fusion of the original audio features and the image features to obtain cross-modal mask features; and the fine semantic embedding module uses an attention mechanism to convert the original audio features into audio requests and integrates them with learnable requests. The process involves several steps: First, a hybrid request is obtained by mixing the hybrid request. Second, a mask feature decoding module decodes the hybrid request into an output request and fuses the output request with the cross-modal mask features to obtain a segmentation mask. Third, a fine-grained semantic embedding module includes an adaptive modal fusion module, a multi-scale attention module, and an audio-visual hybrid module. The adaptive modal fusion module fuses the original audio features with image features at various scales to obtain fused features. The multi-scale attention module converts the fused features into mask features using an attention mechanism. Finally, the audio-visual hybrid module performs cross-modal attention calculations on the original audio features to obtain the cross-modal mask features. The training module is used to train the multi-type audio-guided UAV remote sensing image segmentation model using the multi-type audio UAV remote sensing image guided segmentation dataset. The trained multi-type audio-guided UAV remote sensing image segmentation model can output a segmentation mask for ambient sound or speech based on the input UAV remote sensing image with ambient sound or speech.