Unmanned aerial vehicle image processing method, apparatus, device and medium

Abstract

The present disclosure relates to a UAV image processing method, device, equipment and medium, belonging to the technical field of image processing. The method comprises: acquiring target UAV image output by a target UAV; inputting the target UAV image into an encoder in a preset image processing model to obtain a semantic feature map; inputting the semantic feature map into a decoder in the image processing model to obtain a fusion feature map; and inputting the fusion feature map into a frequency adjuster in the image processing model to obtain a multi-frequency fusion map; wherein the multi-frequency fusion map is used to measure the distribution of a target object in the target UAV image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method, apparatus, device, and medium for processing images from unmanned aerial vehicles (UAVs). Background Technology

[0002] Crab farming is an important aquaculture industry in my country, holding a significant position in freshwater aquaculture. Aquatic plants in crab ponds are not only a crucial source of food for crabs but also a key factor in maintaining the ecological balance of the ponds. Currently, existing methods for monitoring aquatic plants mainly rely on farmers' subjective judgment and manual field surveys, which are not only inefficient but also difficult to obtain accurate quantitative data. Summary of the Invention

[0003] One objective of this disclosure is to provide a new technical solution for unmanned aerial vehicle (UAV) image processing.

[0004] According to a first aspect of this disclosure, a method for processing UAV images is provided, the method comprising: Acquire the target drone image output by the target drone; The target UAV image is input into the encoder in the preset image processing model to obtain a semantic feature map; The semantic feature map is input into the decoder in the image processing model to obtain the fused feature map; The fused feature map is input into the frequency adjuster in the image processing model to obtain a multi-frequency fusion map; wherein, the multi-frequency fusion map is used to measure the distribution of the target object in the target UAV image.

[0005] Optionally, before inputting the target UAV image into the encoder of a preset image processing model to obtain a semantic feature map, the method further includes: Histogram equalization and contrast enhancement are performed on the target UAV image to obtain the processed target UAV image.

[0006] Optionally, the encoder in the image processing model includes an embedding module, multiple sets of encoding modules, and multiple attention modules, with each encoding module corresponding to one of the attention modules; each encoding module includes multiple encoding modules; wherein, the encoding modules are constructed using quadruples set by proxy tokens.

[0007] Optionally, the step of inputting the target UAV image into an encoder in a preset image processing model to obtain a semantic feature map includes: The target UAV image is input into the embedding module to obtain an image block representing the target UAV image; The image block is input into the first group of coding modules to obtain the first coded feature map; The first encoded feature map is input into the first attention module to obtain the first semantic feature map; The i-th encoded feature map is used as the input feature map and input into the i-th encoding module to obtain the i-th encoded feature map; where i is a positive integer; The i-th encoded feature map is input into the i-th attention module to obtain the i-th semantic feature map, and the first semantic feature map and the i-th semantic feature map are used as the semantic feature map.

[0008] Optionally, the decoder in the image processing model includes multiple decoding modules, and each decoding module is configured in a one-to-one correspondence with the attention module; The step of inputting the semantic feature map into the decoder in the image processing model to obtain the fused feature map includes: The last semantic feature map is input into the last decoding module to obtain the last decoded feature map; The j-th decoded feature map and the (j-1)-th semantic feature map are input into the (j-1)-th decoding module to obtain the (j-1)-th fused feature map; where j is a positive integer; The second decoded feature map and the first semantic feature map are input into the first decoding module to obtain the first fused feature map, which is then used as the fused feature map.

[0009] Optionally, before inputting the fused feature map into the frequency adjuster in the image processing model to obtain the multi-frequency fused map, the method further includes: The fused feature map is input into the reshaping module in the image processing model to obtain the reshaped fused feature map; The reshaped fused feature map is input into the deformable convolution module in the image processing model to obtain the fused feature map with enhanced edge details.

[0010] Optionally, the method further includes: Obtain training samples; The training samples are input into the image processing model to obtain the training results; The training results are input into a preset loss function to obtain the loss value; Based on the loss value, the image processing model is adjusted to obtain the adjusted image processing model.

[0011] According to a second aspect of this disclosure, a drone image processing apparatus is also provided, the apparatus comprising: The acquisition module is used to acquire the target drone image output by the target drone; The first obtaining module is used to input the target UAV image into the encoder in the preset image processing model to obtain a semantic feature map; The second obtaining module is used to input the semantic feature map into the decoder in the image processing model to obtain the fused feature map; The third module is used to input the fused feature map into the frequency adjuster in the image processing model to obtain a multi-frequency fusion map; wherein, the multi-frequency fusion map is used to measure the distribution of the target object in the target UAV image.

[0012] According to a third aspect of this disclosure, a computer system is also provided, the computer system including a processor, which implements the UAV image processing method of the first aspect when the processor executes program instructions or code.

[0013] For example, the computer system also includes a memory for storing program instructions or code.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described UAV image processing method at runtime.

[0015] According to a fifth aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed, causes a computer to perform the steps of the above-described UAV image processing method.

[0016] According to a sixth aspect of this disclosure, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described UAV image processing method through the computer program.

[0017] One beneficial effect of this disclosure is that the UAV image processing method provided by the present invention can process the UAV images output by the UAV through a preset image processing model to obtain a multi-frequency fusion map that can measure the distribution of aquatic plants in the target UAV image, thereby effectively improving the efficiency of aquatic plant detection.

[0018] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the embodiments of the present disclosure.

[0020] Figure 1 A flowchart illustrating a drone image processing method according to some embodiments is shown; Figure 2 A schematic diagram of the structure of an image processing model according to some embodiments is shown; Figure 3 A schematic diagram of the structure of a drone image processing apparatus according to some embodiments is shown; Figure 4 A schematic diagram of the hardware structure of an electronic device according to some embodiments is shown. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0026] <Method Implementation> Figure 1 This is a flowchart illustrating a drone image processing method according to one embodiment. The implementing entity is a smart terminal, such as a personal computer, mobile phone, tablet, or server.

[0027] like Figure 1 As shown, the UAV image processing method of this embodiment may include the following steps S110 to S140: Step S110: Acquire the target drone image output by the target drone.

[0028] In this embodiment, the acquired target drone images include five bands: RGB, NIR, and RedEdge. The image overlap of the target drone images is set to 70% forward overlap and 60% lateral overlap to ensure complete coverage of the crab pond. The processed multispectral image of the target drone has a spatial resolution of 5cm, which can clearly display the detailed features of aquatic plants.

[0029] Step S120: Input the target UAV image into the encoder in the preset image processing model to obtain a semantic feature map.

[0030] In some embodiments, prior to step S120, the method further includes the following step S210: Step S210: Perform histogram equalization and contrast enhancement processing on the target UAV image to obtain the processed target UAV image.

[0031] In this embodiment, the target UAV image undergoes geometric correction first. Camera calibration parameters are used to eliminate lens distortion, and GPS and IMU data from the UAV are combined for geometric correction to ensure image geometric accuracy. The radiometric correction stage compensates for atmospheric scattering and absorption using an established radiative transfer model, while simultaneously correcting for uneven illumination to eliminate brightness differences caused by variations in the sun's angle. To highlight the spectral differences between aquatic plants and the background, histogram equalization and contrast enhancement are used to preprocess the target UAV image. Furthermore, the large target UAV image is divided into 224×224 pixel image blocks using a sliding window method, with a window step size of 112 pixels to ensure 50% overlap between adjacent image blocks, avoiding loss of boundary information and meeting the input requirements of subsequent image processing models.

[0032] In some embodiments, the encoder in the image processing model includes an embedding module, multiple sets of encoding modules, and multiple attention modules, with each encoding module corresponding to an attention module; each encoding module includes multiple encoding modules; wherein, the encoding modules are constructed using quadruples set by proxy tokens.

[0033] In some embodiments, step S120 may include the following steps S310 to S350: Step S310: Input the target UAV image into the embedding module to obtain an image block representing the target UAV image.

[0034] Step S320: Input the image block into the first group of coding modules to obtain the first coded feature map.

[0035] Step S330: Input the first encoded feature map into the first attention module to obtain the first semantic feature map.

[0036] Step S340: The i-th encoded feature map is used as the feature map input and input into the i-th encoding module to obtain the i-th encoded feature map; where i is a positive integer.

[0037] Step S350: Input the i-th encoded feature map into the i-th attention module to obtain the i-th semantic feature map, and use the first semantic feature map and the i-th semantic feature map as the semantic feature map.

[0038] In this embodiment, the encoding modules can be four groups. For example, the second encoding feature map is used as the feature map input and input into the second group of encoding modules to obtain the second encoding feature map.

[0039] In this embodiment, the encoder and decoder in the image processing model combine the advantages of the Swin Transformer and are improved for the aquatic plant extraction task. The encoder is based on the Swin Transformer architecture. First, the input 224×224×5 image is segmented into 4×4 patches through a patch embedding layer, generating a total of 3136 patches. Each patch is linearly projected onto a 96-dimensional feature space. The encoder consists of four stages, with feature dimensions of 96, 192, 384, and 768 for each stage, corresponding to feature map sizes of 56×56, 28×28, 14×14, and 7×7. The number of SwinTransformer Blocks in each stage is set to 2, 2, 6, and 2, respectively, and the number of attention heads is set to 3, 6, 12, and 24, respectively. The window size is uniformly set to 7×7.

[0040] The encoder's encoding module introduces an Agent Attention mechanism to replace the standard window multi-head self-attention. Agent Attention is represented as a quadruple (Q, A, K, V), where A represents agent tokens. The core idea of ​​this mechanism is to let agent tokens act as proxies for query tokens, first aggregating information from the key and value, and then passing the information back to the query tokens. Specifically, adaptive average pooling (AdaptiveAvgPool2d) generates a fixed number of 16 agent tokens (arranged in a 4×4 pattern) from the input feature map. The number of these agent tokens is much smaller than the original number of pixels (49), thus significantly reducing computational complexity. Specifically, the Agent Attention mechanism employs a two-stage attention calculation strategy: In the first stage, the agent tokens and the key vector calculate attention, combined with corresponding positional biases, to obtain an aggregated representation of global information: agent_attn = softmax((agent_tokens×scale) @ K^T + position_bias), and then enhanced agent features are obtained through agent_v = agent_attn @V; In the second stage, the query vector and agent tokens calculate attention, combined with another set of positional biases, to achieve the distribution of global information: q_attn = softmax((Q × scale) @ agent_tokens^T + agent_bias), and the final output is x = q_attn @ agent_v. To further enhance feature representation, AgentAttention also integrates deep-wise convolution (DWC). The Value features are further processed through 3×3 grouped convolutions, and then added to the attention output to form residual connections. This design not only maintains the global modeling capabilities of the Transformer but also incorporates the local feature extraction advantages of convolution, making it particularly suitable for handling targets with complex spatial distribution patterns and irregular boundaries, such as aquatic plants. Support for multi-scale processing is another important feature of Agent Attention. The spatial resolution of the Key and Value can be controlled by the sr_ratio parameter. When sr_ratio > 1, convolutional downsampling is used to reduce the resolution of the Key and Value, and then bilinear interpolation is used to restore the original size at the output, further improving computational efficiency while maintaining accuracy.

[0041] In this embodiment, the introduced Agent Attention uses a fixed number of agent tokens (16) generated through adaptive average pooling, which significantly reduces computational complexity. Information transmission is achieved through two-stage attention calculation. At the same time, deep convolution is integrated to enhance local feature extraction capabilities and form residual connections, which effectively improves global context modeling capabilities and incorporates the local feature extraction advantages of convolution.

[0042] In this embodiment, an attention module, which can be a channel-wise affinity attention (CAA) module, is integrated after each encoder stage to further enhance feature representation capabilities. This attention module enhances the semantic relevance of feature representations by learning the affinity relationships between channels. The attention module employs a two-branch structure: a Compact Channel-wise Comparator branch and a Channel Affinity Estimator branch. In the Compact Channel-wise Comparator branch, the input encoded feature map is first transposed from (B, C, N) format to (B, N, C) format, and then processed by query_conv and key_conv respectively. Both convolutional layers use 1×1 convolutions, compressing the number of channels to 1 / 8 of the original, reducing computation while extracting key features. Each convolutional layer is followed by batch normalization and a ReLU activation function to ensure feature stability and non-linear expressive power. The similarity matrix similarity_mat between channels is obtained by calculating the matrix multiplication of query and key.

[0043] In this embodiment, the Channel Affinity Estimator branch is responsible for converting similarity into affinity weights. This branch employs an innovative maximum subtraction operation: first, the maximum value of each row of the similarity matrix is ​​calculated, and then the original similarity is subtracted from the maximum value to obtain the affinity matrix `affinity_mat`. This operation highlights relatively small similarity values ​​and strengthens the connections between weakly correlated channels. The affinity matrix is ​​normalized using softmax to ensure that the weights sum to 1. Simultaneously, the input encoded feature map is processed through `value_conv`, a convolution that maintains the original number of channels and is primarily used for non-linear transformation and normalization of features. The final output semantic feature map is obtained by weighting the `value` features with affinity weights and adding it to the original features through a residual connection controlled by a learnable parameter α: semantic feature map output = α × weighted_features + input. This effectively preserves the original feature information while incorporating the affinity relationships between channels, significantly improving the expressive power of the features.

[0044] In this embodiment, the introduced attention module significantly improves the semantic relevance of feature representations by modeling the affinity relationships between channels. Employing a dual-branch structure of a Compact Channel-wise Comparator and a Channel AffinityEstimator, the affinity matrix is ​​calculated through maximum subtraction, highlighting potential connections between weakly correlated channels. This approach can uncover complex semantic associations between different spectral channels, offering a particular advantage for targets with complex spectral features, such as aquatic plants.

[0045] Step S130: Input the semantic feature map into the decoder in the image processing model to obtain the fused feature map.

[0046] In some embodiments, the decoder in the image processing model includes multiple decoding modules, and each decoding module is configured in a one-to-one correspondence with an attention module; step S130 may include the following steps S410 to S430: Step S410: Input the last semantic feature map into the last decoding module to obtain the last decoded feature map.

[0047] Step S420: Input the j-th decoded feature map and the (j-1)-th semantic feature map into the (j-1)-th decoding module to obtain the (j-1)-th fused feature map; where j is a positive integer.

[0048] Step S430: Input the second decoded feature map and the first semantic feature map into the first decoding module to obtain the first fused feature map, which is then used as the fused feature map.

[0049] In some examples, such as Figure 2 As shown, there are 4 decoding modules. The fourth decoding feature map and the third semantic feature map are input into the third decoding module to obtain the third fused feature map. The third decoding feature map and the second semantic feature map are input into the second decoding module to obtain the second fused feature map.

[0050] In this embodiment, the decoder adopts a symmetrical structure, gradually restoring the spatial resolution of the semantic feature map through four stages. Each decoding stage integrates the skip connection features of the corresponding encoding stage, achieving effective fusion of multi-scale information. To maximize the utilization of the multi-level features extracted by the encoder, the skip connection features are also enhanced using an attention module, ensuring that the features passed from the encoder to the decoder are optimized, high-quality representations.

[0051] In this embodiment, the upsampling operation is implemented using the PatchExpanding module. This module first expands the feature dimension to twice its original size through a linear layer, and then uses pixel shuffle technology to increase the spatial resolution of the decoded feature map to twice its original size while maintaining the spatial continuity of the features. This can better preserve detailed information and reduce the checkerboard effect.

[0052] In this embodiment, during the feature fusion process of the j-th decoded feature map and the (j-1)-th semantic feature map, an adaptive dimension matching mechanism is used. At skip connections, the skip connection features are first converted from BLC format to BCL format to adapt to the attention module, and then converted back to BLC format after enhancement by the attention module. For skip connections with mismatched dimensions, the number of channels is adjusted using 1×1 convolution to ensure correct feature fusion. Simultaneously, residual connections are used to add the skip features to the decoded features, effectively reducing feature redundancy issues that may result from simple concatenation. The final output fused feature map is converted to BCHW format through transpose and reshaping operations, and then upsampled to the original image size using bilinear interpolation.

[0053] Step S140: Input the fused feature map into the frequency adjuster in the image processing model to obtain a multi-frequency fused map; wherein, the multi-frequency fused map is used to measure the distribution of the target object in the target UAV image.

[0054] In this embodiment, the target object can be aquatic plants in a pond or river.

[0055] In this embodiment, the frequency adjuster can be a frequency selection module, which can perform multi-frequency analysis on the fused feature map to improve the representation of aquatic plants with different characteristics in different frequency domains. This module optimizes and balances the high-frequency and low-frequency components in the feature representation by reweighting through spatial variations from a spectral analysis perspective.

[0056] In this embodiment, the frequency adjuster employs a progressive low-pass filter decomposition strategy, using a configurable kernel size list (k_list=[3,5]) to extract low-frequency components layer by layer through average pooling. For each scale k, the frequency adjuster first performs boundary padding using ReplicationPad2d, then performs low-pass filtering using AvgPool2d to obtain the low-frequency component low_part, and obtains the corresponding high-frequency component through the difference operation high_part=pre_x- low_part, ensuring the integrity and hierarchy of the frequency components in the multi-frequency fusion map.

[0057] In this embodiment, the frequency adjuster is based on an adaptive frequency weight generation mechanism. For each frequency component, a dedicated convolutional layer (freq_weight_conv) generates spatially varying frequency weights. These weights are normalized and amplified by a factor of 2 using a sigmoid activation function, forming an adaptive attention distribution. Spatially adaptive frequency selection is achieved by reshaping the frequency weights into a grouped form and multiplying them element-wise with the corresponding frequency components. When low-frequency attention is enabled (lowfreq_att=True), the module also applies an attention mechanism to the final low-frequency components to ensure that low-frequency structural information is also optimized. All weighted frequency components are fused into an enhanced multi-frequency feature representation through channel concatenation (torch.cat), and then the fused 3x channel features are reduced back to the original number of channels through a 1×1 convolution (freq_conv).

[0058] In this embodiment, the frequency adjuster employs a progressive frequency decomposition strategy, using configurable multi-scale low-pass filters to extract different frequency components layer by layer. Then, it optimizes the balance between high-frequency and low-frequency components through spatial adaptive reweighting. High-frequency components primarily reflect the boundary texture details of aquatic plants, while low-frequency components highlight the overall shape and spatial distribution patterns. Spatially varying attention weights are generated for each frequency component of the multi-frequency fusion map, enabling the image processing model to adaptively adjust the importance of different frequency information based on local content. The frequency adjuster utilizes a sigmoid-activated frequency weight generation mechanism, allowing the image processing model to adaptively adjust the importance of different frequency information based on local content. Finally, effective fusion of multi-frequency features in the multi-frequency fusion map is achieved through channel stitching and 1×1 convolution.

[0059] In some embodiments, prior to step S140, the method further includes the following steps S510 and S520: Step S510: Input the fused feature map into the reshaping module in the image processing model to obtain the reshaped fused feature map.

[0060] Step S520: Input the reshaped fused feature map into the deformable convolution module in the image processing model to obtain the fused feature map with enhanced edge details.

[0061] In this embodiment, the final output fused feature map is converted to BCHW format through transpose and reshaping operations of the reshaping module, and then upsampled to the original image size through bilinear interpolation.

[0062] In this embodiment, as Figure 2 As shown, two cascaded deformable convolutional layers are introduced at the end of the image processing module's network to handle the irregular shapes of aquatic plants. The deformable convolutional module of the image processing module contains three key components: offset convolution (offset_conv), modulation convolution (modulator_conv), and regular convolution (regular_conv). The offset convolution has 2 × kernel_size × kernel_size output channels, generating offsets in both the x and y directions for each kernel position; the modulation convolution has 1 × kernel_size × kernel_size output channels, generating a modulation factor for each kernel position; the regular convolution is responsible for the final feature extraction. The modulation factor is processed by a 2 × sigmoid activation function, initialized to 0.5, making the initial state similar to a regular convolution. The first deformable convolutional layer (deform_conv1) mainly learns the coarse shape offset, while the second layer (deform_conv2) further refines the boundary information. Each layer is connected by a ReLU activation function.

[0063] In this embodiment, two deformable convolutional layers are introduced at the end of the image processing module's network. Each layer contains three components: offset convolution, modulation convolution, and standard convolution. By learning spatial offsets and modulation factors, the convolutional operation can adaptively adjust the sampling position and weights. The first layer learns a coarse shape offset, while the second layer further refines the boundary information to adapt to handling targets with complex and varied shapes, such as aquatic plants, significantly improving the detection accuracy of irregularly shaped aquatic plants.

[0064] In some embodiments, the method further includes the following steps S610 to S640: Step S610: Obtain training samples.

[0065] Step S620: Input the training samples into the image processing model to obtain the training results.

[0066] Step S630: Input the training results into the preset loss function to obtain the loss value.

[0067] Step S640: Adjust the image processing model according to the loss value to obtain the adjusted image processing model.

[0068] In this embodiment, the loss function can be a multi-component combined loss function, comprehensively considering pixel-level classification accuracy, segmentation boundary quality, and class balance. The expression of the loss function is as follows: The weighting coefficients for the losses of each component are as follows: =0.3、 =0.3、 =0.2、 =0.2.

[0069] Among them, the binary cross-entropy loss ( ) is responsible for basic pixel-level binary classification, ensuring that the image processing model can correctly distinguish between aquatic plants and background pixels, using Dice loss ( It is specifically optimized for optimizing the overlap of segmented regions, and has a good effect on irregularly shaped aquatic plants. (Focus loss...) By dynamically adjusting the weights of easy and difficult samples, the problem of positive and negative sample imbalance is solved. It is particularly suitable for handling situations with uneven aquatic plant coverage, and the IoU loss ( Optimize the intersection-union ratio index to improve the overall quality of the segmentation results.

[0070] In this embodiment, the training strategy employs a multi-stage approach. The first stage uses a relatively large learning rate (1e-3) for pre-training, quickly converging to a good initial value. The second stage reduces the learning rate to 1e-4 for fine-tuning. The third stage further reduces it to 1e-5 to ensure the model converges to the optimal solution. The optimizer chosen is AdamW, where... 1 = 0.9 2 = 0.999, and the weight decay coefficient is 0.01. The learning rate scheduling adopts a cosine annealing strategy, which smoothly reduces the learning rate in each stage.

[0071] <Equipment Example 1> Figure 3 This is a schematic block diagram of a drone image processing device according to one embodiment. Figure 3 As shown, the UAV image processing device 300 may include: The acquisition module 310 is used to acquire the target drone image output by the target drone; The first module 320 is used to input the target UAV image into the encoder in the preset image processing model to obtain a semantic feature map; The second module 330 is used to input the semantic feature map into the decoder in the image processing model to obtain the fused feature map; The third module 340 is used to input the fused feature map into the frequency adjuster in the image processing model to obtain a multi-frequency fused map; wherein, the multi-frequency fused map is used to measure the distribution of the target object in the target UAV image.

[0072] Optionally, the UAV image processing device 300 further includes a preprocessing module for performing histogram equalization and contrast enhancement processing on the target UAV image to obtain the processed target UAV image.

[0073] Optionally, the first obtaining module 320 is further configured to input the target UAV image into the embedding module to obtain an image block representing the target UAV image; input the image block into the first group of encoding modules to obtain the first encoded feature map; input the first encoded feature map into the first attention module to obtain the first semantic feature map; input the i-th encoded feature map as a feature map into the i-th group of encoding modules to obtain the i-th encoded feature map; where i is a positive integer; input the i-th encoded feature map into the i-th attention module to obtain the i-th semantic feature map, and use the first semantic feature map and the i-th semantic feature map as a semantic feature map.

[0074] Optionally, the second obtaining module 330 is further configured to input the last semantic feature map into the last decoding module to obtain the last decoded feature map; input the j-th decoded feature map and the (j-1)-th semantic feature map into the (j-1)-th decoding module to obtain the (j-1)-th fused feature map; where j is a positive integer; input the second decoded feature map and the first semantic feature map into the first decoding module to obtain the first fused feature map, and use it as the fused feature map.

[0075] Optionally, the UAV image processing device 300 further includes a processing module for inputting the fused feature map into the reshaping module in the image processing model to obtain the reshaped fused feature map; and inputting the reshaped fused feature map into the deformable convolution module in the image processing model to obtain the fused feature map with enhanced edge details.

[0076] Optionally, the UAV image processing device 300 further includes a training module for acquiring training samples; inputting the training samples into the image processing model to obtain training results; inputting the training results into a preset loss function to obtain a loss value; and adjusting the image processing model based on the loss value to obtain an adjusted image processing model.

[0077] <Equipment Example 2> Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to another embodiment.

[0078] like Figure 4 As shown, the electronic device 400 includes a processor 410 and a memory 420, the memory 420 for storing an executable computer program, and the processor 410 for executing methods as described in any of the above method embodiments under the control of the computer program.

[0079] Each module of the above-mentioned UAV image processing device 300 can be implemented by the processor 410 in this embodiment executing the computer program stored in the memory 420, or it can be implemented by other structures, which are not limited here.

[0080] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0081] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0082] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0083] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0084] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0086] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0088] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for processing images from unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire the target drone image output by the target drone; The target UAV image is input into the encoder in the preset image processing model to obtain a semantic feature map; The semantic feature map is input into the decoder in the image processing model to obtain the fused feature map; The fused feature map is input into the frequency adjuster in the image processing model to obtain a multi-frequency fusion map; wherein, the multi-frequency fusion map is used to measure the distribution of the target object in the target UAV image.

2. The method according to claim 1, characterized in that, Before inputting the target UAV image into the encoder of a pre-set image processing model to obtain a semantic feature map, the method further includes: Histogram equalization and contrast enhancement are performed on the target UAV image to obtain the processed target UAV image.

3. The method according to claim 1, characterized in that, The encoder in the image processing model includes an embedding module, multiple sets of encoding modules, and multiple attention modules. The encoding modules are configured in a one-to-one correspondence with the attention modules. The encoding modules include multiple encoding modules. The encoding modules are constructed using quadruples set by proxy tokens.

4. The method according to claim 3, characterized in that, The step of inputting the target UAV image into the encoder of a pre-set image processing model to obtain a semantic feature map includes: The target UAV image is input into the embedding module to obtain an image block representing the target UAV image; The image block is input into the first group of coding modules to obtain the first coded feature map; The first encoded feature map is input into the first attention module to obtain the first semantic feature map; The i-th encoded feature map is used as the input feature map and input into the i-th encoding module to obtain the i-th encoded feature map; where i is a positive integer; The i-th encoded feature map is input into the i-th attention module to obtain the i-th semantic feature map, and the first semantic feature map and the i-th semantic feature map are used as the semantic feature map.

5. The method according to claim 4, characterized in that, The decoder in the image processing model includes multiple decoding modules, and each decoding module is configured to correspond one-to-one with the attention module. The step of inputting the semantic feature map into the decoder in the image processing model to obtain the fused feature map includes: The last semantic feature map is input into the last decoding module to obtain the last decoded feature map; The j-th decoded feature map and the (j-1)-th semantic feature map are input into the (j-1)-th decoding module to obtain the (j-1)-th fused feature map; where j is a positive integer; The second decoded feature map and the first semantic feature map are input into the first decoding module to obtain the first fused feature map, which is then used as the fused feature map.

6. The method according to claim 1, characterized in that, Before inputting the fused feature map into the frequency adjuster in the image processing model to obtain the multi-frequency fused map, the method further includes: The fused feature map is input into the reshaping module in the image processing model to obtain the reshaped fused feature map; The reshaped fused feature map is input into the deformable convolution module in the image processing model to obtain the fused feature map with enhanced edge details.

7. The method according to claim 1, characterized in that, The method further includes: Obtain training samples; The training samples are input into the image processing model to obtain the training results; The training results are input into a preset loss function to obtain the loss value; Based on the loss value, the image processing model is adjusted to obtain the adjusted image processing model.

8. A drone image processing device, characterized in that, The device includes: The acquisition module is used to acquire the target drone image output by the target drone; The first obtaining module is used to input the target UAV image into the encoder in the preset image processing model to obtain a semantic feature map; The second obtaining module is used to input the semantic feature map into the decoder in the image processing model to obtain the fused feature map; The third module is used to input the fused feature map into the frequency adjuster in the image processing model to obtain a multi-frequency fusion map; wherein, the multi-frequency fusion map is used to measure the distribution of the target object in the target UAV image.

9. An electronic device, characterized in that, The system includes a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images