Unmanned aerial vehicle target segmentation and detection method based on large model visual perception

Abstract

The invention discloses an unmanned aerial vehicle target segmentation and detection method based on large model visual perception, and the method comprises the steps: obtaining a to-be-detected data stream in real time through a multi-region spectral imaging module carried by an unmanned aerial vehicle, effectively supplementing the deficiency of single sensor information through the fusion of multi-spectral data, and improving the multi-dimensional information expression of an image; the image of the image quality evaluation network is used for noise evaluation and image optimization processing, the quality of the image is further improved, and the accuracy of a subsequent detection result is ensured; through an adaptive enhancement algorithm and weighted fusion, multi-system feature maps can be fused more accurately, so that multi-level and multi-scale feature information can be fully integrated, and a comprehensive feature map with cross-topology information is obtained. Based on a target segmentation detection model of a deformable structure, an accurate pixel-level segmentation mask and a bounding box can be generated for the problems of target deformation and posture change.

Description

Technical Field

[0001] The present invention relates to the technical field of drone technology and visual processing, and particularly to a method for drone target segmentation and detection based on large model visual perception. Background Art

[0002] Currently, existing drone target detection technologies mainly rely on a single type of sensor, such as an RGB or infrared sensing module. Although they can effectively detect targets in some scenarios, they perform poorly in complex environments, especially in situations with complex cues and noise interference, where the image quality deteriorates, leading to a reduction in the accuracy of target detection.

[0003] Traditional target detection methods mainly focus on the processing of a single image, while ignoring the utilization of multispectral data. Especially in terms of multi-level and multi-scale image feature extraction, existing methods are difficult to achieve efficient and accurate fusion. In addition, existing target segmentation models are mainly based on standard networks and cannot effectively detect target deformation and pose changes. Therefore, how to improve the accuracy and robustness of target detection, especially the adaptability in complex environments, is an urgent problem in the current technology.

[0004] In view of the deficiencies of the existing technology, our invention proposes a method for drone target segmentation and detection based on large model deployment by combining multispectral module imaging and deep learning technology. This method uses the multi-region spectral imaging module carried by the drone to continuously obtain the data stream to be detected. Through the fusion of multispectral data, it can effectively supplement the deficiencies of single-sensor information and enhance the multi-dimensional information expression of the image. The image quality assessment network is used to evaluate the noise of the image and perform image optimization processing, further improving the image quality and ensuring the accuracy of subsequent detection results. Finally, through the adaptive enhancement algorithm and weighted fusion, more accurate fusion of multi-system feature maps can be achieved, enabling the full integration of multi-level and multi-scale feature information, thereby obtaining a comprehensive feature map with cross-topological information. The target segmentation and detection model based on the deformable structure can address the problems of target deformation and pose changes and generate accurate pixel-level segmentation masks and bounding boxes. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract, and the title. However, such simplifications or omissions shall not be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed to solve the problems of robustness and accuracy in complex environment target recognition.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: The image data stream of the area to be detected is acquired in real time by a multi-spectral imaging module carried by a drone;

[0008] The acquired image data stream is evaluated for brightness, contrast, and noise level by an image quality assessment network, and is suppressed for noise and optimized through an adaptive enhancement algorithm to generate a preprocessed image;

[0009] A feature extraction model is constructed based on a hierarchical attention mechanism to perform multi-scale feature extraction on the preprocessed image to obtain multi-level feature maps;

[0010] The multi-level feature maps are spatially aligned through a spatial transformation network, and the multi-level feature maps are fused by combining a weighted fusion mechanism to generate a comprehensive feature map with cross-scale information;

[0011] A target segmentation detection model is constructed based on deformable convolution to perform target segmentation detection on the comprehensive feature map to generate a pixel-level segmentation mask and bounding box coordinates.

[0012] As a preferred solution of the drone target segmentation and detection method based on large model visual perception of the present invention, the multi-spectral imaging module includes multiple spectral band sensors capable of capturing visible light, near-infrared, and short-wave infrared band information;

[0013] The update frequency of the image data stream is greater than 30 frames / second, and the exposure time, gain, and focus parameters are automatically adjusted to adapt to flight conditions.

[0014] As a preferred solution of the drone target segmentation and detection method based on large model visual perception of the present invention, the image quality assessment network adopts a two-stream pyramid structure, including a spatial domain branch and a frequency domain branch. Among them, the spatial domain branch extracts brightness and contrast features through 5 convolutional layers, and the frequency domain branch extracts noise spectrum features through fast Fourier transform and complex convolution;

[0015] The adaptive enhancement algorithm includes a dynamic parameter adjustment strategy, and generates a brightness gain α, a contrast coefficient β, and a denoising intensity γ through a multi-layer perceptron, and the constraint conditions are α ∈ [0.8, 1.2], β ∈ [0.5, 2.0], and γ ∈ [0, 1].

[0016] As a preferred solution of the drone target segmentation and detection method based on large model visual perception of the present invention, the hierarchical attention mechanism includes a hybrid attention module, and generates attention weights through the weighted fusion of a channel attention sub-module and a spatial attention sub-module;

[0017] The multi-scale feature extraction uses 4 parallel dilated convolution branches with dilation rates of 1, 2, 4, and 6 respectively, and realizes cross-scale feature interaction through bidirectional gated connections and a dynamic routing algorithm.

[0018] As a preferred solution of the UAV target segmentation and detection method based on large model visual perception described in the present invention, the spatial transformation network generates an affine transformation matrix θ through a parameter prediction sub-network, including rotation, scaling, and translation parameters;

[0019] The weighted fusion mechanism dynamically weights and sums the aligned feature maps through channel attention weights and spatial attention weights, and combines a dilated spatial pyramid pooling module to enhance cross-scale semantic information.

[0020] As a preferred solution of the UAV target segmentation and detection method based on large model visual perception described in the present invention, the deformable convolution generates a dynamic offset Δpk through an offset prediction network, and its initial constraint range is ±1.5 pixels;

[0021] The target segmentation and detection model adopts an encoder-decoder structure. The encoder consists of 4 deformable convolution blocks, and the decoder generates a segmentation mask and a bounding box through a feature pyramid network and bilinear upsampling.

[0022] As a preferred solution of the UAV target segmentation and detection method based on large model visual perception described in the present invention, the segmentation mask is generated through transposed convolution and a Sigmoid activation function, and the coordinates of the bounding box are generated through an anchor point adaptive mechanism and tanh normalization regression.

[0023] As a preferred solution of the UAV target segmentation and detection method based on large model visual perception described in the present invention, morphological closing operation is performed on the segmentation mask to optimize edge continuity, and overlapping bounding boxes are eliminated through non-maximum suppression, and the IoU threshold is set to 0.5.

[0024] Advantages of the present invention:

[0025] 1. By evaluating the image quality and performing corresponding enhancement processing, the reconstructed image better meets the requirements of subsequent feature extraction, ensuring the quality and accuracy of the image;

[0026] 2. Through image hierarchical attention, it can automatically focus on the most informative part, enhancing the impact recognition ability of complex targets. Especially when the target size difference increases, through multi-dimensional extraction, it can simultaneously focus on the part and the overall structure, further improving the accuracy and precision of detection;

[0027] 3. The spatial transformation network can accurately calibrate feature maps with different topologies, avoiding information loss caused by spatial topology. It can perform weighted clustering on features from different systems, effectively integrating multi-pixel information, and enhancing the model's target recognition and localization capabilities. Especially when facing targets with deformation hotspots, by fusing cross-dimensional information, it can achieve more accurate target detection, providing high-quality comprehensive feature maps for subsequent target segmentation and recognition.

[0028] 4. By dynamically adjusting the structure detection model, the model can better adapt to the shape and pose changes of the target. Especially in complex scenarios, it can perform more accurate target segmentation. Different, deformable frameworks have additional flexibility, which can improve the detection ability for irregular-shaped targets and maintain the partitioning effect even when the size changes significantly. The finally generated pixel-level segmentation masks and bounding box coordinates provide important data support for the drone's target tracking, recognition, and subsequent decision-making, enhancing the reliability and practicality of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0030] Figure 1 It is a schematic flowchart of the method for drone target segmentation and detection based on large model visual perception shown in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present invention in conjunction with the drawings of the specification. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0032] Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0034] According to the embodiments of the present invention, in combination with Figure 1The shown flowchart depicts a method for drone target segmentation and detection based on large model visual perception, which specifically includes the following steps:

[0035] S1. Real-time acquisition of the image data stream of the area to be detected through the multispectral imaging module carried by the drone. Here, it should be noted in this step that:

[0036] Install the multispectral imaging module on the drone fuselage to ensure its stable acquisition of image data;

[0037] During the flight of the drone, real-time acquisition and transmission of the multispectral image data stream are carried out. The image data stream includes the spectral band information of each area to be detected;

[0038] The image data stream is expressed in the form of a multi-dimensional data array. Each frame of the image contains the numerical values of multiple spectral channels and is transmitted to the ground control station in real-time under wireless communication conditions;

[0039] By automatically adjusting the exposure time, gain, and focus parameters of the imaging module, ensure the stable acquisition of image data that meets the requirements of subsequent processing during flight.

[0040] As an example, the multispectral imaging module is equipped with an automatic exposure control unit, an automatic gain adjustment unit, and an automatic focus unit. According to the real-time acquired ambient light conditions and the brightness of the target area, it automatically adjusts the exposure, gain, and focus parameters to ensure that the image can maintain good quality under different flight conditions. For example, the exposure time and gain can be automatically adjusted according to the current ambient light intensity, and the automatic focus is adjusted by calculating the clarity evaluation function of the target area to ensure the clarity of the image data.

[0041] In an alternative embodiment, by installing the multispectral imaging module on the drone fuselage, ensure its stable operation and acquisition of image data during flight. The multispectral imaging module has high resolution and fast image acquisition capabilities to meet the detection requirements in different environments.

[0042] Exemplarily, the multispectral imaging module consists of sensors in multiple spectral bands for capturing image data in different wavelength ranges, including visible light, near-infrared, and short-wave infrared. Through these spectral information, different physical characteristics of the target can be effectively identified and distinguished.

[0043] It should be noted that during the flight of the drone, the carried multispectral imaging module continuously and real-time acquires the image data stream. The data stream is an image sequence, and the output of the image data stream is a continuous video stream or several frames of static images. Each frame of the image contains the spectral band information at that moment.

[0044] As an example, the update frequency of the image data stream should be at least 30 frames per second to ensure the capture and analysis of dynamic targets during flight. For static targets, the update frequency is adaptively reduced according to the actual application situation. For example, if the image frame is represented as I(t) which is the image data stream captured at time t, then the expression of the image data stream is as follows:

[0045] I(t) = {I1(t), I2(t),..., I n (t)}

[0046] Among them, I(t) represents the image data corresponding to the nth spectral band at time t, t is the time variable with the unit of second, and n is the number of spectral bands.

[0047] Exemplarily, the image data stream includes two-dimensional image data (image frames) of the area to be detected and information of each spectral band. For example, the visible light band is used to obtain color and brightness information, the near-infrared band is used to obtain information related to the temperature of the object, and the short-wave infrared is used to identify features related to the material and composition of the object.

[0048] In an alternative embodiment, the expression form of the image data stream is a multi-dimensional data array, which is displayed in the form of image frames. Each frame of image contains multiple spectral channels. For example, if the image captured by the multispectral module has 5 spectral bands (visible light, near-infrared, short-wave infrared), then each frame of image consists of the values of 5 channels, and each channel represents the light intensity data of one band; this data stream is transmitted to the ground control station in real time through wireless communication for subsequent processing and analysis.

[0049] S2. Use the image quality assessment network to evaluate the brightness, contrast, and noise level of the collected image data stream, and perform noise suppression and optimization processing on it through an adaptive enhancement algorithm to generate a preprocessed image. It should be noted in this step that:

[0050] Use the image quality assessment network to evaluate the brightness, contrast, and noise level of the collected image data stream;

[0051] According to the evaluation results, use the adaptive enhancement algorithm to perform noise suppression, brightness, and contrast optimization on the image, enhance the visibility of the image and suppress noise;

[0052] Automatically adjust the parameters of the enhancement algorithm to ensure that the image details are not over-enhanced, and generate a clear, noise-free, and brightness- and contrast-optimized preprocessed image.

[0053] In an alternative embodiment, the image quality assessment network adopts a two-stream pyramid structure, including a spatial domain branch and a frequency domain branch. Among them, the spatial domain branch extracts luminance and contrast features through 5 convolutional layers (the convolutional kernel size is 3×3, and the number of channels is 16, 32, 64, 128, 256 respectively), and the frequency domain branch extracts noise spectrum features through fast Fourier transform and 3 complex convolutional layers (the convolutional kernel size is 5×5);

[0054] Furthermore, the features in the spatial domain and the frequency domain are fused through feature concatenation and inverse Fourier transform. The fusion process adopts the following formula:

[0055] F fusion = GeLU(Conv1x1([F spatial ; F -1 (F freq )))

[0056] Among them, F spatial and F freq respectively represent the features in the spatial domain and the frequency domain, F -1 is the inverse Fourier transform, [;] represents the feature concatenation operation, and GeLU is the activation function.

[0057] Exemplarily, the luminance is evaluated by calculating the normalized luminance value of a local area (such as a 16×16 pixel block). The luminance calculation formula is:

[0058]

[0059] Among them, L(x,y) is the luminance value at position (x,y), I R (i), I G (i), I B (i) respectively represent the pixel values of the red, green, and blue color channels, Ω xy represents the local area, and N is the number of pixels in the area.

[0060] Exemplarily, the contrast is evaluated through an improved Weber contrast formula:

[0061]

[0062] Among them, C(x,y) is the contrast at position (x,y), Ω xy (I) represents the pixel values in the area, and ∈ is the smoothing constant.

[0063] Exemplarily, the noise evaluation is the noise energy spectrum output by the frequency domain branch. The noise energy calculation formula is:

[0064]

[0065] Among them, N(f) is the noise energy at frequency component f, K is the number of sub-band divisions, and K = 8.

[0066] In an alternative embodiment, the adaptive enhancement algorithm includes a dynamic parameter adjustment strategy, and generates a brightness gain α, a contrast coefficient β, and a denoising intensity γ through a multi-layer perceptron;

[0067] Specifically, the dynamic parameter adjustment strategy generates an enhancement parameter matrix according to the brightness, contrast, and noise evaluation results:

[0068] [α, β, γ] = MLP([AvgPool(ML), MaxPool(MC), StdDev(N(f))])

[0069] Among them, MLP is a multi-layer perceptron, including 3 fully connected layers (the number of hidden units is 64, 32, and 3 respectively), and the inputs are the brightness matrix ML, the contrast matrix MC, and the standard deviation of the noise spectrum N(f).

[0070] Furthermore, the enhancement parameter constraint conditions include:

[0071] The brightness gain α ∈ [0.8, 1.2];

[0072] The contrast coefficient β ∈ [0.5, 2.0];

[0073] The denoising intensity γ ∈ [0, 1].

[0074] In this embodiment, noise suppression uses adaptive wavelet threshold denoising, brightness compensation is performed through local histogram matching, and contrast enhancement is implemented using an improved contrast-limited adaptive histogram equalization (CLAHE) algorithm, which can be carried out by means and methods in the prior art and will not be elaborated herein.

[0075] It should be further noted that the preprocessed image I pre is a three-channel tensor, and its range is [0, 255], representing the image after optimizing brightness and contrast:

[0076] I pre ∈ [0, 255] H×W×3

[0077] Among them, H and W are the height and width of the image respectively, and 3 represents three channels.

[0078] Exemplarily, the difference between the image preprocessing result and the ideal image is minimized through the optimization model, and at the same time, the enhancement parameters are optimized:

[0079]

[0080] Among them, I idealis the ideal image, and λ1 and λ2 are regularization coefficients.

[0081] S3. Build a feature extraction model based on the hierarchical attention mechanism, perform multi-scale feature extraction on the preprocessed image, and obtain a multi-level feature map. It should be noted that in this step:

[0082] Build a feature extraction model based on the hierarchical attention mechanism, perform feature extraction on the input preprocessed image, and enhance the key features in the image by adaptively adjusting the attention weights;

[0083] Perform multi-scale feature extraction on the preprocessed image, use convolution kernels and pooling operations of different sizes to extract multi-level features of the image;

[0084] The generated multi-level feature map includes image feature information of multiple scales and levels, representing local details, global information, and high-level semantic features of the image.

[0085] In an optional implementation manner, the hierarchical attention mechanism includes a hybrid attention module, generates attention weights through weighted fusion of a channel attention sub-module and a spatial attention sub-module. The multi-scale feature extraction adopts 4 parallel dilated convolution branches with dilation rates of 1, 2, 4, and 6 respectively, and realizes cross-scale feature interaction through bidirectional gated connections and a dynamic routing algorithm;

[0086] As an example, in order to extract features of different scales, through 4 parallel convolution branches, each branch uses a dilated convolution kernel with a different dilation rate (dilation rate = 1, 2, 4, 6), and each branch contains 3 convolution blocks. The hierarchy of each convolution block is as follows:

[0087] Block l = Conv(3×3, dr l ) → LayerNorm → GeLU → HA l

[0088] where dr l is the dilation rate of the l-th branch, and HA l is the hybrid attention module of this branch;

[0089] Furthermore, the channel attention sub-module:

[0090] Obtain the attention weights of the channels by calculating the weighted features of the input feature map after different pooling operations. The formula is:

[0091] CA(F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F))) ⊙ F

[0092] Among them, σ is the Sigmoid activation function, MLP is a multi-layer perceptron, which includes a two-layer fully connected network, the pooling operation includes average pooling and max pooling, and ⊙ is element-wise multiplication;

[0093] Furthermore, the spatial attention sub-module:

[0094] Calculate the spatial attention of the input feature map, and weight the space through convolution operation. The formula is:

[0095] SA(F) = σ(Conv(7×7, [MaxPool(F); AvgPool(F)])) ⊙ F

[0096] Among them, the kernel size of the convolution operation is 7×7, and the pooling operation is performed after splicing the features through max pooling and average pooling;

[0097] By fusing channel attention and spatial attention, a hybrid attention map is obtained:

[0098] HA l (F) = λ l ·CA(F) + (1 - λ l )·SA(F)

[0099] Among them, λ l is a learnable hierarchical balance parameter, λ l = 0.8, λ l+1 = 0.7 (exponentially decaying for each layer).

[0100] In an alternative embodiment, the input preprocessed image I ∈ R H×W×3 , and an initial feature map group is generated through a multi-branch convolution module:

[0101]

[0102] Among them, represents the feature map of the l-th layer, and the resolution of the initial feature map is H×W;

[0103] Perform four-level downsampling operations (depthwise separable convolution with a stride of 2), and the resolution of each level of feature map is reduced to 1 / 2 of the previous level.

[0104] Exemplarily, a bidirectional gated connection is established between each level of feature maps to enhance the interaction between features of different scales:

[0105]

[0106] Among them, G up is the gating weight, σ is the Sigmoid activation function, and UpSample is the upsampling operation.

[0107] Exemplarily, the feature transfer path is optimized through a routing algorithm, and the routing weight calculation formula is:

[0108]

[0109] where sim(F i , F j ) is the cosine similarity, and τ = 0.5 is the temperature coefficient.

[0110] In an alternative embodiment, the expression form of the multi-level feature map is to output four groups of feature tensors with different scales:

[0111]

[0112] Then, channel recombination is performed on each level of the feature map:

[0113] F lenhanced = DepthwiseConv(5×5, GroupNorm(GeLU(F l )))

[0114] where depth convolution is used to process each channel, and the number of groups is set to 1 / 4 of the number of channels;

[0115] Training is performed through a multi-objective loss function to ensure feature reconstruction and contrast optimization:

[0116]

[0117] where L recon is the feature reconstruction loss, L contrast is the contrast loss, γ l , μ l are the weight coefficients;

[0118] As an example, the feature reconstruction loss is the L1 distance between the image and the downsampled image:

[0119]

[0120] The contrast loss calculates the similarity between two feature maps to optimize the contrast ability of the model:

[0121]

[0122] where and are the feature maps of the positive sample and the negative sample respectively, and sim(·) is the cosine similarity;

[0123] When the contrast loss is greater than twice the feature reconstruction loss, adjust μ l : μ l ← μ l·0.7。

[0124] It should be noted that through the hierarchical attention mechanism and multi-scale feature extraction, the scale sensitivity problem caused by the change of the shooting height of the UAV image is solved. In the above steps, by adaptively adjusting the attention weights in the feature extraction process, the model can pay more attention to the key features in the image, while ensuring the consistency of the image at different scales. The generated multi-level feature maps provide a good processing basis for subsequent spatial alignment and object detection.

[0125] S4. Perform spatial alignment on the multi-level feature maps through the spatial transformation network, and combine the weighted fusion mechanism to fuse the multi-level feature maps to generate a comprehensive feature map with cross-scale information. Among them, it should be noted in this step that:

[0126] Generate an affine transformation parameter matrix through the parameter prediction sub-network, and use this matrix to perform spatial alignment on the multi-level feature maps;

[0127] Calculate the channel and spatial attention weights for each aligned feature map, and perform weighted fusion according to these weights to generate a comprehensive feature map with cross-scale information.

[0128] In an alternative embodiment, by learning the spatial transformation parameters of the input feature map, an affine transformation matrix is generated, so as to perform spatial alignment on feature maps of different scales. The affine transformation parameter matrix represents the transformations of rotation, scaling, and translation; and the parameter prediction sub-network uses a 3-layer convolutional neural network (CNN), and the convolutional kernel sizes of each layer of convolution are 5×5, 3×3, and 1×1 in sequence. Among them, the number of input channels is the number of channels of the feature map, and the number of output channels is 6, representing the 6 elements of the affine transformation matrix: rotation, scaling, and displacement parameters;

[0129] As an example, its mathematical expression is:

[0130]

[0131] Among them, a, b, c, d control rotation and scaling, t x ,t y control translation;

[0132] The structure of each layer of convolution:

[0133] The first layer: Conv(5×5, input channels → 16);

[0134] The second layer: Conv(3×3, 16 → 32);

[0135] The third layer: Conv(1×1, 32 → 6) (output transformation parameters).

[0136] Exemplarily, the input multi-level feature map is set as where L is the number of feature map layers, and the resolution of each layer is arranged in descending order as:

[0137]

[0138] For each feature map F l , the corresponding affine transformation matrix θ is generated through the parameter prediction sub-network l ;

[0139] Bilinear interpolation is used for alignment, and the transformed coordinates are:

[0140] (x′, y′) = θ l · [x, y, 1] T

[0141] where θ l is the affine transformation matrix;

[0142] The aligned feature map F laligned is generated using bilinear interpolation:

[0143] F laligned (x, y) = ∑ i,j F l (i, j) · max(0, 1 - |x′ - i|) · max(0, 1 - |y′ - j|)

[0144] where (x′, y′) are the transformed coordinates;

[0145] In an alternative embodiment, for each aligned feature map F laligned , the channel attention weight is calculated

[0146]

[0147] where σ is the Sigmoid activation function, and MLP is a multi-layer perceptron with one hidden layer (the dimensionality reduction rate is 16);

[0148] The spatial attention weight map is calculated

[0149]

[0150] where the convolution kernel is 3×3, and the feature maps F laligned and F 1-laligned are concatenated along the channel dimension;

[0151] Each layer of the feature map is uniformly upsampled to the highest resolution H×W;

[0152] The cross-scale integrated feature map is generated by weighted summation:

[0153]

[0154] Among them, F fusion is the fused feature map, and is the feature map after the first layer of alignment.

[0155] As an example, the cross-scale integrated feature map is output as a four-dimensional tensor:

[0156]

[0157] Among them, H and W are the height and width of the input image, C = 256 is the number of channels, aggregating the semantic information and detailed features of each level.

[0158] It should be noted that the Atrous Spatial Pyramid Pooling (ASPP) module is used to capture multi-scale context, and four parallel atrous convolutions (with dilation rates of 6, 12, 18, and 24 respectively) are adopted. After the output channels are concatenated, they are reduced in dimension through a 1×1 convolution.

[0159] It should also be noted that when the geometric consistency loss value L align is greater than the threshold τ (τ = 0.15), the last convolutional layer of the parameter prediction sub-network is re-initialized to ensure spatial consistency during the alignment process.

[0160] S5. Build an object segmentation detection model based on deformable convolution, perform object segmentation detection on the integrated feature map, and generate pixel-level segmentation masks and bounding box coordinates. Among them, it should be noted in this step that:

[0161] In an alternative implementation, the network structure of the object segmentation detection model includes an encoder, a decoder, a segmentation branch, and a detection branch, where: the encoder is composed of 4 deformable convolution blocks, each convolution block contains 2 deformable convolution layers, and the output channel numbers are 256, 512, 1024, and 2048 in sequence;

[0162] As an example, the deformable convolution formula:

[0163]

[0164] Among them, p represents the position coordinates of the output feature map, p k is the k-th position in the standard convolution sampling grid, Δp k is the learnable offset generated by the offset prediction network, w k is the weight of the convolution kernel, K is the size of the convolution kernel, and x is the pixel value of the input feature map.

[0165] The offset of the deformable convolution is predicted by the offset prediction sub-network, and this network is composed of two layers of 1×1 convolutions:

[0166] The number of output channels in the first layer is 2K, where K is the convolutional kernel size;

[0167] The second layer uses the ReLU activation function to generate a normalized offset;

[0168] The size of the convolutional kernel is set to 3×3, and the dilation coefficient is dynamically adjusted as needed. The initial offset is constrained within the range of ±1.5 pixels;

[0169] The decoder consists of a Feature Pyramid Network (FPN) and bilinear upsampling layers, and the upsampling ratios for each stage are 8x, 4x, 2x, 1x in sequence;

[0170] The segmentation branch uses transposed convolution to generate a pixel-level segmentation mask, and the generation formula is:

[0171] M seg =σ(Conv(1×1,Deconv(F fusion )))

[0172] where σ is the Sigmoid activation function, Deconv is the transposed convolution operation, and F fusion is the fused feature map;

[0173] The detection branch uses an anchor adaptive mechanism to generate the coordinates of the target bounding box, and the calculation formula is:

[0174] (x,y,w,h)=φ(Conv(3×3,F fusion ))

[0175] where φ is the coordinate regressor, and the output coordinates are normalized by the tanh function. x and y are the center positions of the bounding box, and w and h are the width and height of the bounding box;

[0176] Furthermore, perform deformable RoI pooling on the feature map, use a gated attention mechanism to fuse feature maps at different levels, and synchronously optimize the segmentation and detection tasks through a multi-task loss function, where:

[0177] Perform deformable RoI pooling on the input integrated feature map, and the calculation formula is:

[0178]

[0179] where Δx i ,Δy i are the learnable offsets for each pooling bin, bin i is the i-th pooling bin, and n is the number of pooling bins;

[0180] Use a gated attention mechanism to fuse feature maps at different levels, and the weight calculation formula is:

[0181] w l = softmax(MLP(AvgPool(F l )))

[0182] where softmax is a normalization operation, MLP is a multi-layer perceptron, and the output weight w l is used for weighted fusion;

[0183] The segmentation and detection tasks are simultaneously optimized through a multi-task loss function. The calculation formula of the loss function is:

[0184] L total = δ1L dice + δ2L smooth-L1 + δ3||Δp|| 2

[0185] where L dice is the Dice loss, used for segmentation, L smooth-L1 is the smooth L1 loss, used for bounding box regression, ||Δp|| 2 is the regularization term, and δ1, δ2, and δ3 are balance coefficients.

[0186] As an example, through the implementation of the above steps, a pixel-level segmentation mask and bounding box coordinates are finally generated, where:

[0187] Pixel-level segmentation mask: The output is a two-dimensional probability matrix M ∈ [0, 1] H×W , and a binary mask is generated after thresholding;

[0188] Bounding box coordinates: The output format is (x center , y center , width, height), which is converted to the original image coordinate system after denormalization;

[0189] And overlapping detection boxes are eliminated through non-maximum suppression (NMS) (IoU threshold is set to 0.5), and the continuity of the edges of the segmentation mask is optimized through morphological closing operations to ensure that the generated bounding boxes completely contain the target segmentation region.

[0190] It should be noted that through deformable convolutions and a multi-task detection network, accurate segmentation of the target and bounding box detection are achieved. The deformable convolution kernel processes the comprehensive feature map, dynamically adjusts the offset of the convolution kernel, uses a gated attention mechanism to fuse multi-level feature maps, and through the joint loss function, the segmentation and detection tasks are synchronously optimized. Finally, the generated pixel-level segmentation mask and bounding box coordinates provide accurate target detection outputs.

[0191] Preferably, through the synergistic effect of the above steps, the embodiments of the present invention provide an efficient and robust method for UAV target segmentation and detection. With the support of the multispectral imaging module, high-quality images can be obtained in real time in various complex environments. Through image quality optimization, feature fusion, and spatial alignment, the image processing process is optimized from different angles, and it closely cooperates with the subsequent steps to jointly achieve efficient target segmentation and detection in complex environments, greatly improving the adaptability and performance of UAVs in practical applications.

[0192] Preferably, for the agricultural pest and disease monitoring scenario, to verify the improvement effect of the target detection accuracy of the UAV target segmentation and detection method based on large model visual perception proposed by the present invention under complex lighting conditions, in this embodiment, a multi-scenario dataset is constructed to compare the differences between the traditional method and the method of the present invention in key indicators;

[0193] The traditional method has the following defects: using fixed-threshold image enhancement leads to overexposure or under-enhancement, it is difficult to capture small-size targets based on single-scale feature extraction, and the rigid convolution kernel cannot adapt to the morphological diversity of crops;

[0194]

Experimental Environment Configuration

[0195] Hardware platform: DJI Matrice 300RTK UAV, equipped with RedEdge-MX multispectral camera (5 bands: 475nm / 560nm / 668nm / 717nm / 840nm), flight altitude 80m, ground resolution 6.8cm / pixel;

[0196] Software environment: PyTorch 1.12 framework, CUDA 11.6 acceleration, Intel Xeon Gold 6248R processor;

[0197] Test dataset: 8526 images collected from 3 typical agricultural areas in Jiangsu Province, including:

[0198] Noon strong light (sleep intensity > 120klux): 2784 images;

[0199] Twilight weak light (dark intensity < 20klux): 1935 images;

[0200] Cloudy diffused light: 3807 images;

[0201] Disease types included: 6 common disease characteristics such as wheat scab, rice sheath blight, powdery mildew, gray mold, leaf spot, and rice blast.

[0202]

Implementation Process

[0203] Traditional method:

[0204] Enhancement is performed using histogram equalization to improve the dynamic recovery of the image;

[0205] The SIFT (Scale-Invariant Feature Transform) algorithm is used to extract key point features;

[0206] The method of pyramid feature fusion is adopted to fuse features of different dimensions to enhance the recognition ability of multi-dimensional targets;

[0207] Object detection is carried out based on the detection model of Faster R-CNN.

[0208] The method of the present invention:

[0209] An image quality evaluation network containing three Inception modules is constructed to evaluate indicators such as the noise distribution entropy, noise ratio, and recovery dispersion of the image, providing a reference for subsequent image processing;

[0210] According to the image quality evaluation results, the parameters of the adaptive enhancement algorithm are dynamically adjusted to optimize the brightness, gain, and noise suppression effects of the image;

[0211] The ResNet-50 backbone network is adopted, and multi-dimensional feature extraction is carried out through a 4-level feature pyramid combined with an emotion module;

[0212] Using a spatial transformation network, spatial alignment of the feature map is performed through thin plate spline transformation to reduce the spatial tray between feature maps, thereby improving the target localization accuracy;

[0213] Based on Mask R-CNN, by learning the deformation (offset) of the target at different positions, the sampling area of the contour operation is adjusted, and the offset of each pixel is self-learned, so that the sampling area with a clear contour can better match the actual shape of the target.

[0214] The generated experimental data is recorded as follows:

[0215] Table 1. Experimental data record table

[0216]

[0217] Referring to Table 1, it can be intuitively seen that the method of the present invention is significantly superior to the traditional method in various indicators. In the strong light scenario, the average precision (AP) is increased from 67.05% of the traditional method to 92.65%, with an increase of 38.2%. This is due to the fact that the adaptive enhancement algorithm stabilizes the noise suppression index above 24 dB, which is 6 - 8 dB higher than the traditional fixed parameter method. The feature consistency index reflects the multi-scale feature fusion effect. The present invention reaches 84.23%, while the traditional method is 62.75%, proving that the hierarchical attention mechanism effectively maintains the semantic coherence of cross-scale features.

[0218] Under low-light conditions, the present invention still maintains an AP value of 83.05%, which is more than 100% higher than that of the traditional method (40.5%). The spatial transformation network controls the feature alignment error within 4.3 pixels, reducing the error by 44.9% compared to the 7.8-pixel error of the traditional SIFT method. The learning of the deformation parameters of the deformable convolution increases the detection rate of small targets (<50 pixels) from 31.6% to 79.4%, overcoming the geometric limitations of the traditional rigid convolution kernel.

[0219] In terms of processing efficiency, the average time consumption of the present invention is 0.43 s / frame, which is 30.6% higher than that of the traditional method (0.62 s / frame). This is because the image quality assessment network reduces the number of ineffective enhancement iterations by 83%, the weighted fusion mechanism reduces the feature dimension by 40%, and the deformable convolution reduces the search area by 65% through deformation prediction.

[0220] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A method for UAV target segmentation and detection based on large model visual perception, characterized in that: include: The image data stream of the area to be inspected is obtained in real time through the multispectral imaging module carried by the drone; Using an image quality assessment network to assess the brightness, contrast and noise level of the collected image data stream, and using an adaptive enhancement algorithm to suppress noise and optimize the image data stream to generate a pre-processed image; Building a feature extraction model based on a hierarchical attention mechanism, performing multi-scale feature extraction on the preprocessed image to obtain a multi-level feature map; The multi-level feature maps are spatially aligned through a spatial transformation network, and the multi-level feature maps are fused in combination with a weighted fusion mechanism to generate a comprehensive feature map with cross-scale information; An object segmentation detection model is constructed based on deformable convolution, object segmentation detection is performed on the comprehensive feature map, and pixel-level segmentation masks and bounding box coordinates are generated.

2. The method for UAV target segmentation and detection based on large model visual perception according to claim 1 is characterized in that: The multispectral imaging module includes multiple spectral band sensors capable of capturing visible light, near infrared and short-wave infrared band information; The image data stream is updated at a frequency of greater than 30 frames per second, and automatically adjusts exposure time, gain and focus parameters to adapt to flight conditions.

3. The method for UAV target segmentation and detection based on large model visual perception according to claim 1 is characterized in that: The image quality assessment network adopts a dual-stream pyramid structure, including a spatial domain branch and a frequency domain branch, wherein the spatial domain branch extracts brightness and contrast features through 5 convolutional layers, and the frequency domain branch extracts noise spectrum features through fast Fourier transform and complex convolution; The adaptive enhancement algorithm includes a dynamic parameter adjustment strategy, and generates brightness gain α, contrast coefficient β and denoising strength γ through a multi-layer perceptron, with constraints of α∈[0.8,1.2], β∈[0.5,2.0] and γ∈[0,1].

4. The method for UAV target segmentation and detection based on large model visual perception according to claim 1 is characterized in that: The hierarchical attention mechanism includes a hybrid attention module, which generates attention weights through weighted fusion of a channel attention submodule and a spatial attention submodule; The multi-scale feature extraction adopts 4 parallel dilated convolution branches with expansion rates of 1, 2, 4, and 6 respectively, and realizes cross-scale feature interaction through bidirectional gating connection and dynamic routing algorithm.

5. The method for UAV target segmentation and detection based on large model visual perception according to claim 1 is characterized in that: The spatial transformation network generates an affine transformation matrix θ through a parameter prediction subnetwork, including rotation, scaling, and translation parameters; The weighted fusion mechanism dynamically weights and sums the aligned feature maps through channel attention weights and spatial attention weights, and combines the dilated spatial pyramid pooling module to enhance cross-scale semantic information.

6. The method for UAV target segmentation and detection based on large model visual perception according to claim 1 is characterized in that: The deformable convolution generates a dynamic offset Δpk through an offset prediction network, with an initial constraint range of ±1.5 pixels; The target segmentation detection model adopts an encoder-decoder structure. The encoder consists of 4 deformable convolution blocks, and the decoder generates segmentation masks and bounding boxes through a feature pyramid network and bilinear upsampling.

7. The method for UAV target segmentation and detection based on large model visual perception according to claim 6 is characterized in that: The segmentation mask is generated by transposed convolution and sigmoid activation function, and the coordinates of the bounding box are generated by anchor point adaptation mechanism and tanh normalized regression.

8. The method for UAV target segmentation and detection based on large model visual perception according to claim 7 is characterized in that: The segmentation mask is morphologically closed to optimize edge continuity, and overlapping bounding boxes are eliminated by non-maximum suppression, and the IoU threshold is set to 0.5.

Images