Unmanned aerial vehicle auxiliary photographing correction method based on improved RT-DETR model

Abstract

The invention discloses an unmanned aerial vehicle auxiliary photographing correction method based on an improved RT-DETR model. The method comprises the steps of obtaining target image data; constructing an RT-DETR-V1 detection model, and inputting the target image data into the RT-DETR-V1 detection model, so as to obtain a first target RT-DETR-V1 detection model; constructing a multi-task joint loss function to obtain total loss; the first target RT-DETR-V1 detection model is optimized and adjusted on the basis of the total loss, and a second target RT-DETR-V1 detection model is obtained; and carrying the second target RT-DETR-V1 detection model on the unmanned aerial vehicle. On the basis of an RT-DETR-V1 detection model, in combination with a dynamic anchor point correction mechanism and contrast loss optimization, open set detection and cross-modal feature depth alignment without annotation data are achieved, the angle, the focal length multiple and exposure parameters of a holder are adjusted in real time through an unmanned aerial vehicle dynamic regulation and control system, and the accuracy and the reliability of the system in the complex illumination and shielding environment are improved. The unmanned aerial vehicle can identify the inspection image accurately; and the auxiliary deviation correction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicles (UAVs), and in particular to a UAV-assisted photography correction method based on an improved RT-DETR model. Background Art

[0002] With the rapid development of smart grids, drones are increasingly being used for power inspections. The safe operation of power equipment relies on high-precision image data. However, drone photography can be challenging due to issues like unstable flight attitude, changing ambient lighting, and equipment obstruction. This can lead to image distortion, blurring, or missing information, severely impacting the accuracy of subsequent defect detection and 3D modeling. Furthermore, traditional manual adjustment of flight parameters is inefficient and reliant on experience, making it difficult to adapt to the real-time dynamic demands of complex scenarios. Therefore, an intelligent drone photography correction method is urgently needed to improve the adaptability and image data quality of power inspections.

[0003] Early drone camera correction technologies primarily relied on single-modal vision sensors, using image feature matching or geometric transformations to achieve camera angle calibration. However, these methods are prone to failure when lighting changes dramatically, when equipment textures are missing, or when obstructions occur. In recent years, multimodal fusion technologies combining vision and lidar have improved environmental perception capabilities, but their high hardware costs and system complexity have hindered widespread adoption. Furthermore, deep learning-based visual inspection can achieve image correction through end-to-end training models, but this relies on large amounts of labeled data and suffers from insufficient generalization capabilities. These technical limitations still require manual intervention in drone power inspections, hindering the development of intelligent inspections. Summary of the Invention

[0004] The present invention provides a drone-assisted photography correction method based on an improved RT-DETR model. Based on the RT-DETR-V1 detection model, the method combines a dynamic anchor point correction mechanism with contrast loss optimization to achieve open set detection and cross-modal feature depth alignment without the need for labeled data. The method also adjusts the pan / tilt angle, focal length multiple, and exposure parameters in real time through the drone's dynamic control system, thereby improving the drone's recognition accuracy of inspection images and the accuracy of assisted correction in complex lighting and occlusion environments.

[0005] The technical solution adopted by the present invention is:

[0006] A UAV-assisted photography correction method based on an improved RT-DETR model includes:

[0007] S1. Acquire target image data;

[0008] S2. Build an RT-DETR-V1 detection model, input the target image data into the RT-DETR-V1 detection model, and obtain a first target RT-DETR-V1 detection model; wherein the RT-DETR-V1 detection model includes an image encoder, a visual cue encoder, a text cue encoder, and an output decoder;

[0009] S3. Construct a multi-task joint loss function to obtain the total loss;

[0010] S4. Optimize and adjust the first target RT-DETR-V1 detection model based on the total loss to obtain the second target RT-DETR-V1 detection model with efficient visual-semantic joint reasoning;

[0011] S5. Mount the second target RT-DETR-V1 detection model on the drone. The second target RT-DETR-V1 detection model is used to assist the drone in taking photos and correcting the deviation.

[0012] S51. During flight, the drone collects raw image data in real time through the onboard camera, analyzes ambient lighting conditions, and triggers the automatic dimming module to adjust exposure parameters to ensure that the quality of the input raw image data meets the detection requirements;

[0013] S52: The drone flies to the cross-arm side of the tower head and performs target inspection in a left-to-right and top-to-bottom order, including inspecting the cross-arm side components connecting the tower body and insulators, the insulators, and the conductor side components connecting the insulators and conductors.

[0014] S53: Identify and judge the target detection, and perform auxiliary photo correction:

[0015] 1) If the target is identified, the drone system automatically performs the following steps:

[0016] ① Based on the maximum viewing angle of the camera and the ratio of the center deviation of the target frame to the screen, the deviation angle is calculated and the gimbal rotation is corrected;

[0017] ② Calculate the zoom ratio based on the current size and target size, and then adjust the focal length to assist the drone in taking photos;

[0018] 2) If the target is not recognized, wait until the grayscale value of the camera image stabilizes, then calculate the grayscale value of the current camera image, and then calculate the grayscale value of the camera image in a loop at equal intervals. Adjust the grayscale value of the target area to 125, set the EV value and adjust the exposure, and then recognize again;

[0019] 3) When the set drone adjustment time is reached, the loop calculation is exited and the drone-assisted photo correction ends.

[0020] Furthermore, step S1 includes the following specific steps:

[0021] S11, obtaining raw image data collected by the drone through an onboard camera inspection during flight;

[0022] S12, performing denoising and distortion correction processing on the collected original image data in sequence to obtain first image data;

[0023] S13. Use the Labelme image annotation tool to annotate the first image data with a rectangular frame, and add category text information to obtain target image data, where the target image data includes multimodal text, frame, and point data.

[0024] Furthermore, step S2 includes the following specific steps:

[0025] S2. Build an RT-DETR-V1 detection model, input the target image data into the RT-DETR-V1 detection model for training, and obtain a first target RT-DETR-V1 detection model; wherein the RT-DETR-V1 detection model includes an image encoder, a visual cue encoder, a text cue encoder, and an output decoder;

[0026] S21. Obtain a multi-scale feature map based on the image encoder in the RT-DETR-V1 detection model;

[0027] S22, based on the visual cue encoder in the RT-DETR-V1 detection model, obtain the visual cue embedding feature;

[0028] S23, based on the text prompt encoder in the RT-DETR-V1 detection model, obtain the text prompt embedding feature;

[0029] S24. Based on the output decoder in the RT-DETR-V1 detection model, the predicted target box, predicted category label and text understanding output are obtained.

[0030] Furthermore, step S21 includes the following specific steps:

[0031] 1) Input the target image data into the image encoder in the RT-DETR-V1 detection model;

[0032] 2) Based on the YOLOv11 feature extraction network in the image encoder, multi-scale feature extraction is performed on the target image data to obtain a multi-scale feature map;

[0033] Wherein, the multi-scale feature map is recorded as:

[0034]

[0035] Where C, H, and W are the length, height, and width of the multi-scale feature map, respectively; L is the number of multi-scale feature layers; and i is the index of the feature layer number, which is used to distinguish feature maps of different scales.

[0036] Furthermore, step S22 includes the following specific steps:

[0037] 1) For the two visual cue types, box and point, two different coordinate encodings are established and the dimensions are unified:

[0038] ① Given K user-specified 4D normalized box coordinates on the target image data

[0039] b j =(x j ,y j ,w j ,h j ),j∈{1,2,...,K} and 2D normalized point coordinates p j =(x j ,y j ),j∈{1,2,...,K};

[0040] ② Establish differentiated encoding methods for the 4D normalized frame coordinates and the 2D normalized point coordinates to obtain two coordinate encodings;

[0041] ③ Encode the two coordinates into position embeddings through a fixed sine-cosine embedding layer;

[0042] ④Use two different linear layers to project the position embedding into a unified dimension B:

[0043] B=Linear(PE(b1,…,b k );θ B ):R K×4D →R K×D

[0044] B=Linear(PE(p1,…,p k );θ B ):R K×2D →R K×D

[0045] Where Linear(.;θ) represents a linear projection operation with parameter θ; PE represents position embedding; K is the number of prompts; D is the embedding dimension;

[0046] 2) Model the boxes and points as different types of visual cues, initialize a learnable content embedding, and expand it to match the number of cues K through a broadcast operation, denoted as C∈R K×D ; In addition, a general category label C′∈R is introduced 1×D, used to aggregate features across multiple visual cues to accommodate scenarios where users provide multiple visual cues in a single image;

[0047] 3) Concatenate the content embedding and the position embedding along the channel dimension to obtain a joint embedding;

[0048] 4) Apply a linear projection layer to map the joint embedding to the query space, obtaining the input query embedding feature Q:

[0049]

[0050] Where, Represents a parameter Linear projection operation; CAT represents the merging of channel dimensions, B′ and P′ represent global position embeddings, which are derived from the global normalized coordinates [0.5, 0.5, 1, 1] and 0.5, 0.5;

[0051] 5) Based on the multi-scale deformable cross attention layer, the visual cue features are extracted from the multi-scale feature map under the condition of visual cue:

[0052] For the jth visual cue, the query feature Q is obtained after the multi-scale deformable cross attention layer calculation j ′:

[0053]

[0054] Where Q j ′ is the query feature of the j-th visual cue after being calculated by the multi-scale deformable cross attention layer, MSDeformAttn is the multi-scale deformable cross attention layer function, which is used to extract features conditioned on the visual cue from the multi-scale feature map; Q j is the jth original query feature; b j The 4D normalized box coordinates specified by the jth user, indicating the location and size information of the box; is a multi-scale feature map set composed of the j-th layer feature map, which is used to provide image features of different scales; p j The 2D normalized point coordinates specified by the jth user, indicating the location information of the point;

[0055] 6) Based on the extracted visual cue features, a self-attention layer is used to model the dependencies between different queries, and a feed-forward layer FFN is used for projection to obtain the global query features, which are used as the visual cue embedding features V for fully representing the semantic and spatial information of the target area:

[0056] V = FFN(SelfAttn(Q′))[-1]

[0057] Where FFN is the feedforward neural network layer; SelfAttn is the self-attention layer; Q′ is the visual cue feature; [-1] is the last element of the feedforward layer output sequence as the final feature representation.

[0058] Furthermore, step S23 includes the following specific steps:

[0059] 1) Select a pre-trained CLIP text encoder; the CLIP text encoder is used to encode category names or descriptive phrases;

[0060] 2) Use the hidden state corresponding to the [CLS] token to extract text prompt features from the multi-scale feature map to obtain global semantic features, which are used as text prompt features;

[0061] 3) Normalize the global semantic features to obtain the text hint embedding feature T.

[0062] Furthermore, step S24 includes the following specific steps:

[0063] 1) Predicting the target box:

[0064] Each detection query is represented as a 4D anchor coordinate (x, y, w, h), and the position is fine-tuned in the decoder layer. The decoder layer is the decoder of the RT-DETR-V1 model, which is responsible for combining the features extracted by the encoder with the hint embedding provided by the user to generate the final detection result.

[0065] ② Each anchor point in the decoder feature layer is initialized with a coordinate, and the similarity between the encoder feature and the visual cue embedding feature and the text cue embedding feature is calculated;

[0066] ③Select the first 300 similarity indices corresponding to each hint embedding to generate the initial position embedding;

[0067] ④ After multiple transformer decoder layers, the offset of the cumulative predicted anchor point (Δx, Δy, Δw, Δh) is summed with the initial anchor point, and the predicted target box is output:

[0068] (Δx, Δy, Δw, Δh)=MLP(Q dec )

[0069] Box=(x+Δx,y+Δy,v+Δw,h+Δh)

[0070] Where, MLP is a multi-layer perceptron; Q dec is the predicted query from the Box decoder;

[0071] 2) Predicting category labels:

[0072] Use visual cue embedding features or text cue embedding features as the weights of the classification layer. By updating the cue embedding library online, you can support the detection of new categories without retraining:

[0073]

[0074] Where C represents the total number of visual cue categories and N represents the number of detection queries; both visual cue and text object detection tasks share the same image encoder and box decoder for predicting target boxes.

[0075] 3) Text understanding output:

[0076] ① The RoIAlign algorithm is used to extract local features of the detected target from the feature map extracted by the YOLOv11 feature extraction network. RoIAlign extracts local features from the feature map based on the anchor box for target classification and regression; the visual cue embedding features and the text cue embedding features are combined to form object labels;

[0077] ② Align the visual cue embedding features with the text cue embedding features through linear projection to ensure the effectiveness of cross-modal interaction;

[0078] ③ The lightweight language decoder cross-modally fuses the visual cue embedding features and the text cue embedding features in an autoregressive manner and outputs cross-modal joint features.

[0079] Furthermore, step S3 includes the following specific steps:

[0080] S31, use the mean absolute error (L1 Loss) and generalized intersection-over-union loss (GIOU Loss) to perform box regression from the two dimensions of pixel-level offset and geometric shape similarity, and obtain the box regression loss L of the target box box , used to accurately optimize the position and size of the target box:

[0081] L box =λ1L L1 (B pred ,B gt )+λ2L GIOU (B pred ,B gt )

[0082] Where, L box is the box regression loss function of the target box; L L1 is the mean absolute error; L GIOU is the generalized intersection-over-union loss; λ1 and λ2 are hyperparameters, B pred is the predicted box coordinate, B gt is the real frame coordinate;

[0083] S32, the classification task uses contrast loss to measure the difference between the target and the embedding. It consists of two independent task heads, visual cue embedding and text cue embedding, which calculate the similarity with the detection query respectively and optimize it with FocalLoss to obtain the total classification loss L cls (General classification, because the classification includes two categories: visual and text), among which:

[0084] 1) The classification head is predicted by calculating the dot product between each detection query feature and the visual cue embedding feature through the dot product calculation of the image feature, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the visual cue embedding feature

[0085]

[0086] Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query, V j Embedding for matching visual cues;

[0087] 2) At the same time, the dot product prediction classification head is calculated between each detection query and text embedding of the image feature by dot product calculation, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the text hint embedding feature

[0088]

[0089] Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query; T k Prompt embed for matching text;

[0090] 3) Classification loss based on detection query features and visual cue embedding features and the classification loss of the detection query features and the text hint embedding features Calculate and get the total classification loss L cls :

[0091]

[0092] Where, L cls is the total classification loss; They are the classification loss between the detection query features and the visual cue embedding features and the classification loss between the detection query features and the textual cue embedding features;

[0093] S33. Calculate the contrast loss between the visual cue embedding features and the text cue embedding features to obtain the cross-modal contrast loss L align ;

[0094] Specifically, given an input image and K visual cue embedding features V = (v1,...,v K ) and the text hint embedding features T = (t1,...,t K ), calculate the InfoNCE loss between the visual cue embedding feature and the text cue embedding feature to improve the visual cue embedding feature v i and its corresponding text hint embedding feature t i Similarity, reduction and other text embeddings t j The similarity of is used to achieve the fusion of visual prompt features and text prompt features:

[0095]

[0096] Where, L align is the contrast loss function, v i ·t i Embedding feature v for the i-th visual cue i and the corresponding text hint embedding feature t i The dot product of the two measures the similarity, t j are other textual cue embedding features, and K are visual-textual cue feature pairs.

[0097] S34, box regression loss L based on target box box , total classification loss L cls and cross-modal contrast loss L align Calculate and get the total loss L of the multi-task joint loss function loss :

[0098] L loss =L box +L cls +L align .

[0099] Compared with the prior art, the present invention has the following beneficial effects:

[0100] 1) A multimodal interactive input method of "text + box selection + click selection" is used to achieve deep visual-semantic alignment. Users can dynamically specify detection targets by drawing a box, clicking, or entering text information. There is no need to predefine fixed categories or retrain models, significantly reducing the cost of adapting to new scenarios.

[0101] 2) Deeply integrate visual spatial information with textual semantic information to improve the model's adaptability to complex conditions such as lighting changes and device occlusion, significantly enhancing the robustness of detection capabilities;

[0102] 3) InfoNCE loss is used to enhance the similarity between visual cue embedding and corresponding text embedding, suppress irrelevant feature interference, and improve the recognition stability of the model under lighting changes and occlusion scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0104] Figure 1 This is a flow chart of a method for correcting deviation of a UAV-assisted photograph based on an improved RT-DETR model according to the present invention;

[0105] Figure 2 It is a schematic diagram of the composition structure of the RT-DETR-V1 detection model in the present invention. DETAILED DESCRIPTION

[0106] Example

[0107] like Figures 1 to 2 As shown in FIG, a UAV-assisted photography correction method based on an improved RT-DETR model includes:

[0108] S1. Acquire target image data;

[0109] Specifically, S11, obtaining original image data collected by the drone through an onboard camera inspection during flight;

[0110] S12, performing denoising and distortion correction processing on the collected original image data in sequence to obtain first image data;

[0111] S13, using the Labelme image annotation tool to annotate the first image data with a rectangular frame, and adding category text information to obtain target image data, wherein the target image data includes multimodal text, frame, and point data;

[0112] S2. Build an RT-DETR-V1 detection model, input the target image data into the RT-DETR-V1 detection model, and obtain a first target RT-DETR-V1 detection model; wherein the RT-DETR-V1 detection model includes an image encoder, a visual cue encoder, a text cue encoder, and an output decoder;

[0113] S21, based on the image encoder in the RT-DETR-V1 detection model, obtain a multi-scale feature map;

[0114] Specifically, 1) the target image data is input into the image encoder in the RT-DETR-V1 detection model;

[0115] 2) Based on the YOLOv11 feature extraction network in the image encoder, multi-scale feature extraction is performed on the target image data to obtain a multi-scale feature map;

[0116] Wherein, the multi-scale feature map is recorded as:

[0117] f i ∈RCi×Hi×Wi,i∈{1,2,...,L}

[0118] Where C, H, and W are the length, height, and width of the multi-scale feature map, respectively; L is the number of multi-scale feature layers; i is the index of the feature layer number, which is used to distinguish feature maps of different scales;

[0119] In this embodiment, if the input size of an image in the target image data is [8, 640, 640, 3], the output size of the target feature map obtained by the YOLOv11 feature extraction network is [8, 512, 8400]; where 8 represents the batch size, 640×640×3 is the input size of an image in the target image data, 512 is the number of feature channels, and 8400 is the total number of predicted positions of feature maps of different scales;

[0120] S22, based on the visual cue encoder in the RT-DETR-V1 detection model, obtain the visual cue embedding feature;

[0121] Specifically, 1) for the two visual cue types of box and point, two different coordinate encodings are established and the dimensions are unified:

[0122] ① Given K user-specified 4D normalized box coordinates on the target image data

[0123] b j =(x j ,y j ,w j ,h j ),j∈{1,2,...,K} and 2D normalized point coordinates p j =(x j ,y j ),j∈{1,2,...,K};

[0124] ② Establish differentiated encoding methods for the 4D normalized frame coordinates and the 2D normalized point coordinates to obtain two coordinate encodings;

[0125] ③ Encode the two coordinates into position embeddings through a fixed sine-cosine embedding layer;

[0126] ④Use two different linear layers to project the position embedding into a unified dimension B:

[0127] B=Linear(PE(b1,…,b k );θ B ):R K×4D →R K×D

[0128] B=Linear(PE(p1,…,p k );θ B ):R K×2D →R K×D

[0129] Where Linear(.;θ) represents a linear projection operation with parameter θ; PE represents position embedding; K is the number of prompts; D is the embedding dimension;

[0130] 2) Model the boxes and points as different types of visual cues, initialize a learnable content embedding, and expand it to match the number of cues K through a broadcast operation, denoted as C∈R K×D ; In addition, a general category label C′∈R is introduced 1×D , used to aggregate features across multiple visual cues to accommodate scenarios where users provide multiple visual cues in a single image;

[0131] 3) Concatenate the content embedding and the position embedding along the channel dimension to obtain a joint embedding;

[0132] 4) Apply a linear projection layer to map the joint embedding to the query space, obtaining the input query embedding feature Q:

[0133]

[0134] Where, Represents a parameter Linear projection operation; CAT represents the merging of channel dimensions, B′ and P′ represent global position embeddings, which are derived from the global normalized coordinates [0.5, 0.5, 1, 1] and 0.5, 0.5;

[0135] 5) Based on the multi-scale deformable cross attention layer, the visual cue features are extracted from the multi-scale feature map under the condition of visual cue:

[0136] For the jth visual cue, the query feature Q is obtained after the multi-scale deformable cross attention layer calculation j ′:

[0137]

[0138] Where Qj ′ is the query feature of the j-th visual cue after being calculated by the multi-scale deformable cross attention layer, MSDeformAttn is the multi-scale deformable cross attention layer function, which is used to extract features conditioned on the visual cue from the multi-scale feature map; Q j is the jth original query feature; b j The 4D normalized box coordinates specified by the jth user, indicating the location and size information of the box; is a multi-scale feature map set composed of the j-th layer feature map, which is used to provide image features of different scales; p j The 2D normalized point coordinates specified by the jth user, indicating the location information of the point;

[0139] Description: The perception range of deformable attention is limited to the coordinate neighborhood of the visual cue. For each query, a limited set of multi-scale features surrounding the visual cue is selectively focused on, ensuring that the visual cue of the target area is captured.

[0140] 6) Based on the extracted visual cue features, a self-attention layer is used to model the dependencies between different queries, and a feed-forward layer FFN is used for projection to obtain the global query features, which are used as the visual cue embedding features V for fully representing the semantic and spatial information of the target area:

[0141] V = FFN(SelfAttn(Q′))[-1]

[0142] Where FFN is the feedforward neural network layer; SelfAttn is the self-attention layer; Q′ is the visual cue feature; [-1] is the last element of the feedforward layer output sequence as the final feature representation.

[0143] In this embodiment, the coordinates of the annotated box in the annotation file, i.e., the target image data obtained in S1, contain the coordinate information of the target annotated box, which is extracted from [8,n,4] or [8,n,2] to [8,n,512] by the visual cue encoder; where n is the number of GT box categories in the current batch, 4 is the coordinates (x,y,w,h) of the box, 2 is the coordinates (x,y) of the point, and 512 is the dimension of the visual cue embedding feature after feature extraction by the visual cue encoder;

[0144] S23. Based on the text prompt encoder in the RT-DETR-V1 detection model, the text prompt embedding feature is obtained:

[0145] Specifically, 1) a pre-trained CLIP text encoder is selected; wherein the CLIP text encoder is used to encode category names or descriptive phrases;

[0146] 2) Use the hidden state corresponding to the [CLS] token to extract text prompt features from the multi-scale feature map to obtain global semantic features, which are used as text prompt features;

[0147] 3) Normalize the global semantic features to obtain the text hint embedding feature T;

[0148] In this example, three text categories, "insulator," "conductor side," and "crossarm side," are set as text category inputs. The pre-trained CLIP text encoder is used to extract features from the text to 512 dimensions: [8, 3, 512], where 8 represents the number of batches, 3 represents the number of categories, and 512 represents the feature dimension of the text hint embedding after extraction by the CLIP text encoder.

[0149] S24. Based on the output decoder in the RT-DETR-V1 detection model, obtain the predicted target box, predicted category label and text understanding output;

[0150] Specifically, 1) predict the target box:

[0151] Each detection query is represented as a 4D anchor coordinate (x, y, w, h), and the position is fine-tuned in the decoder layer. The decoder layer is the decoder of the RT-DETR-V1 model, which is responsible for combining the features extracted by the encoder with the hint embedding provided by the user to generate the final detection result.

[0152] ② Each anchor point in the decoder feature layer is initialized with a coordinate, and the similarity between the encoder feature and the visual cue embedding feature and the text cue embedding feature is calculated;

[0153] ③Select the first 300 similarity indices corresponding to each hint embedding to generate the initial position embedding;

[0154] ④ After multiple transformer decoder layers, the offset of the cumulative predicted anchor point (Δx, Δy, Δw, Δh) is summed with the initial anchor point, and the predicted target box is output:

[0155] (Δx, Δy, Δw, Δh)=MLP(Q dec )

[0156] Box=(x+Δx,y+Δy,v+Δw,h+Δh)

[0157] Where, MLP is a multi-layer perceptron; Q dec is the predicted query from the Box decoder;

[0158] 2) Predicting category labels:

[0159] Use visual cue embedding features or text cue embedding features as the weights of the classification layer. By updating the cue embedding library online, you can support the detection of new categories without retraining:

[0160]

[0161] Where C represents the total number of visual cue categories and N represents the number of detection queries; both visual cue and text object detection tasks share the same image encoder and box decoder for predicting target boxes.

[0162] 3) Text understanding output:

[0163] ① The RoIAlign algorithm is used to extract local features of the detected target from the feature map extracted by the YOLOv11 feature extraction network. RoIAlign extracts local features from the feature map based on the anchor box for target classification and regression; the visual cue embedding features and the text cue embedding features are combined to form object labels;

[0164] ② Align the visual cue embedding features with the text cue embedding features through linear projection to ensure the effectiveness of cross-modal interaction;

[0165] ③ The lightweight language decoder cross-modally fuses the visual cue embedding features and the text cue embedding features in an autoregressive manner to output a cross-modal joint feature;

[0166] S3. Construct a multi-task joint loss function to obtain the total loss;

[0167] Specifically, S31 uses the mean absolute error (L1 Loss) and generalized intersection-over-union loss (GIOU Loss) to perform box regression from the two dimensions of pixel-level offset and geometric shape similarity, respectively, to obtain the box regression loss L of the target box. box , used to accurately optimize the position and size of the target box:

[0168] L box =λ1L L1 (B pred ,B gt )+λ2L GIOU (B pred ,B gt )

[0169] Where, L box is the box regression loss function of the target box; L L1 is the mean absolute error; L GIOU is the generalized intersection-over-union loss; λ1 and λ2 are hyperparameters, B pred is the predicted box coordinate, B gt is the real frame coordinate;

[0170] S32, the classification task uses contrast loss to measure the difference between the target and the embedding. It consists of two independent task heads, visual cue embedding and text cue embedding, which calculate the similarity with the detection query respectively and optimize it with FocalLoss to obtain the total classification loss L cls (General classification, because the classification includes two categories: visual and text), among which:

[0171] 1) The classification head is predicted by calculating the dot product between each detection query feature and the visual cue embedding feature through the dot product calculation of the image feature, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the visual cue embedding feature

[0172]

[0173] Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query, V j Embedding for matching visual cues;

[0174] 2) At the same time, the dot product prediction classification head is calculated between each detection query and text embedding of the image feature by dot product calculation, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the text hint embedding feature

[0175]

[0176] Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query; T k Prompt embed for matching text;

[0177] 3) Classification loss based on detection query features and visual cue embedding features and the classification loss of the detection query features and the text hint embedding features Calculate and get the total classification loss L cls :

[0178]

[0179] Where, L cls is the total classification loss; They are the classification loss between the detection query features and the visual cue embedding features and the classification loss between the detection query features and the textual cue embedding features;

[0180] The detection query features are a set of learnable feature vectors in the decoder, each corresponding to a potential detection target, such as an insulator or tower. During training and inference, these detection query features interact with encoder features and visual cue embedding features through the Transformer decoder layers to gradually optimize the target bounding box and category prediction.

[0181] S33. Calculate the contrast loss between the visual cue embedding features and the text cue embedding features to obtain the cross-modal contrast loss L align ;

[0182] Specifically, given an input image and K visual cue embedding features V = (v1,...,v K ) and the text hint embedding features T = (t1,...,t K ), calculate the InfoNCE loss between the visual cue embedding feature and the text cue embedding feature to improve the visual cue embedding feature v i and its corresponding text hint embedding feature t i Similarity, reduction and other text embeddings t j The similarity of is used to achieve the fusion of visual prompt features and text prompt features:

[0183]

[0184] Where, L align is the contrast loss function, v i ·t i Embedding feature v for the i-th visual cue i and the corresponding text hint embedding feature t i The dot product of the two measures the similarity, t j are other textual cue embedding features, and K are visual-textual cue feature pairs.

[0185] S34, box regression loss L based on target box box , total classification loss L cls and cross-modal contrast loss L align Calculate and get the total loss L of the multi-task joint loss function loss :

[0186] L loss =L box +L cls +L align .

[0187] S4. Optimize and adjust the first target RT-DETR-V1 detection model based on the total loss to obtain the second target RT-DETR-V1 detection model with efficient visual-semantic joint reasoning;

[0188] Specifically, the visual cue embedding features and text cue embedding features are adjusted based on the total loss;

[0189] S5. Mount the second target RT-DETR-V1 detection model on the drone. The second target RT-DETR-V1 detection model is used to perform auxiliary correction on the drone photos in a cycle within a set time:

[0190] S51. During flight, the drone collects raw image data in real time through the onboard camera, analyzes ambient lighting conditions, and triggers the automatic dimming module to adjust exposure parameters to ensure that the quality of the input raw image data meets the detection requirements;

[0191] S52: The drone flies to the cross-arm side of the tower head and performs target detection and identification in a left-to-right and top-to-bottom order, including detecting the cross-arm side components connecting the tower body and insulators, the insulators, and the conductor side components connecting the insulators and conductors.

[0192] S53: Identify and judge the target detection, and perform auxiliary photo correction:

[0193] 1) If the target is identified, the drone system automatically performs the following steps:

[0194] ① Based on the maximum viewing angle of the camera and the ratio of the center deviation of the target frame to the screen, the deviation angle is calculated and the gimbal rotation is corrected;

[0195] ② Calculate the zoom ratio based on the current size and target size, and then adjust the focal length to assist the drone in taking photos;

[0196] 2) If the target is not recognized, wait until the grayscale value of the camera image stabilizes, then calculate the grayscale value of the current camera image, and then calculate the grayscale value of the camera image in a loop at equal intervals. Adjust the grayscale value of the target area to 125, set the EV value and adjust the exposure, and then recognize again;

[0197] 3) When the set drone adjustment time is reached, the loop calculation is exited and the drone-assisted photo correction is ended.

[0198] Note: When multiple targets are identified or misidentified during the target detection process, the drone's shooting target is determined through human intervention. The drone operation interface equipped with the second target RT-DETR-V1 detection model supports frame, point, and text input respectively. The coordinates of the pixel points can be directly obtained by clicking on the interface: ① Frame: Draw a frame on the drone control screen to obtain the target frame, and enter the point coordinates of the upper left and lower right corners of the target frame; ② Point: Automatically identify the target by clicking the target pixel on the flight control handle screen. The input is a random point coordinate within the detection target pixel; ③ Text: Enter the target category text in the terminal, such as "insulator", to accurately identify all insulator targets on the image.

[0199] The drone of the present invention, equipped with the second target RT-DETR-V1 detection model, has the following auxiliary functions when performing photo correction: the input of the model is an image, and the optional input is a frame, point or text. The target frame, label or text understanding is output, and real-time detection of all understandable targets in the drone-shot image, such as insulators and tower bodies, is supported; at the same time, according to the position of the target frame in the shooting picture, the drone system adjusts the drone flight direction in real time so that the target is in a centered field of view, and automatically flies near the target while maintaining a safe distance; according to the preset navigation logic, the model detects all targets and realizes automatic inspection and photography; when human intervention is required, you can choose to manually enter a descriptive text in the terminal. For example, if the model recognizes multiple towers, you can enter "middle tower", "leftmost tower" or "highest tower" through text to detect the desired target, and adjust the drone's flight parameters to place the target frame at the center of the target; or you can choose to manually enter the coordinates of a point or a frame. The model supports detection of all similar targets on the image, achieves target area recognition accuracy, and achieves the effect of centering the click target perspective in the same way.

[0200] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the principles and essence of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A UAV-assisted photography correction method based on an improved RT-DETR model, characterized in that: include: S1. Acquire target image data; S2. Build an RT-DETR-V1 detection model, input the target image data into the RT-DETR-V1 detection model, and obtain a first target RT-DETR-V1 detection model; wherein the RT-DETR-V1 detection model includes an image encoder, a visual cue encoder, a text cue encoder, and an output decoder; S3. Construct a multi-task joint loss function to obtain the total loss; S4. Optimize and adjust the first target RT-DETR-V1 detection model based on the total loss to obtain the second target RT-DETR-V1 detection model with efficient visual-semantic joint reasoning; S5. Mount the second target RT-DETR-V1 detection model on the drone. The second target RT-DETR-V1 detection model is used to assist the drone in taking photos and correcting the deviation. S51. During flight, the drone collects raw image data in real time through the onboard camera, analyzes ambient lighting conditions, and triggers the automatic dimming module to adjust exposure parameters to ensure that the quality of the input raw image data meets the detection requirements; S52: The drone flies to the cross-arm side of the tower head and performs target inspection in a left-to-right and top-to-bottom order, including inspecting the cross-arm side components connecting the tower body and insulators, the insulators, and the conductor side components connecting the insulators and conductors. S53: Identify and judge the target detection, and perform auxiliary photo correction: 1) If the target is identified, the drone system automatically performs the following steps: ① Based on the camera's maximum viewing angle and the ratio of the target frame's center deviation to the image, the deviation angle is calculated and the gimbal rotation is corrected. ② Calculate the zoom ratio based on the current size and target size, and then adjust the focal length to assist the drone in taking photos; 2) If the target is not recognized, wait until the grayscale value of the camera image stabilizes, then calculate the grayscale value of the current camera image, and then calculate the grayscale value of the camera image in a loop at equal intervals. Adjust the grayscale value of the target area to 125, set the EV value and adjust the exposure, and then recognize again; 3) When the set drone adjustment time is reached, the loop calculation is exited and the drone-assisted photo correction ends.

2. The method for correcting deviation of unmanned aerial vehicle (UAV)-assisted photography based on the improved RT-DETR model according to claim 1, characterized in that: Step S1 includes the following steps: S11, obtaining raw image data collected by the drone through an onboard camera inspection during flight; S12, performing denoising and distortion correction processing on the collected original image data in sequence to obtain first image data; S13. Use the Labelme image annotation tool to annotate the first image data with a rectangular frame, and add category text information to obtain target image data, where the target image data includes multimodal text, frame, and point data.

3. The method for correcting deviation of unmanned aerial vehicle (UAV)-assisted photography based on the improved RT-DETR model according to claim 1, characterized in that: Step S2 includes the following steps: S2. Build an RT-DETR-V1 detection model, input the target image data into the RT-DETR-V1 detection model for training, and obtain a first target RT-DETR-V1 detection model; wherein the RT-DETR-V1 detection model includes an image encoder, a visual cue encoder, a text cue encoder, and an output decoder; S21, based on the image encoder in the RT-DETR-V1 detection model, obtain a multi-scale feature map; S22, based on the visual cue encoder in the RT-DETR-V1 detection model, obtain the visual cue embedding feature; S23, based on the text prompt encoder in the RT-DETR-V1 detection model, obtain the text prompt embedding feature; S24. Based on the output decoder in the RT-DETR-V1 detection model, the predicted target box, predicted category label and text understanding output are obtained.

4. The method for correcting deviation of unmanned aerial vehicle-assisted photography based on the improved RT-DETR model according to claim 3, characterized in that: Step S21 includes the following steps: 1) Input the target image data into the image encoder in the RT-DETR-V1 detection model; 2) Based on the YOLOv11 feature extraction network in the image encoder, multi-scale feature extraction is performed on the target image data to obtain a multi-scale feature map; Wherein, the multi-scale feature map is recorded as: Where C, H, and W are the length, height, and width of the multi-scale feature map, respectively; L is the number of multi-scale feature layers; and i is the index of the feature layer number, which is used to distinguish feature maps of different scales.

5. The method for correcting deviation of unmanned aerial vehicle-assisted photography based on the improved RT-DETR model according to claim 4, characterized in that: Step S22 includes the following steps: 1) For the two visual cue types, box and point, two different coordinate encodings are established and the dimensions are unified: ① Given K user-specified 4D normalized box coordinates b on the target image data j =(x j ,y j ,w j ,h j ),j∈{1,2,...,K} and 2D normalized point coordinates p j =(x j ,y j ),j∈{1,2,...,K}; ② Establish differentiated encoding methods for the 4D normalized frame coordinates and the 2D normalized point coordinates to obtain two coordinate encodings; ③ Encode the two coordinates into position embeddings through a fixed sine-cosine embedding layer; ④Use two different linear layers to project the position embedding into a unified dimension B: B=Linear(PE(b1,…,b k );i B ):R K×4D →R K×D B=Linear(PE(p1,…,p k );i B ):R K×2D →R K×D Where Linear(.;θ) represents a linear projection operation with parameter θ; PE represents position embedding; K is the number of prompts; D is the embedding dimension; 2) Model the boxes and points as different types of visual cues, initialize a learnable content embedding, and expand it to match the number of cues K through a broadcast operation, denoted as C∈R K×D ; also , introduce the general category label C′∈R 1×D , used to aggregate features across multiple visual cues to accommodate scenarios where users provide multiple visual cues in a single image; 3) Concatenate the content embedding and the position embedding along the channel dimension to obtain a joint embedding; 4) Apply a linear projection layer to map the joint embedding to the query space, obtaining the input query embedding feature Q: Where, Represents a parameter Linear projection operation; CAT represents the merging of channel dimensions, B′ and P′ represent global position embeddings, which are derived from the global normalized coordinates [0.5, 0.5, 1, 1] and 0.5, 0.5; 5) Based on the multi-scale deformable cross attention layer, the visual cue features are extracted from the multi-scale feature map under the condition of visual cue: For the jth visual cue, the query feature Q is obtained after the multi-scale deformable cross attention layer calculation j ′: Where Q j ′ is the query feature of the j-th visual cue after being calculated by the multi-scale deformable cross attention layer, MSDeformAttn is the multi-scale deformable cross attention layer function, which is used to extract features conditioned on the visual cue from the multi-scale feature map; Q j is the jth original query feature; b j The 4D normalized box coordinates specified by the jth user, indicating the location and size information of the box; is a multi-scale feature map set composed of the j-th layer feature map, which is used to provide image features of different scales; p j The 2D normalized point coordinates specified by the jth user, indicating the location information of the point; 6) Based on the extracted visual cue features, a self-attention layer is used to model the dependencies between different queries, and a feed-forward layer FFN is used for projection to obtain the global query features, which are used as the visual cue embedding features V for fully representing the semantic and spatial information of the target area: V = FFN(SelfAttn(Q′))[-1] Where FFN is the feedforward neural network layer; SelfAttn is the self-attention layer; Q′ is the visual cue feature; [-1] is the last element of the feedforward layer output sequence as the final feature representation.

6. The method for correcting deviation of unmanned aerial vehicle-assisted photography based on the improved RT-DETR model according to claim 5, characterized in that: Step S23 includes the following steps: 1) Select a pre-trained CLIP text encoder; the CLIP text encoder is used to encode category names or descriptive phrases; 2) Use the hidden state corresponding to the [CLS] token to extract text prompt features from the multi-scale feature map to obtain global semantic features, which are used as text prompt features; 3) Normalize the global semantic features to obtain the text hint embedding feature T.

7. The method for correcting deviation of drone-assisted photography based on the improved RT-DETR model according to claim 6, characterized in that: Step S24 includes the following specific steps: 1) Predicting the target box: Each detection query is represented as a 4D anchor coordinate (x, y, w, h), and the position is fine-tuned in the decoder layer. The decoder layer is the decoder of the RT-DETR-V1 model, which is responsible for combining the features extracted by the encoder with the hint embedding provided by the user to generate the final detection result. ② Each anchor point in the decoder feature layer is initialized with a coordinate, and the similarity between the encoder feature and the visual cue embedding feature and the text cue embedding feature is calculated; ③Select the first 300 similarity indices corresponding to each hint embedding to generate the initial position embedding; ④ After multiple transformer decoder layers, the offset of the cumulative predicted anchor point (Δx, Δy, Δw, Δh) is summed with the initial anchor point, and the predicted target box is output: (Δx,Δy,Δw,Δh)=MLP(Q dec ) Box=(x+Δx,y+Δy,v+Δw,h+Δh) Where, MLP is a multi-layer perceptron; Q dec is the predicted query from the Box decoder; 2) Predicting category labels: Use visual cue embedding features or text cue embedding features as the weights of the classification layer. By updating the cue embedding library online, you can support the detection of new categories without retraining: Where C represents the total number of visual cue categories and N represents the number of detection queries; both visual cue and text object detection tasks share the same image encoder and box decoder for predicting target boxes. 3) Text understanding output: ① The RoIAlign algorithm is used to extract local features of the detected target from the feature map extracted by the YOLOv11 feature extraction network. RoIAlign extracts local features from the feature map based on the anchor box for target classification and regression; the visual cue embedding features and the text cue embedding features are combined to form object labels; ② Align the visual cue embedding features with the text cue embedding features through linear projection to ensure the effectiveness of cross-modal interaction; ③ The lightweight language decoder cross-modally fuses the visual cue embedding features and the text cue embedding features in an autoregressive manner and outputs cross-modal joint features.

8. The method for correcting deviation of unmanned aerial vehicle-assisted photography based on the improved RT-DETR model according to claim 1, characterized in that: Step S3 includes the following steps: S31, use the mean absolute error (L1 Loss) and generalized intersection-over-union loss (GIOU Loss) to perform box regression from the two dimensions of pixel-level offset and geometric shape similarity, and obtain the box regression loss L of the target box box , used to accurately optimize the position and size of the target box: L box =λ1L L1 (B pred ,B gt )+λ2L GIOU (B pred ,B gt ) Where, L box is the box regression loss function of the target box; L L1 is the mean absolute error; L GIOU is the generalized intersection-over-union loss; λ1 and λ2 are hyperparameters, B pred is the predicted box coordinate, B gt is the real frame coordinate; S32, the classification task uses contrast loss to measure the difference between the target and the embedding. It consists of two independent task heads, visual cue embedding and text cue embedding, which calculate the similarity with the detection query respectively and optimize it with FocalLoss to obtain the total classification loss L cls (General classification, because the classification includes two categories: visual and text), among which: 1) The classification head is predicted by calculating the dot product between each detection query feature and the visual cue embedding feature through the dot product calculation of the image feature, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the visual cue embedding feature Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query, V j Embedding for matching visual cues; 2) At the same time, the dot product prediction classification head is calculated between each detection query and text embedding of the image feature by dot product calculation, and the FocalLoss optimization is performed on each matching box encoder category to obtain the classification loss of the detection query feature and the text hint embedding feature Where N is the number of queries, FL(·) is the loss calculated for the predicted probability and the true label, σ(·) represents the Sigmoid function, which maps the dot product similarity to the probability value, and Q i is the feature vector of the i-th detection query; T k Prompt embed for matching text; 3) Classification loss based on detection query features and visual cue embedding features and the classification loss of the detection query features and the text hint embedding features Calculate and get the total classification loss L cls : Where, L cls is the total classification loss; They are the classification loss between the detection query features and the visual cue embedding features and the classification loss between the detection query features and the textual cue embedding features; S33. Calculate the contrast loss between the visual cue embedding features and the text cue embedding features to obtain the cross-modal contrast loss L align ; Specifically, given an input image and K visual cue embedding features V = (v1,...,v K ) and the text hint embedding features T = (t1,...,t K ), calculate the InfoNCE loss between the visual cue embedding feature and the text cue embedding feature to improve the visual cue embedding feature v i and its corresponding text hint embedding feature t i Similarity, reduction and other text embeddings t j The similarity of is used to achieve the fusion of visual prompt features and text prompt features: Where, L align is the contrast loss function, v i ·t i Embedding feature v for the i-th visual cue i and the corresponding text hint embedding feature t i The dot product of the two measures the similarity, t j are other textual cue embedding features, and K are visual-textual cue feature pairs. S34, box regression loss L based on target box box , total classification loss L cls and cross-modal contrast loss L align Calculate and get the total loss L of the multi-task joint loss function loss : L loss =L box +L cls +L align 。

Images