Unmanned aerial vehicle image distortion correction and coordinate positioning compensation method

Abstract

The invention relates to the technical field of digital image processing, in particular to an unmanned aerial vehicle image distortion correction and coordinate positioning compensation method, which comprises the following steps: collecting a large amount of unmanned aerial vehicle image data containing a damaged house, labeling the damaged house in the image, and generating a labeling file required by a YOLOv5 model; performing parameter calibration on the unmanned aerial vehicle lens to obtain a distortion coefficient of the lens, and constructing a nonlinear distortion correction model based on a lens parameter calibration result; establishing a dynamic affine transformation matrix in combination with the real-time flight attitude data of the unmanned aerial vehicle and the pixel coordinates of the detection frame; in a YOLOv5 detection stage, fusing feature maps with different resolutions, and fusing corrected coordinate data with a damaged house detection frame output by a YOLOv5 model; calculating coordinates of a center point of the damaged house according to the real-time latitude and longitude of the unmanned aerial vehicle; through dual optimization of image distortion correction and dynamic positioning compensation, the problem of coordinate misalignment caused by hardware limitation and unstable flight attitude in traditional unmanned aerial vehicle detection is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a method for correcting image distortion and compensating coordinate positioning of an unmanned aerial vehicle (UAV). Background Art

[0002] In existing technologies, house damage detection relies on manual on-site surveys or traditional image processing methods, which are inefficient and have poor real-time performance, making it difficult to cope with disaster emergency scenarios. When traditional models are deployed in the cloud, there are problems with high network latency and slow response speed, which cannot meet the needs of real-time monitoring and coordinate reporting. The positioning accuracy of damaged targets is insufficient, and there is a lack of automated center point coordinate calculation methods based on real-time drone images, resulting in inaccurate geographic location information feedback.

[0003] Application number CN202011441765.0 discloses a wildlife protection drone monitoring system based on edge computing. This technical solution deploys monitoring equipment in an ecological protection area to monitor the areas where wild animals appear and live. It then assigns a drone group to the monitoring equipment to collect data and process it on the edge computing equipment. Some valid data is saved and uploaded to the data center after the return trip. Based on the processed image information, an improved reference point-based fast non-dominated sorting method is used to dynamically plan the patrol path. After the patrol is completed, the return route is replanned based on the wildlife monitoring situation, and the power on and off of each monitoring device is determined, reducing unnecessary energy consumption. The status of damaged equipment is evaluated and reported to the maintenance process. The solution of this application significantly reduces the coordinate calculation error caused by hardware limitations and flight environment interference through the dual optimization of lens distortion correction and drone attitude compensation. In traditional methods, lens distortion will cause the center point of the detection frame to deviate from the actual damaged area, and the drone positioning deviation further amplifies the coordinate error, resulting in incorrect allocation of rescue resources. The present invention uses a dynamic correction algorithm to improve the geographic coordinate accuracy of the center point of the damaged house to sub-meter level, effectively avoiding the problem of misjudgment of rescue targets due to coordinate deviation.

[0004] Application number CN202410566129.2 discloses a drone edge computing processing method assisted by a 360-degree surround view camera. This technical solution uses the 360-degree surround view camera on the drone to capture images and video data of the surrounding environment and perform preprocessing, including conversion into grayscale images, cropping or scaling. The drone processes the data on the local edge computing device and applies an adaptive region extraction network for target monitoring. Based on the edge computing results, the drone's flight state or camera shooting angle is adjusted in real time in conjunction with an adaptive PID algorithm to achieve more precise drone control. The present invention significantly reduces the coordinate calculation error caused by hardware limitations and flight environment interference through the dual optimization of lens distortion correction and drone attitude compensation. In traditional methods, lens distortion will cause the center point of the detection frame to deviate from the actual damaged area, and the drone positioning deviation further causes coordinate error, resulting in errors or delays in the allocation of rescue resources. The present invention uses a dynamic correction algorithm to improve the geographic coordinate accuracy of the center point of the damaged house to sub-meter level, effectively avoiding the problem of misjudgment of rescue targets due to coordinate deviation.

[0005] To this end, the present invention provides a method for correcting UAV image distortion and compensating coordinate positioning. Summary of the Invention

[0006] The present invention aims to solve the technical problems existing in the prior art and provides a method for correcting the distortion of unmanned aerial vehicle images and compensating for coordinate positioning.

[0007] The present invention solves the above-mentioned technical problem with the following technical solution: A method for correcting image distortion and compensating coordinate positioning of a UAV, comprising the following steps:

[0008] S101. Collect a large amount of drone image data containing damaged houses, record the real-time flight attitude data of the drone and the corresponding geographic coordinate information, annotate the damaged houses in the image, generate the annotation files required by the YOLOv5 model, and train the model;

[0009] S102, calibrating the parameters of the drone lens to obtain the lens distortion coefficient, constructing a nonlinear distortion correction model based on the lens parameter calibration results, and correcting the original drone image pixels using polynomial fitting;

[0010] S103, combining the real-time flight attitude data of the UAV and the pixel coordinates of the detection frame, establishing a dynamic affine transformation matrix, and mapping the two-dimensional pixel position to a three-dimensional geographic coordinate system through the dynamic affine transformation matrix;

[0011] S104. In the YOLOv5 detection stage, feature maps of different resolutions are fused to enhance the detection capability of small-sized damaged houses. The corrected coordinate data is fused with the damaged house detection frame output by the YOLOv5 model to form a high-precision positioning result.

[0012] S105. Upload the trained YOLOv5 weight file to the Yunguan 2 platform to replace the original model. View the drone's camera image and the identified damaged house frame at the Yunguan push stream's rtsp address. Develop the identification code for the center point of the identification frame. Calculate the coordinates of the center point of the damaged house based on the drone's real-time latitude and longitude. Upload the obtained coordinates to the server in real time.

[0013] In a preferred embodiment, in said S101, the specific geographical area for data collection is determined based on the area where the house damage occurred and the historical house damage situation, and the flight plan of the drone is specified, including the flight route, altitude, speed and overlap rate parameters. The drone is controlled to collect images according to the flight plan, and the drone status and image collection quality are monitored in real time during the process. The real-time flight attitude data of the drone and the corresponding geographic coordinate information are recorded synchronously. After the image collection is completed, the acquired image data is preliminarily screened to eliminate blurred, repeated and abnormally exposed images. The screened image data is imported using a labeling tool, and the damaged houses in the image are labeled. The target of the annotation is assigned a category. After the annotation is completed, the annotation tool is used to generate the annotation file required by the YOLOv5 model. The collected and annotated image data is divided into training set, validation set and test set. Corresponding folders are created to store the image files and annotation files of the training set, validation set and test set respectively. The model is trained using the collected and annotated damaged house image data. The training script is run and the training parameters are set, including the learning rate, batch size and number of training rounds. The training parameters are continuously adjusted during the training process to improve the detection accuracy. The loss function value and the evaluation index of the validation set during the training process are monitored. The training effect of the model is judged based on the loss function value and the evaluation index of the validation set.

[0014] In a preferred embodiment, in S102, the calibration plate is placed on a stable plane, the drone is controlled to shoot the calibration plate from different angles, distances, and postures, a sufficient number of calibration images are collected, the drone flight posture data and geographic coordinate information corresponding to each image are recorded, the collected calibration images are read using an image processing library, the corner point detection algorithm is used to extract the coordinates of the corner points on the calibration plate, the extracted corner point coordinates are used to calibrate the lens parameters using a camera calibration algorithm, a nonlinear distortion correction model is constructed based on the lens parameters and distortion coefficients obtained by calibration, and for each pixel point, a polynomial fitting algorithm is used to calculate its corrected position. The specific calculation formula of the nonlinear distortion correction model is as follows:

[0015] x corrected =x(1+k1r 2 +k2r 4 +k3r 6 )

[0016] y corrected =y(1+k1r 2 +k2r 4 +k3r 6 )

[0017] Among them, (x, y) represents the original pixel coordinates, x corrected ,y corrected represents the pixel coordinates after correction, k1, k2, k3 represent the radial distortion coefficients, r 2 Represents the square of the distance from the original pixel (x, y) to the center of the image, r 4 Represents the square of the distance from the original pixel to the center of the image, r 6 Represents the cube of the square of the distance from the original pixel to the center of the image. As the r value changes, r 2 、r 4 、r 6 The value of will also change accordingly, thereby correcting the original pixel coordinates to varying degrees to compensate for the radial distortion of the lens. Each pixel point of the original image is traversed, and its corrected position is calculated according to the constructed nonlinear distortion correction model. The corrected pixel value is assigned to the pixel at the corresponding position in the new image to obtain the corrected image.

[0018] In a preferred embodiment, in S103, the step of establishing the dynamic affine transformation matrix includes the following steps:

[0019] S1. Obtain the drone's flight attitude data in real time, including altitude h, pitch angle θ, and yaw angle ψ. Obtain the pixel coordinates of the damaged building detection frame from the YOLOv5 model output, usually the pixel coordinates of the upper left and lower right corners of the detection frame (x1, y1) and (x2, y2), and then calculate the pixel coordinates of the detection frame center. Get the current geographic coordinates of the drone (X0, Y0, Z0), where Z0 = h;

[0020] S2. Define the rotation matrix R to describe the attitude change of the drone, considering the pitch angle and yaw angle respectively.

[0021]

[0022] Among them, R θ represents the pitch angle rotation matrix, R ψRepresents the yaw angle rotation matrix, R represents the rotation matrix, and combined with the height information and the rotation matrix, constructs the affine transformation matrix A from the two-dimensional pixel coordinate system to the three-dimensional geographic coordinate system. Let the intrinsic parameter matrix of the camera be K, which can be expressed as:

[0023]

[0024] Among them, f x and f y Indicates the focal length of the camera in the x and y directions, (c x ,c y ) is the principal point coordinate of the image, then the affine transformation matrix A can be expressed as:

[0025]

[0026] Let the two-dimensional pixel coordinates be P = [x p ,y p ,1] T , the three-dimensional geographic coordinates are P = [X, Y, Z] T , the mapping relationship is:

[0027]

[0028] S3, repeat steps S1 and S2 every 500 seconds, and update the affine transformation matrix A according to the latest flight attitude data of the UAV and the pixel coordinates of the detection frame;

[0029] After the affine transformation, outlier detection is added to automatically identify incorrect mapping points caused by target occlusion and texture loss, and abnormal coordinates are repaired through interpolation of adjacent frames. The specific steps include:

[0030] S1. Perform affine transformation on the pixel coordinates of the centers of all detection frames to obtain the corresponding three-dimensional geographic coordinates;

[0031] S2, using the RANSAC algorithm to detect outliers on the transformed three-dimensional geographic coordinates;

[0032] S3. Randomly select a part of the data points as the interior point set and calculate the model parameters of the interior point set. The model parameters are the plane equations.

[0033] S4. Calculate the distance between other data points and the model, and add data points whose distance is less than a certain threshold into the inliers.

[0034] S5, repeat steps S2-S4 multiple times, and select the model with the largest inlier set as the final model;

[0035] S6. Data points with distances greater than the threshold are determined to be outliers;

[0036] S7. For outliers, repair them by interpolating the adjacent frames, find the corresponding points of the outliers in the adjacent frames, perform linear interpolation based on the coordinate information of the adjacent frames, and obtain the repaired coordinates.

[0037] In a preferred embodiment, in S104, during the forward propagation process of the YOLOv5 model, feature maps of different levels are extracted in sequence according to the network structure, marked as F1, F2, and F3. For each feature map, its size and number of channels are recorded. For the top-down path, starting from the highest-level feature map F3, its resolution is increased to the same as the next-level feature map F2 through upsampling operation. The upsampled feature map is laterally connected to F2, and then feature fusion is performed through convolution operation to obtain the fused feature map F 2_fused Repeat the above steps to make F 2_fused After upsampling, it is horizontally connected and convolved with F1 to obtain the final fused feature map F 1_fused , by fusion, we get the feature map F with different resolutions and multi-scale information 1_fused 、F 2_fused , F3, input the fused feature maps into the subsequent detection heads of the YOLOv5 model. Each detection head is responsible for predicting the bounding boxes and category probability information of targets of different scales. The detection head generates prediction results through convolution and pooling operations based on the information on the feature maps;

[0038] Extract the detection box information of the damaged house from the output of the YOLOv5 model, including the coordinates of the upper left corner of the detection box (x 1_yolo ,y 1_yolo ), lower right corner (x 2_yolo ,y 2_yolo ) and the corresponding category probability and confidence score, after the previous lens distortion correction and coordinate positioning compensation, the corrected three-dimensional geographic coordinates (X, Y, Z) of the center point of each detection frame are obtained. When the corrected coordinates are three-dimensional geographic coordinates, according to the camera's projection model and related parameters, they are converted into two-dimensional coordinates (x corrected ,y corrected ), match the converted coordinates with the coordinates of the YOLOv5 detection frame to ensure that they correspond to the same damaged house target. For the matched detection frame and the corrected coordinates, integrate the corrected coordinate information into the detection frame, use the corrected coordinates as the center coordinates of the detection frame, recalculate the coordinates of the upper left and lower right corners of the detection frame, and obtain the fused detection frame (x 1_fused ,y 1_fused )、(x 2_fused ,y 2_fused), retain the original category probability and confidence score information of the YOLOv5 detection box, organize and output the fused detection box information and related category probabilities, confidence scores, etc. to form a high-precision positioning result.

[0039] In a preferred embodiment, in S105, a real-time connection with the drone is established, and the current real-time latitude and longitude information of the drone is obtained through the corresponding interface or device. The pixel coordinates calculated by the previously developed identification code of the center point of the identification frame are combined with the real-time latitude and longitude information of the drone. The pixel coordinates are converted into corresponding geographic coordinates using a geospatial conversion algorithm to obtain the actual geographic coordinates of the center point of the damaged house. The calculated coordinates of the center point of the damaged house are sorted and the data is formatted into a format that can be received by the server. According to the selected upload method and protocol, code is written to send the sorted coordinate data to the server in real time.

[0040] The beneficial effects of the present invention are: the present invention solves the coordinate inaccuracy problem caused by hardware limitations and unstable flight attitude in traditional UAV detection through dual optimization of image distortion correction and dynamic positioning compensation. Lens distortion correction reduces the deviation between the center point of the detection frame and the actual position of the damaged house to the sub-meter level, avoiding miscalibration caused by the distortion of the wide-angle lens edge. The dynamic positioning compensation algorithm significantly weakens the influence of UAV hovering jitter and tilted viewing angle on geographic coordinate mapping by fusing flight attitude parameters, ensuring that the latitude and longitude positioning error of the damaged house is controlled within an acceptable range for rescue operations. In practical applications, this technology can greatly reduce the problem of misjudgment of rescue resources due to coordinate deviation, improve the reliability of target positioning at the disaster site, gain a critical time window for emergency response, and provide high-precision spatial data support for subsequent disaster damage assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0043] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more, unless otherwise specifically specified.

[0044] In the description of this application, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art will recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0045] like Figure 1 This embodiment provides: a method for correcting drone image distortion and compensating coordinate positioning, which specifically includes the following steps:

[0046] S101. Collect a large amount of drone image data containing damaged houses, record the real-time flight attitude data of the drone and the corresponding geographic coordinate information, annotate the damaged houses in the image, generate the annotation files required by the YOLOv5 model, and train the model;

[0047] Furthermore, based on the area where the house damage occurred and the historical house damage situation, the specific geographical area for data collection was determined, and the flight plan of the drone was specified, including the flight route, altitude, speed and overlap rate parameters. The drone was controlled to collect images according to the flight plan, and the drone status and image collection quality were monitored in real time during the process. The real-time flight attitude data of the drone and the corresponding geographic coordinate information were recorded synchronously. After the image collection was completed, the acquired image data was preliminarily screened to eliminate blurred, repeated and abnormally exposed images. The filtered image data was imported using the annotation tool, and the damaged houses in the image were annotated, and a category was assigned to each annotated target. After the annotation is completed, use the annotation tool to generate the annotation files required by the YOLOv5 model, divide the collected and annotated image data into training set, validation set and test set, create corresponding folders to store the image files and annotation files of the training set, validation set and test set respectively, use the collected and annotated damaged house image data to train the model, run the training script, set the training parameters, including learning rate, batch size and number of training rounds, continuously adjust the training parameters during the training process to improve the detection accuracy, monitor the loss function value and the evaluation index of the validation set during the training process, and judge the training effect of the model based on the loss function value and the evaluation index of the validation set.

[0048] S102, calibrating the parameters of the drone lens to obtain the lens distortion coefficient, constructing a nonlinear distortion correction model based on the lens parameter calibration results, and correcting the original drone image pixels using polynomial fitting;

[0049] Furthermore, the calibration plate is placed on a stable plane, and the drone is controlled to shoot the calibration plate from different angles, distances, and postures. A sufficient number of calibration images are collected, and the drone flight attitude data and geographic coordinate information corresponding to each image are recorded. The collected calibration images are read using the image processing library, and the corner point detection algorithm is used to extract the coordinates of the corner points on the calibration plate. The extracted corner point coordinates are used to calibrate the lens parameters using the camera calibration algorithm. Based on the lens parameters and distortion coefficients obtained by calibration, a nonlinear distortion correction model is constructed. For each pixel point, a polynomial fitting algorithm is used to calculate its corrected position. The specific calculation formula of the nonlinear distortion correction model is as follows:

[0050] x corrected =x(1+k1r 2 +k2r 4 +k3r 6 )

[0051] y corrected =y(1+k1r 2 +k2r 4 +k3r 6 )

[0052] Among them, (x, y) represents the original pixel coordinates, x corrected ,y corrected represents the pixel coordinates after correction, k1, k2, k3 represent the radial distortion coefficients, r 2 Represents the square of the distance from the original pixel (x, y) to the center of the image, r 4 Represents the square of the distance from the original pixel to the center of the image, r 6 Represents the cube of the square of the distance from the original pixel to the center of the image. As the r value changes, r 2 、r 4 、r 6 The value of will also change accordingly, thereby correcting the original pixel coordinates to varying degrees to compensate for the radial distortion of the lens. Each pixel point of the original image is traversed, and its corrected position is calculated according to the constructed nonlinear distortion correction model. The corrected pixel value is assigned to the pixel at the corresponding position in the new image to obtain the corrected image.

[0053] S103, combining the real-time flight attitude data of the UAV and the pixel coordinates of the detection frame, establishing a dynamic affine transformation matrix, and mapping the two-dimensional pixel position to a three-dimensional geographic coordinate system through the dynamic affine transformation matrix;

[0054] Furthermore, the step of establishing the dynamic affine transformation matrix includes the following steps:

[0055] S1. Obtain the drone's flight attitude data in real time, including altitude h, pitch angle θ, and yaw angle ψ. Obtain the pixel coordinates of the damaged building detection frame from the YOLOv5 model output, usually the pixel coordinates of the upper left and lower right corners of the detection frame (x1, y1) and (x2, y2), and then calculate the pixel coordinates of the detection frame center. Get the current geographic coordinates of the drone (X0, Y0, Z0), where Z0 = h;

[0056] S2. Define the rotation matrix R to describe the attitude change of the drone, considering the pitch angle and yaw angle respectively. The specific calculation formula of the rotation matrix is as follows:

[0057]

[0058]

[0059] Among them, R θ Represents the pitch angle rotation matrix, R ψ Represents the yaw angle rotation matrix, R represents the rotation matrix, and combined with the height information and the rotation matrix, constructs the affine transformation matrix A from the two-dimensional pixel coordinate system to the three-dimensional geographic coordinate system. Let the intrinsic parameter matrix of the camera be K, which can be expressed as:

[0060]

[0061] Among them, f x and f y Indicates the focal length of the camera in the x and y directions, (c x ,c y ) is the principal point coordinate of the image, then the affine transformation matrix A can be expressed as:

[0062]

[0063] Let the two-dimensional pixel coordinates be P = [x p ,y p ,1] T , the three-dimensional geographic coordinates are P = [X, Y, Z] T , the mapping relationship is:

[0064]

[0065] S3, repeat steps S1 and S2 every 500 seconds, and update the affine transformation matrix A according to the latest flight attitude data of the UAV and the pixel coordinates of the detection frame;

[0066] After the affine transformation, outlier detection is added to automatically identify incorrect mapping points caused by target occlusion and texture loss, and abnormal coordinates are repaired through interpolation of adjacent frames. The specific steps include:

[0067] S1. Perform affine transformation on the pixel coordinates of the centers of all detection frames to obtain the corresponding three-dimensional geographic coordinates;

[0068] S2, using the RANSAC algorithm to detect outliers on the transformed three-dimensional geographic coordinates;

[0069] S3. Randomly select a part of the data points as the interior point set and calculate the model parameters of the interior point set. The model parameters are the plane equations.

[0070] S4. Calculate the distance between other data points and the model, and add data points whose distance is less than a certain threshold into the inliers.

[0071] S5, repeat steps S2-S4 multiple times, and select the model with the largest inlier set as the final model;

[0072] S6. Data points with distances greater than the threshold are determined to be outliers;

[0073] S7. For outliers, repair them by interpolating the adjacent frames, find the corresponding points of the outliers in the adjacent frames, perform linear interpolation based on the coordinate information of the adjacent frames, and obtain the repaired coordinates.

[0074] S104. In the YOLOv5 detection stage, feature maps of different resolutions are fused to enhance the detection capability of small-sized damaged houses. The corrected coordinate data is fused with the damaged house detection frame output by the YOLOv5 model to form a high-precision positioning result.

[0075] Furthermore, in the forward propagation process of the YOLOv5 model, feature maps of different levels are extracted in sequence according to the network structure, marked as F1, F2, and F3. For each feature map, its size and number of channels are recorded. For the top-down path, starting from the highest-level feature map F3, its resolution is increased to the same as the next-level feature map F2 through upsampling operation. The upsampled feature map is horizontally connected with F2, and then the feature fusion is performed through convolution operation to obtain the fused feature map F 2_fused Repeat the above steps to make F 2_fused After upsampling, it is horizontally connected and convolved with F1 to obtain the final fused feature map F 1_fused , by fusion, we get the feature map F with different resolutions and multi-scale information 1_fused 、F 2_fused , F3, input the fused feature maps into the subsequent detection heads of the YOLOv5 model. Each detection head is responsible for predicting the bounding boxes and category probability information of targets of different scales. The detection head generates prediction results through convolution and pooling operations based on the information on the feature maps;

[0076] Extract the detection box information of the damaged house from the output of the YOLOv5 model, including the coordinates of the upper left corner of the detection box (x 1_yolo ,y 1_yolo ), lower right corner (x 2_yolo ,y 2_yolo ) and the corresponding category probability and confidence score, after the previous lens distortion correction and coordinate positioning compensation, the corrected three-dimensional geographic coordinates (X, Y, Z) of the center point of each detection frame are obtained. When the corrected coordinates are three-dimensional geographic coordinates, according to the camera's projection model and related parameters, they are converted into two-dimensional coordinates (x corrected ,y corrected ), match the converted coordinates with the coordinates of the YOLOv5 detection frame to ensure that they correspond to the same damaged house target. For the matched detection frame and the corrected coordinates, integrate the corrected coordinate information into the detection frame, use the corrected coordinates as the center coordinates of the detection frame, recalculate the coordinates of the upper left and lower right corners of the detection frame, and obtain the fused detection frame (x 1_fused ,y 1_fused )、(x 2_fused ,y 2_fused), retain the original category probability and confidence score information of the YOLOv5 detection box, organize and output the fused detection box information and related category probabilities, confidence scores, etc. to form a high-precision positioning result.

[0077] S105. Upload the trained YOLOv5 weight file to the Yunguan 2 platform to replace the original model. View the drone's camera image and the identified damaged house frame at the Yunguan push stream's rtsp address. Develop the identification code for the center point of the identification frame. Calculate the coordinates of the center point of the damaged house based on the drone's real-time latitude and longitude. Upload the obtained coordinates to the server in real time.

[0078] Furthermore, a real-time connection with the drone is established, and the current real-time latitude and longitude information of the drone is obtained through the corresponding interface or device. The pixel coordinates calculated by the previously developed identification code for the center point of the identification frame are combined with the real-time latitude and longitude information of the drone. The pixel coordinates are converted into corresponding geographic coordinates using the geospatial conversion algorithm to obtain the actual geographic coordinates of the center point of the damaged house. The calculated coordinates of the center point of the damaged house are sorted and formatted into a format that the server can receive. According to the selected upload method and protocol, code is written to send the sorted coordinate data to the server in real time.

[0079] It should be noted that the size of the calibration plate and the size of the grid should be determined according to the actual situation to ensure that it can be clearly identified in the drone image. During the shooting process, the calibration plate must be completely visible in the image and have different rotation and tilt angles. The camera calibration algorithm used in this invention is the Zhang Zhengyou calibration method. During the calibration process, the intrinsic parameters and distortion coefficients of the lens will be calculated.

[0080] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0081] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0086] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for correcting image distortion and compensating coordinate positioning of a drone, characterized in that: The following steps are involved: S101. Collect a large amount of drone image data containing damaged houses, record the real-time flight attitude data of the drone and the corresponding geographic coordinate information, annotate the damaged houses in the image, generate the annotation files required by the YOLOv5 model, and train the model; S102, calibrating the parameters of the drone lens to obtain the lens distortion coefficient, constructing a nonlinear distortion correction model based on the lens parameter calibration results, and correcting the original drone image pixels using polynomial fitting; S103, combining the real-time flight attitude data of the UAV and the pixel coordinates of the detection frame, establishing a dynamic affine transformation matrix, and mapping the two-dimensional pixel position to a three-dimensional geographic coordinate system through the dynamic affine transformation matrix; S104. In the YOLOv5 detection stage, feature maps of different resolutions are fused to enhance the detection capability of small-sized damaged houses. The corrected coordinate data is fused with the damaged house detection frame output by the YOLOv5 model to form a high-precision positioning result. S105. Upload the trained YOLOv5 weight file to the Yunguan 2 platform to replace the original model. View the drone's camera image and the identified damaged house frame at the Yunguan push stream rtsp address to develop the identification code for the center point of the identification frame. Calculate the coordinates of the center point of the damaged house based on the real-time latitude and longitude of the drone, and upload the obtained coordinates to the server in real time.

2. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 1, wherein: In S101, the specific geographical area for data collection is determined based on the area where the house damage occurred and the historical house damage situation, and the flight plan of the drone is specified, including the flight route, altitude, speed and overlap rate parameters. The drone is controlled to collect images according to the flight plan. During the process, the drone status and image collection quality are monitored in real time, and the real-time flight attitude data of the drone and the corresponding geographic coordinate information are recorded synchronously. After the image collection is completed, the acquired image data is preliminarily screened to eliminate blurred, repeated and abnormally exposed images. The screened image data is imported using the annotation tool, and the damaged houses in the image are annotated. A specific target is assigned to each annotated target. After the labeling is completed, the labeling tool is used to generate the labeling files required by the YOLOv5 model. The collected and labeled image data are divided into training set, validation set and test set. Corresponding folders are created to store the image files and labeling files of the training set, validation set and test set respectively. The model is trained using the collected and labeled damaged house image data. The training script is run and the training parameters are set, including the learning rate, batch size and number of training rounds. The training parameters are continuously adjusted during the training process to improve the detection accuracy. The loss function value and the evaluation index of the validation set are monitored during the training process. The training effect of the model is judged based on the loss function value and the evaluation index of the validation set.

3. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 1, wherein: In S102, the calibration plate is placed on a stable plane, and the drone is controlled to shoot the calibration plate from different angles, distances, and postures to collect a sufficient number of calibration images. The drone flight posture data and geographic coordinate information corresponding to each image are recorded. The collected calibration images are read using an image processing library, and the corner point coordinates on the calibration plate are extracted using a corner point detection algorithm. The extracted corner point coordinates are used to calibrate the lens parameters using a camera calibration algorithm. Based on the lens parameters and distortion coefficients obtained by calibration, a nonlinear distortion correction model is constructed. For each pixel point, the corrected position is calculated using a polynomial fitting algorithm. The specific calculation formula of the nonlinear distortion correction model is as follows: x corrected =x(1+k1r 2 +k2r 4 +k3r 6 ) and corrected =y(1+k1r 2 +k2r 4 +k3r 6 ) Among them, (x, y) represents the original pixel coordinates, x corrected ,y corrected represents the pixel coordinates after correction, k1, k2, k3 represent the radial distortion coefficients, r 2 Represents the square of the distance from the original pixel (x, y) to the center of the image, r 4 Represents the square of the distance from the original pixel to the center of the image, r 6 Represents the cube of the square of the distance from the original pixel to the center of the image. As the r value changes, r 2 、r 4 、r 6 The value of will also change accordingly, thereby correcting the original pixel coordinates to varying degrees to compensate for the radial distortion of the lens. Each pixel point of the original image is traversed, and its corrected position is calculated according to the constructed nonlinear distortion correction model. The corrected pixel value is assigned to the pixel at the corresponding position in the new image to obtain the corrected image.

4. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 1, wherein: In S103, the step of establishing the dynamic affine transformation matrix includes the following steps: S1. Obtain the drone's flight attitude data in real time, including altitude h, pitch angle θ, and yaw angle ψ. Obtain the pixel coordinates of the damaged building detection frame from the YOLOv5 model output, usually the pixel coordinates of the upper left and lower right corners of the detection frame (x1, y1) and (x2, y2), and then calculate the pixel coordinates of the detection frame center. Get the current geographic coordinates of the drone (X0, Y0, Z0), where Z0 = h; S2. Define the rotation matrix R to describe the attitude change of the drone, considering the pitch angle and yaw angle respectively. Among them, R θ Represents the pitch angle rotation matrix, R ψ Represents the yaw angle rotation matrix, R represents the rotation matrix, and combined with the height information and the rotation matrix, constructs the affine transformation matrix A from the two-dimensional pixel coordinate system to the three-dimensional geographic coordinate system. Let the intrinsic parameter matrix of the camera be K, which can be expressed as: Among them, f x and f y Indicates the focal length of the camera in the x and y directions, (c x ,c y ) is the principal point coordinate of the image, then the affine transformation matrix A can be expressed as: Let the two-dimensional pixel coordinates be P = [x p ,y p ,1] T , the three-dimensional geographic coordinates are P = [X, Y, Z] T , the mapping relationship is: S3. Repeat steps S1 and S2 every 500 seconds to update the affine transformation matrix A based on the latest flight attitude data of the drone and the pixel coordinates of the detection frame.

5. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 4, wherein: After the affine transformation, outlier detection is added to automatically identify incorrect mapping points caused by target occlusion and texture loss, and abnormal coordinates are repaired through interpolation of adjacent frames. The specific steps include: S1. Perform affine transformation on the pixel coordinates of the centers of all detection frames to obtain the corresponding three-dimensional geographic coordinates; S2, using the RANSAC algorithm to detect outliers on the transformed three-dimensional geographic coordinates; S3. Randomly select a part of the data points as the interior point set and calculate the model parameters of the interior point set. The model parameters are the plane equations. S4. Calculate the distance between other data points and the model, and add data points whose distance is less than a certain threshold into the inliers. S5, repeat steps S2-S4 multiple times, and select the model with the largest inlier set as the final model; S6. Data points with distances greater than the threshold are determined to be outliers; S7. For outliers, repair them by interpolating the adjacent frames, find the corresponding points of the outliers in the adjacent frames, perform linear interpolation based on the coordinate information of the adjacent frames, and obtain the repaired coordinates.

6. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 1, wherein: In the S104, during the forward propagation process of the YOLOv5 model, feature maps of different levels are extracted in sequence according to the network structure, marked as F1, F2, and F3. For each feature map, its size and number of channels are recorded. For the top-down path, starting from the highest-level feature map F3, its resolution is increased to the same as the next-level feature map F2 through upsampling operation. The upsampled feature map is horizontally connected with F2, and then the feature fusion is performed through convolution operation to obtain the fused feature map F 2_fused Repeat the above steps to make F 2_fused After upsampling, it is horizontally connected and convolved with F1 to obtain the final fused feature map F 1_fused , by fusion, we get the feature map F with different resolutions and multi-scale information 1_fused 、F 2_fused , F3, and input the fused feature maps into the subsequent detection heads of the YOLOv5 model. Each detection head is responsible for predicting the bounding boxes and category probability information of targets of different scales. The detection head generates prediction results through convolution and pooling operations based on the information on the feature maps.

7. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 6, wherein: Extract the detection box information of the damaged house from the output of the YOLOv5 model, including the coordinates of the upper left corner of the detection box (x 1_yolo ,y 1_yolo ), lower right corner (x 2_yolo ,y 2_yolo ) and the corresponding category probability and confidence score, after the previous lens distortion correction and coordinate positioning compensation, the corrected three-dimensional geographic coordinates (X, Y, Z) of the center point of each detection frame are obtained. When the corrected coordinates are three-dimensional geographic coordinates, according to the camera's projection model and related parameters, they are converted into two-dimensional coordinates (x corrected ,y corrected ), match the converted coordinates with the coordinates of the YOLOv5 detection frame to ensure that they correspond to the same damaged house target. For the matched detection frame and the corrected coordinates, integrate the corrected coordinate information into the detection frame, use the corrected coordinates as the center coordinates of the detection frame, recalculate the coordinates of the upper left and lower right corners of the detection frame, and obtain the fused detection frame (x 1_fused ,y 1_fused )、(x 2_fused ,y 2_fused ), retain the original category probability and confidence score information of the YOLOv5 detection box, organize and output the fused detection box information and related category probabilities, confidence scores, etc. to form a high-precision positioning result.

8. The method for correcting image distortion and compensating coordinate positioning of a drone according to claim 1, wherein: In the above S105, a real-time connection is established with the UAV, and the current real-time latitude and longitude information of the UAV is obtained through the corresponding interface or device. The pixel coordinates calculated by the previously developed identification code for the center point of the identification frame are combined with the real-time latitude and longitude information of the UAV. The pixel coordinates are converted into corresponding geographic coordinates using a geospatial conversion algorithm, thereby obtaining the actual geographic coordinates of the center point of the damaged house. The calculated coordinates of the center point of the damaged house are sorted and formatted into a format that can be received by the server. According to the selected upload method and protocol, code is written to send the sorted coordinate data to the server in real time.

Images