Method for determining position of target object in unmanned aerial vehicle image

Abstract

The invention discloses a method for determining the position of a target object in an unmanned aerial vehicle image. The method comprises the following steps: receiving an initial data set acquired by an unmanned aerial vehicle, and obtaining an original target image and platform auxiliary information based on the initial data set; determining a geographic area covered by the original target image based on the platform auxiliary information; based on the geographic area, searching a satellite image tile corresponding to the geographic area from a satellite tile database; respectively extracting feature points of the original target image and the satellite image tile by adopting a target feature extraction algorithm to obtain a first group of feature points corresponding to the original target image and a second group of feature points corresponding to the satellite image tile; the technical problem that in the prior art, in the analysis process of the image shot by the unmanned aerial vehicle, a closed loop cannot be rapidly closed, and the information of the target object in the image shot by the unmanned aerial vehicle cannot be rapidly provided is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle image processing, and in particular to a method for determining the position of a target object in an unmanned aerial vehicle image. Background Art

[0002] The target analysis of the original images taken by drones includes processing steps such as geographic correction, target picking, target positioning, and target reporting. In the target environment, the geographic location information of the target object in the original image is highly timely, and the fast closed-loop target analysis process of drone images can provide important support for attacking the target object. At present, in the above closed-loop process, there are the following problems: drone images are collected quickly, with a wide range and high information density, and manual analysis cannot close the above process in time. The existing various automation methods in the above process lack systematic integration, resulting in delays in data processing and reporting of the geographic location information of the target object, making it difficult to form an efficient closed-loop of the geographic location of the target object, and unable to fully meet the needs of real-time situation awareness in target operation tasks. Due to factors such as resolution loss, distortion and weakening of target features, the corrected orthophoto is not suitable for direct target analysis by personnel or artificial intelligence, and the original image usually has poor results. Summary of the invention

[0003] An embodiment of the present invention provides a method for determining the position of a target object in a drone image, so as to at least solve the technical problem in the prior art that during the analysis process of images taken by drones, the loop cannot be quickly closed and the information of the target object in the images taken by drones cannot be quickly provided.

[0004] According to one aspect of an embodiment of the present invention, a method for determining the position of a target object in an unmanned aerial vehicle image is provided. The method may include: receiving an initial data set collected by an unmanned aerial vehicle, obtaining an original target image and platform auxiliary information based on the initial data set; determining the geographical area covered by the original target image based on the platform auxiliary information; retrieving the satellite image tile corresponding to the geographical area from a satellite tile database based on the geographical area; using a target feature extraction algorithm to extract feature points of the original target image and the satellite image tile respectively, obtaining a first group of feature points corresponding to the original target image and a second group of feature points corresponding to the satellite image tile; obtaining a homography matrix based on the first group of feature points and the second group of feature points; using a target detection algorithm to perform target detection on the original target image, obtaining the annotation information and target list of the target object in the original target image, wherein the target list includes the target name and target pixel coordinates of the target object; determining the pixel coordinates of the target object in the satellite image tile based on the target pixel coordinates and the homography matrix; determining the geographical location of the target object based on the pixel coordinates of the target object in the satellite image tile.

[0005] Optionally, obtaining the original target image and the platform auxiliary information based on the initial data set includes: decoding the initial data set to obtain the original target image and the platform auxiliary information.

[0006] Optionally, the platform auxiliary information includes: the longitude of the drone position, the latitude of the drone position, the roll angle of the drone, the heading angle of the drone, the azimuth angle of the optoelectronic platform, the pitch angle of the optoelectronic platform and the focal length of the optoelectronic platform.

[0007] Optionally, determining the geographical area covered by the original target image based on the platform auxiliary information includes: obtaining four-dimensional geographical coordinates of the original target image based on the platform auxiliary information; and determining the geographical area covered by the original target image based on the four-dimensional geographical coordinates.

[0008] Optionally, obtaining the homography matrix based on the first group of feature points and the second group of feature points includes: obtaining target feature point pairs based on the first group of feature points and the second group of feature points; and obtaining the homography matrix based on the target feature point pairs.

[0009] Optionally, obtaining the target feature point pair based on the first group of feature points and the second group of feature points includes: using a target feature matching algorithm to match the first group of feature points and the second group of feature points to obtain an initial feature point pair, wherein the initial feature point pair includes a target feature point pair and a noise point pair; and using a random sampling consensus algorithm to eliminate the noise point pairs in the initial feature points to obtain the target feature point pair.

[0010] Optionally, after determining the geographic location of the target object based on the pixel coordinates of the target object in the satellite image tile, the method further includes: updating the target list based on the geographic location of the target object to obtain an updated target list; saving the labeling information of the target object and the updated target list, and sending the labeling information of the target object and the updated target list to the target user.

[0011] Beneficial effects of the present invention: The present invention proposes a method for determining the position of a target object in an unmanned aerial vehicle image. The present invention effectively introduces satellite tile data into the geographic information processing of unmanned aerial vehicle images, accurately calculates the geographic boundaries of the image through platform auxiliary information, and quickly retrieves the satellite tile data matching the range, which greatly improves the efficiency and accuracy of geographic information association. In this process, advanced image feature extraction and feature matching technology are used to calculate the homography matrix of the unmanned aerial vehicle image mapped to the satellite tile data, thereby indirectly establishing a mapping relationship between the unmanned aerial vehicle image and the geographic coordinate system. This mapping ensures the geographic consistency of the image data, making subsequent geographic information applications more accurate and efficient. Through this innovative mapping method, unmanned aerial vehicles Each pixel point of the image can be quickly located to the corresponding geographic coordinates, providing a solid data foundation for further analysis of the targets in the image. In addition, by rationally utilizing satellite tile data, the cumbersome geographic data matching process in traditional methods is avoided, significantly improving the overall computing efficiency. In terms of target detection, the YOLOv5 target detection model trained with a proprietary data set is used to ensure high detection accuracy in various complex environments and different terrain conditions. In order to speed up the target detection process, the system uses GPU to accelerate the inference of the model, and quickly picks up all targets of interest in a single image within 30 milliseconds. This efficient detection capability greatly improves the real-time response capability of the drone to handle target tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 The present invention is a flowchart of a method for determining the position of a target object in a drone image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0015] Example 1 According to an embodiment of the present invention, a method for determining the position of a target object in a drone image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0016] Figure 1 is a flow chart of a method for determining the position of a target object in a drone image according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, receiving an initial data set collected by a drone, and obtaining an original target image and platform auxiliary information based on the initial data set.

[0017] In the technical solution provided in the above step S101 of the present invention, the UAV ground station system receives the initial data set collected by the UAV payload in real time, wherein the initial data set includes the original target image and platform auxiliary information, and processes the initial data set to obtain the original target image and platform auxiliary information.

[0018] Step S102: determining the geographical area covered by the original target image based on the platform auxiliary information.

[0019] In the technical solution provided in the above step S102 of the present invention, the platform auxiliary information is processed by a conventional algorithm to obtain the geographical area covered by the original target image.

[0020] Step S103 , based on the geographical area, searching the satellite image tiles corresponding to the geographical area from the satellite tile database.

[0021] In the technical solution provided in the above step S103 of the present invention, according to the geographical area, the satellite image tiles corresponding to the geographical area are retrieved from the satellite tile database, wherein the geographical area is the approximate geographical range covered by the original target image.

[0022] Step S104, using a target feature extraction algorithm to extract feature points of the original target image and the satellite image tile respectively, to obtain a first group of feature points corresponding to the original target image and a second group of feature points corresponding to the satellite image tile.

[0023] In the technical solution provided in the above step S104 of the present invention, the SuperPoint feature extraction algorithm is used to extract feature points of the original target image and the satellite image tile respectively, to obtain a first group of feature points corresponding to the original target image and a second group of feature points corresponding to the satellite image tile.

[0024] Step S105: obtaining a homography matrix based on the first set of feature points and the second set of feature points.

[0025] In the technical solution provided in step S105 of the present invention, the first set of feature points and the second set of feature points are used to determine a homography matrix, wherein the expression of the homography matrix is:

[0026] in, is the homography matrix, and To control the original target image X stretching or rotation in the direction of an axis, and To control the stretching or rotation of the original target image in the Y-axis direction, and To control the translation of the original target image in the X-axis and Y-axis directions, and To introduce perspective transformation, is 1.

[0027] Step S106 , performing target detection on the original target image using a target detection algorithm, and obtaining labeling information and a target list of the target object in the original target image, wherein the target list includes the target name and target pixel coordinates of the target object.

[0028] In the technical solution provided in the above step S106 of the present invention, target detection is performed on the original target image using a target detection algorithm (YOLOV5), which includes the following steps: By using drones to capture images under different conditions (such as various lighting environments, weather changes, different flight altitudes and angles), we collect a diverse data set. To ensure that the model has strong generalization capabilities, the data set covers different postures of the target objects and background complexity.

[0029] Use annotation tools to annotate images, accurately mark the categories and bounding box coordinates of the targets of interest, and convert the annotation files into YOLO format. This process generates a large amount of annotation data, which is divided into training set, validation set, and test set in a ratio of 7:2:1.

[0030] The training set is preprocessed and enhanced, and normalized to ensure the consistency of the input data. Random flipping, rotation, scaling, color jittering, and noise addition are used. This preprocessing can alleviate the overfitting problem.

[0031] Perform model configuration and training. Configure the data.yaml file of YOLOv5, specify the number of categories and dataset path, run the training script, set the image size to 960×960, the number of training rounds to 300, and the learning rate to 0.0001. During the training process, use GPU to accelerate reasoning, improve computing efficiency and shorten training time. The model will dynamically monitor the downward trend of the loss function during training, and evaluate the accuracy (such as mAP, precision and recall) through the validation set. If necessary, adjust the hyperparameters or perform early stopping to avoid overfitting or underfitting and ensure robust convergence of the model.

[0032] After the training is completed, the trained model is used to perform target detection on the original target image. The size of the original target image is adjusted to 960×960 to ensure input consistency. After the target object is detected, the non-maximum suppression (NMS) algorithm is applied to remove redundant bounding boxes and only the detection results with the highest confidence are retained to ensure the accuracy of the output results.

[0033] The detected target object is marked in the original target image, and a target name and target pixel coordinates of the target object are generated, wherein the target pixel coordinates are the pixel coordinates at the midpoint of the target object.

[0034] Step S107, determining the pixel coordinates of the target object in the satellite image tile based on the target pixel coordinates and the homography matrix.

[0035] In the technical solution provided in step S107 of the present invention, the target pixel coordinates Multiply it with the homography matrix to get the pixel coordinates of the target object in the satellite image tile The expression is:

[0036] in, is the target pixel coordinate, is the target pixel horizontal coordinate, is the target pixel ordinate, is the pixel coordinate of the satellite image tile, is the pixel horizontal coordinate of the satellite image tile, It is the pixel vertical coordinate of the satellite image tile.

[0037] Step S108, determining the geographic location of the target object based on the pixel coordinates of the target object in the satellite image tile.

[0038] In the technical solution provided in the above step S108 of the present invention, the expression for determining the geographical location of the target object according to the pixel coordinates of the target object in the satellite image tile is:

[0039]

[0040] in, X is the longitude of the target object's geographic location, Y is the latitude of the target object’s geographic location, ， ， ， ， ， are the six coefficients of the affine transformation.

[0041] The above method of this embodiment is further introduced below.

[0042] As an optional implementation manner, step S101, obtaining the original target image and platform auxiliary information based on the initial data set, includes: decoding the initial data set to obtain the original target image and platform auxiliary information.

[0043] In this embodiment, the initial data set is decoded to obtain the original target image and platform auxiliary information.

[0044] As an optional implementation method, the platform auxiliary information includes: the longitude of the drone position, the latitude of the drone position, the roll angle of the drone, the heading angle of the drone, the azimuth angle of the optoelectronic platform, the pitch angle of the optoelectronic platform and the focal length of the optoelectronic platform.

[0045] As an optional implementation method, step S101, determining the geographical area covered by the original target image based on the platform auxiliary information, includes: obtaining the four-dimensional geographical coordinates of the original target image based on the platform auxiliary information; and determining the geographical area covered by the original target image based on the four-dimensional geographical coordinates.

[0046] In this embodiment, a coordinate system conversion algorithm is used according to platform auxiliary information to obtain the four-dimensional geographic coordinates of the original target image. The four-dimensional geographic coordinates are the four corners around the original target image. Based on the four-dimensional geographic coordinates, the approximate geographic range covered by the original target image is determined.

[0047] As an optional implementation manner, step S105, obtaining a homography matrix based on the first group of feature points and the second group of feature points, includes: obtaining target feature point pairs based on the first group of feature points and the second group of feature points; obtaining a homography matrix based on the target feature point pairs.

[0048] In this embodiment, the SuperGlue feature matching algorithm and the random sampling consensus algorithm are used to process the first group of feature points and the second group of feature points to obtain target feature point pairs, and a homography matrix is ​​obtained based on the target feature point pairs.

[0049] As an optional implementation method, the target feature point pairs are obtained based on the first group of feature points and the second group of feature points, including: using a target feature matching algorithm to match the first group of feature points and the second group of feature points to obtain initial feature point pairs, wherein the initial feature point pairs include target feature point pairs and noise point pairs; using a random sampling consensus algorithm to eliminate noise point pairs in the initial feature points to obtain target feature point pairs.

[0050] In this embodiment, a target feature matching algorithm is used to match the first group of feature points with the second group of feature points to obtain initial feature point pairs, and a random sampling consensus algorithm is used to eliminate noise point pairs in the initial feature points to obtain target feature point pairs.

[0051] As an optional implementation mode, after determining the geographic location of the target object based on the pixel coordinates of the target object in the satellite image tile, the method also includes: updating the target list based on the geographic location of the target object to obtain an updated target list; saving the labeling information of the target object and the updated target list, and sending the labeling information of the target object and the updated target list to the target user.

[0052] In this embodiment, the geographic location of the target object is stored in the target list to obtain an updated target list; the marking information of the target object and the updated target list are saved, and the marking information of the target object and the updated target list are sent to the target user.

[0053] In an embodiment of the present invention, an initial data set collected by a drone is received, and based on the initial data set, an original target image and platform auxiliary information are obtained; based on the platform auxiliary information, the geographical area covered by the original target image is determined; based on the geographical area, satellite image tiles corresponding to the geographical area are retrieved from a satellite tile database; a target feature extraction algorithm is used to extract feature points of the original target image and the satellite image tiles, respectively, to obtain a first group of feature points corresponding to the original target image and a second group of feature points corresponding to the satellite image tiles; based on the first group of feature points and the second group of feature points, a homography matrix is ​​obtained; a target detection algorithm is used to perform target detection on the original target image to obtain annotation information and a target list of target objects in the original target image, wherein the target list includes a target name and target pixel coordinates of the target object; based on the target pixel coordinates and the homography matrix, the pixel coordinates of the target object in the satellite image tile are determined; based on the pixel coordinates of the target object in the satellite image tile, the geographical location of the target object is determined, thereby solving the problem of image detection by drones in the prior art. During the analysis process, the technical problem of not being able to quickly close the loop and quickly provide information on target objects in the images taken by drones was solved. The satellite tile data was introduced into the geographic information processing of drone images. The geographic boundaries of the image were accurately calculated through platform auxiliary information, and the satellite tile data matching the range was quickly retrieved, which greatly improved the efficiency and accuracy of geographic information association. In this process, advanced image feature extraction and feature matching techniques were used to calculate the homography matrix of the drone image mapped to the satellite tile data, thereby indirectly establishing a mapping relationship between the drone image and the geographic coordinate system. This mapping ensures the geographic consistency of the image data, making subsequent geographic information applications more accurate and efficient. Through this innovative mapping method, each pixel point of the drone image can be quickly located to the corresponding geographic coordinates, providing a solid data foundation for further analysis of the targets in the image. By rationally utilizing satellite tile data, the cumbersome geographic data matching process in traditional methods is avoided, and the technical effect of significantly improving the overall computing efficiency is achieved.

[0054] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0055] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0056] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0057] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

Claims

1. A method for determining the position of a target object in an unmanned aerial vehicle image, characterized in that: include: Receive the initial data set collected by the UAV, and obtain the original target image and platform auxiliary information based on the initial data set; Based on the platform auxiliary information, determine the geographical area covered by the original target image; Based on the geographical area, the satellite image tiles corresponding to the geographical area are retrieved from the satellite tile database; The target feature extraction algorithm is used to extract the feature points of the original target image and the satellite image tile respectively, so as to obtain the first group of feature points corresponding to the original target image and the second group of feature points corresponding to the satellite image tile; Based on the first set of feature points and the second set of feature points, a homography matrix is ​​obtained; Performing target detection on the original target image using a target detection algorithm to obtain labeling information and a target list of the target object in the original target image, wherein the target list includes a target name and a target pixel coordinate of the target object; Based on the target pixel coordinates and homography matrix, the pixel coordinates of the target object in the satellite image tile are determined; The geographic location of the target object is determined based on the pixel coordinates of the target object in the satellite image tile.

2. The method according to claim 1, characterized in that The original target image and platform auxiliary information are obtained based on the initial data set, including: The initial data set is decoded to obtain the original target image and platform auxiliary information.

3. The method according to claim 2, characterized in that The platform auxiliary information includes: the longitude of the drone position, the latitude of the drone position, the roll angle of the drone, the heading angle of the drone, the azimuth angle of the optoelectronic platform, the pitch angle of the optoelectronic platform and the focal length of the optoelectronic platform.

4. The method according to claim 3, characterized in that The determining of the geographical area covered by the original target image based on the platform auxiliary information includes: Based on the platform auxiliary information, the four-dimensional geographic coordinates of the original target image are obtained; Based on the four-dimensional geographic coordinates, determine the geographic area covered by the original target image.

5. The method according to claim 1, characterized in that The step of obtaining a homography matrix based on the first set of feature points and the second set of feature points includes: Based on the first set of feature points and the second set of feature points, a target feature point pair is obtained; Based on the target feature point pairs, the homography matrix is ​​obtained.

6. The method according to claim 5, characterized in that The step of obtaining a target feature point pair based on the first set of feature points and the second set of feature points includes: Using a target feature matching algorithm to match the first group of feature points with the second group of feature points to obtain initial feature point pairs, wherein the initial feature point pairs include target feature point pairs and noise point pairs; The random sampling consensus algorithm is used to remove the noise point pairs in the initial feature points to obtain the target feature point pairs.

7. The method according to claim 1, characterized in that After determining the geographic location of the target object based on the pixel coordinates of the target object in the satellite image tile, the method further includes: updating the target list based on the geographic location of the target object to obtain an updated target list; The target object's annotation information and the updated target list are saved, and the target object's annotation information and the updated target list are sent to the target user.

8. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

9. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

10. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images