Multi-machine mutual dynamic monitoring method and system based on unmanned aerial vehicle carrying

Abstract

The invention discloses a multi-machine mutual dynamic monitoring method and system based on unmanned aerial vehicle carrying, and relates to the technical field of target monitoring. When the target object deviates from the view field range of the camera of the first unmanned aerial vehicle, SIFT feature extraction is carried out on a first image shot by the first unmanned aerial vehicle and second images shot by a plurality of second unmanned aerial vehicles; matching the SIFT features of the first image with the SIFT features of the second images to obtain homonymy point pairs of the second images and the first image; if the homonymy point pair of the second image and the first image shot by a certain second unmanned aerial vehicle contains a key point of a target object in the first image, taking the target unmanned aerial vehicle as a to-be-switched target unmanned aerial vehicle, and taking the homonymy point pair of the shot second image and the first image as a target homonymy point pair; performing gross error elimination on the target homonymy point pairs; and adjusting exterior orientation elements of the camera on the target unmanned aerial vehicle through a bundle method, and switching a monitoring image into an image shot by the target unmanned aerial vehicle. According to the method, the computing power resource consumption can be reduced while the target is dynamically monitored.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of target monitoring, and particularly relates to a multi-machine mutual dynamic monitoring method and system based on unmanned aerial vehicles. BACKGROUND

[0002] With the progress of science and technology, the technology of unmanned aerial vehicles has developed rapidly, and its application in various fields is also becoming more and more widespread. The multi-machine mutual dynamic monitoring technology based on unmanned aerial vehicles can use multiple cameras carried by multiple unmanned aerial vehicles to cooperatively monitor a target area, and through communication and data analysis and recognition between the background and the unmanned aerial vehicles, dynamic monitoring of the target can be achieved.

[0003] At present, the multi-machine mutual dynamic monitoring technology based on unmanned aerial vehicles uses multiple unmanned aerial vehicles, and each unmanned aerial vehicle collects images in real time and feeds back the images to the background. If dynamic tracking of the target is to be achieved, real-time target recognition analysis of the images collected by each unmanned aerial vehicle is required, which results in a large amount of computing resources being consumed.

[0004] Therefore, how to provide an effective solution to reduce the consumption of computing resources while achieving dynamic monitoring of the target has become a difficult problem to be solved in the prior art. SUMMARY

[0005] The purpose of the present application is to provide a multi-machine mutual dynamic monitoring method and system based on unmanned aerial vehicles to solve the above problems existing in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a multi-machine mutual dynamic monitoring method based on unmanned aerial vehicles, comprising: When the target object to be monitored deviates from the camera field of view range of the first unmanned aerial vehicle, SIFT feature extraction is performed on the first image currently captured by the first unmanned aerial vehicle and containing the target object and the second image captured by the multiple second unmanned aerial vehicles cooperatively monitored; The SIFT features of the first image are matched with the SIFT features of the second images captured by each second unmanned aerial vehicle, to obtain homonym point pairs of the second images captured by each second unmanned aerial vehicle and the first image; If the homonym point pairs of the second image captured by a certain second unmanned aerial vehicle and the first image contain the key points of the target object in the first image, the certain second unmanned aerial vehicle is taken as a target unmanned aerial vehicle to be switched, and the homonym point pairs of the second image captured by the certain second unmanned aerial vehicle and the first image are taken as target homonym point pairs; The target homonym point pairs are subjected to gross error elimination through a RANSAC algorithm, wherein the gross error is an abnormal point pair in the target homonym point pairs; The feature points in the target homonym point pairs after the gross error elimination are taken as control points in the first image, the exterior orientation elements of the camera of the first unmanned aerial vehicle are fixed, and the exterior orientation elements of the camera on the target unmanned aerial vehicle are adjusted through a bundle adjustment to make the center of the field of view of the camera on the target unmanned aerial vehicle face the target object, the exterior orientation elements of the camera including the pitch angle, the roll angle and the yaw angle of the camera; The monitoring image is switched to the image captured by the camera on the target unmanned aerial vehicle.

[0007] In one possible design, the method further includes: The target object in the image captured by the first unmanned aerial vehicle is identified through a target recognition method; The distance between the target object and the center point or the edge of the continuous multiple frames of images captured by the first unmanned aerial vehicle is calculated; If the distance between the target object and the center point of the continuous multiple frames of images exceeds a first preset distance, or the distance between the target object and one edge of the continuous multiple frames of images is lower than a second preset distance, it is determined that the target object being monitored deviates from the field of view of the camera of the first unmanned aerial vehicle.

[0008] In one possible design, after the monitoring image is switched to the image captured by the camera on the target unmanned aerial vehicle, the method further includes: The image captured by the camera on the target unmanned aerial vehicle is scaled or the focal length of the camera on the target unmanned aerial vehicle is adjusted to scale the image captured by the camera on the target unmanned aerial vehicle.

[0009] In one possible design, the scaling of the image captured by the camera on the target unmanned aerial vehicle includes: The target object in the image captured by the camera on the target unmanned aerial vehicle is identified through a target recognition method; The image captured by the camera on the target unmanned aerial vehicle is scaled based on the size of the target object in the image captured by the first unmanned aerial vehicle and the size of the target object in the image captured by the camera on the target unmanned aerial vehicle.

[0010] In one possible design, the matching of the SIFT features of the first image with the SIFT features of the second image captured by the second unmanned aerial vehicle to obtain the homonym point pairs between the second image captured by the second unmanned aerial vehicle and the first image includes: The descriptors corresponding to the key points in the first image are matched with the descriptors corresponding to the key points in the second image captured by the second unmanned aerial vehicle to obtain the matching degrees between each key point in the first image and the key points in the second image; determine the initial corresponding point pairs between the second image captured by the second unmanned aerial vehicle and the first image based on the matching degrees between each key point in the first image and each key point in the second image; remove abnormal corresponding point pairs from the initial corresponding point pairs based on geometric constraints to obtain the corresponding point pairs between the second image captured by the second unmanned aerial vehicle and the first image.

[0011] In one possible design, the matching degrees between each key point in the first image and each key point in the second image are determined by matching the descriptors corresponding to the key points in the first image with the descriptors corresponding to the key points in the second image, including: calculating the cosine similarity between the descriptors corresponding to the key points in the first image and the descriptors corresponding to the key points in the second image; determining the matching degrees between each key point in the first image and each key point in the second image based on the cosine similarity between the descriptors corresponding to the key points in the first image and the descriptors corresponding to the key points in the second image.

[0012] In one possible design, the RANSAC algorithm is used to remove outliers from the target corresponding point pairs, including: constructing a similarity transformation model; selecting multiple groups of point pairs from the target corresponding point pairs; solving the model parameters of the similarity transformation model based on the multiple groups of point pairs to obtain an initial similarity transformation model; substituting all the point pairs in the target corresponding point pairs into the initial similarity transformation model to calculate the projection error of each group of point pairs under the similarity transformation model; regarding the point pairs with the projection error less than a preset projection error threshold as inliers, and regarding the point pairs with the projection error greater than or equal to the preset projection error threshold as outliers; removing the outliers from the target corresponding point pairs; reselecting multiple groups of point pairs from the target corresponding point pairs to solve the model parameters of the similarity transformation model, and calculating the projection error of the remaining point pairs in the target corresponding point pairs after removing the outliers under the similarity transformation model, until an iteration stopping condition is met, using the remaining inliers to calculate the model parameters of the similarity transformation model to obtain a latest similarity transformation model; substituting all the point pairs in the target corresponding point pairs into the latest similarity transformation model to calculate the projection error of each group of point pairs under the latest similarity transformation model; regarding the point pairs with the projection error less than a preset projection error threshold as inliers, and regarding the point pairs with the projection error greater than or equal to the preset projection error threshold as outliers and removing the outliers.

[0013] In a second aspect, the present application provides a multi-machine dynamic monitoring system based on unmanned aerial vehicles, comprising: a feature extraction unit configured to perform SIFT feature extraction on a first image captured by a first unmanned aerial vehicle and a second image captured by a second unmanned aerial vehicle when a target object deviates from a camera field of view range of the first unmanned aerial vehicle, wherein the first image contains the target object and the second image is captured by the second unmanned aerial vehicle for cooperative monitoring; a feature matching unit configured to match SIFT features of the first image with SIFT features of the second image captured by each second unmanned aerial vehicle to obtain homonym point pairs of the second image captured by each second unmanned aerial vehicle and the first image; a first switching unit configured to, if the second image captured by a certain second unmanned aerial vehicle contains a key point of the target object in the homonym point pairs of the second image captured by the certain second unmanned aerial vehicle and the first image, take the certain second unmanned aerial vehicle as a target unmanned aerial vehicle to be switched, and take the homonym point pairs of the second image captured by the certain second unmanned aerial vehicle and the first image as target homonym point pairs; a rejection unit configured to perform gross error rejection on the target homonym point pairs by using a RANSAC algorithm, wherein the gross error is an abnormal point pair in the target homonym point pairs; an adjustment unit configured to take feature points in the first image corresponding to the target homonym point pairs after the gross error rejection as control points, fix exterior orientation elements of a camera of the first unmanned aerial vehicle, and adjust the exterior orientation elements of the camera on the target unmanned aerial vehicle by using a bundle adjustment method so that a field of view center of the camera on the target unmanned aerial vehicle is directed towards the target object, wherein the exterior orientation elements of the camera include a pitch angle, a roll angle and a yaw angle of the camera; a second switching unit configured to switch a monitoring image to an image captured by the camera on the target unmanned aerial vehicle.

[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor and a transceiver connected in sequence, wherein the memory is configured to store a computer program, the transceiver is configured to receive and send messages, and the processor is configured to read the computer program and execute the multi-machine dynamic monitoring method based on unmanned aerial vehicles according to the first aspect or any possible design of the first aspect.

[0015] In a fourth aspect, the present application provides a computer readable storage medium having instructions stored thereon, wherein when the instructions are executed on a computer, the multi-machine dynamic monitoring method based on unmanned aerial vehicles according to the first aspect or any possible design of the first aspect is executed.

[0016] In a fifth aspect, the present application provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method for dynamic monitoring of a target object by multiple unmanned aerial vehicles according to the first aspect or any possible design of the first aspect.

[0017] Advantages: The present application extracts SIFT features from a first image captured by a first unmanned aerial vehicle and a second image captured by a second unmanned aerial vehicle when the target object deviates from the field of view of the first unmanned aerial vehicle, matches the SIFT features of the first image with the SIFT features of the second image captured by the second unmanned aerial vehicle, obtains corresponding point pairs of the second image captured by the second unmanned aerial vehicle and the first image, selects the second unmanned aerial vehicle as a target unmanned aerial vehicle if the corresponding point pairs of the second image captured by the second unmanned aerial vehicle and the first image contain key points of the target object in the first image, and selects the corresponding point pairs of the second image captured by the second unmanned aerial vehicle and the first image as target corresponding point pairs. The present application performs gross error elimination on the target corresponding point pairs by using a RANSAC algorithm, selects feature points in the first image as control points from the target corresponding point pairs after the gross error elimination, fixes the exterior orientation elements of the camera of the first unmanned aerial vehicle, adjusts the exterior orientation elements of the camera of the target unmanned aerial vehicle by using a bundle adjustment method, so that the center of the field of view of the camera of the target unmanned aerial vehicle is directed towards the target object, and switches the monitoring image to the image captured by the camera of the target unmanned aerial vehicle. In this way, when the target object deviates from the monitoring range of the unmanned aerial vehicle, the target unmanned aerial vehicle covering the target object can be determined based on SIFT feature matching, the exterior orientation elements of the camera of the target unmanned aerial vehicle are adjusted by using a bundle adjustment method, so that the center of the field of view of the camera of the target unmanned aerial vehicle is directed towards the target object, and finally the monitoring image is switched to the image captured by the camera of the target unmanned aerial vehicle. Thus, the monitoring image can be smoothly transitioned to the monitoring image of another unmanned aerial vehicle, dynamic tracking and monitoring of the target object are achieved, and real-time target recognition and analysis of the images collected by each unmanned aerial vehicle are not required in the process. Therefore, the algorithm resource consumption can be reduced while the dynamic monitoring of the target object is achieved, which facilitates practical application and promotion. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of the method for dynamic monitoring of a target object by multiple unmanned aerial vehicles according to an embodiment of the present application is shown in the figure. Figure 2 A block diagram of the system for dynamic monitoring of a target object by multiple unmanned aerial vehicles according to an embodiment of the present application is shown in the figure. Figure 3 A block diagram of the electronic device according to an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the present application will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the drawings only constitutes some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application.

[0020] It should be understood that although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element can be called a second element, and similarly a second element can be called a first element without departing from the scope of the example embodiments of the present application.

[0021] It should be understood that for the term "and / or" which can appear in the present application, it only describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which means that there are three cases of A alone, B alone, and A and B together. For the term "and" which can appear in the present application, it describes another association object relationship, which means that there can be two kinds of relationships, for example, A and B, which means that there are two cases of A alone and A and B together. In addition, for the character " / " which can appear in the present application, it generally means that the associated objects before and after are an "or" relationship.

[0022] It should be understood that in the following description, specific details are provided in order to facilitate a complete understanding of the example embodiments. However, those skilled in the art should understand that the example embodiments can be implemented without these specific details. For example, systems can be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other instances, well-known processes, structures and techniques can not be shown in unnecessary detail in order to avoid obscuring the example embodiments.

[0023] In order to facilitate the dynamic monitoring of the target object, the present application provides a multi-machine dynamic monitoring method and system based on a UAV, which can reduce the consumption of computing resources while achieving dynamic monitoring of the target.

[0024] The multi-machine dynamic monitoring method based on a UAV provided by the present application can be applied to a background monitoring terminal in communication connection with multiple cooperative monitoring UAVs. It can be understood that the execution subject does not constitute a limitation on the embodiments of the present application.

[0025] The following will provide a detailed description of the multi-aircraft dynamic monitoring method based on UAVs provided in the embodiments of this application.

[0026] like Figure 1 The diagram shown is a flowchart of a multi-aircraft dynamic monitoring method based on a drone, provided in the first aspect of the present application. This multi-aircraft dynamic monitoring method based on a drone may include, but is not limited to, the following steps S101-S106.

[0027] Step S101. When the monitored target object deviates from the camera field of view of the first UAV, SIFT feature extraction is performed on the first image captured by the first UAV that contains the target object and the second image captured by multiple second UAVs that are cooperating in the monitoring.

[0028] In one or more embodiments, the camera lens of the UAV is adjustable. During the monitoring of the target object, the first UAV can dynamically adjust the angle of its camera lens as the target object moves. When the adjustment angle of the camera lens of the first UAV reaches the maximum adjustment angle and the monitored target object deviates from the field of view of the first UAV's camera, SIFT (Scale-invariant feature transform) feature extraction can be performed on the first image captured by the first UAV that contains the target object and the second image captured by multiple second UAVs that are cooperating in the monitoring.

[0029] When identifying whether the monitored target object deviates from the camera field of view of the first UAV, the target object in the image captured by the first UAV can be identified first by the target recognition method, and the distance between the target object and the center point or edge of the consecutive frames captured by the first UAV can be calculated. If the distance between the target object and the center point of the consecutive frames exceeds the first preset distance, or the distance between the target object and one of the edges of the consecutive frames is lower than the second preset distance, it is determined that the monitored target object deviates from the camera field of view of the first UAV.

[0030] When performing SIFT extraction on the first image and the second images captured by each of the second UAVs, key points and their corresponding SIFT descriptors in the first image and key points and their corresponding descriptors in the second images captured by each of the second UAVs can be extracted.

[0031] Step S102. Match the SIFT features of the first image with the SIFT features of the second images captured by each second UAV to obtain the corresponding point pairs between the second images captured by each second UAV and the first image.

[0032] In one or more embodiments, matching the SIFT features of the first image with the SIFT features of the second images captured by each second UAV to obtain the corresponding point pairs between the second images captured by each second UAV and the first image may include, but is not limited to, the following steps S1021-S1023.

[0033] Step S1021. Match the descriptors corresponding to each key point in the first image with the descriptors corresponding to each key point in the second image captured by the second UAV to obtain the matching degree between each key point in the first image and each key point in the second image.

[0034] In one or more embodiments, the cosine similarity between the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image can be calculated. Based on the cosine similarity between the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image, the matching degree between each key point in the first image and each key point in the second image can be obtained. A higher cosine similarity indicates a higher matching degree.

[0035] Understandably, in some other embodiments, the Euclidean distance between the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image can also be calculated, and the matching degree between each key point in the first image and each key point in the second image can be obtained based on the Euclidean distance between the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image. The smaller the Euclidean distance, the higher the matching degree.

[0036] Step S1022. Based on the matching degree between each key point in the first image and each key point in the second image, determine the initial corresponding point pairs between the second image captured by the second UAV and the first image.

[0037] In one or more embodiments, for any key point in the first image, a key point in the second image with the highest matching degree and above a preset matching degree threshold can be selected to form a homonymous point pair with the key point. A homonymous point pair represents two key points from two different images that represent the same physical point.

[0038] Step S1023. Based on geometric constraints, remove abnormal corresponding point pairs from the initial corresponding point pairs to obtain corresponding point pairs between the second image captured by the second UAV and the first image.

[0039] In one or more embodiments, after determining the initial homonymous point pairs between the second image captured by the second drone and the first image, it can be determined based on geometric constraints whether there are abnormal homonymous point pairs in the initial homonymous point pairs. If there are abnormal homonymous point pairs, the abnormal homonymous point pairs in the initial homonymous point pairs are removed to obtain the homonymous point pairs between the second image captured by the second drone and the first image.

[0040] Specifically, determining whether a pair of initial identical points is an anomalous pair based on geometric constraints involves checking whether the keypoints adjacent to the first keypoint in the first image and the keypoints adjacent to the second keypoint in the second image in the same direction also form identical point pairs. If so, the pair of initial identical points is considered normal; otherwise, it is considered anomalous.

[0041] Step S103. If the second image captured by a certain second UAV contains key points of the target object in the first image in a pair of corresponding points, then the certain second UAV is used as the target UAV to be switched, and the pair of corresponding points captured by the second UAV and the first image is used as the target pair of corresponding points.

[0042] If the second image captured by a second drone matches the corresponding point pair in the first image, it means that the target object is within the field of view of the camera of the second drone. In this case, the second drone can be used as the target drone to be switched, and the corresponding point pair between the second image captured by the second drone and the first image can be used as the target corresponding point pair.

[0043] Understandably, if multiple second images captured by second drones contain key points of the target object in the first image within corresponding point pairs, then based on the positions of the key points of the target object in each second image, the second image whose key point is closest to the center point can be selected as the target image. The second drone corresponding to the target image will then be selected as the target drone to be switched, and the corresponding point pairs between the target image and the first image will be selected as the target corresponding point pairs.

[0044] Step S104. Use the RANSAC algorithm to remove gross errors from the target pairs of points with the same name.

[0045] Gross errors refer to aberrant point pairs among the target pairs of corresponding points.

[0046] In one or more embodiments, the gross error removal of target homonymous point pairs by using the RANSAC (Random Sample Consensus) algorithm may include, but is not limited to, the following steps S1041-S1049.

[0047] Step S1041. Construct a similarity transformation model.

[0048] Step S1042. Select multiple sets of point pairs from the target pairs of points with the same name.

[0049] Step S1043. Solve the model parameters of the similarity transformation model based on the multiple sets of point pairs to obtain the initial similarity transformation model.

[0050] Step S1044. Substitute all point pairs in the target pair of points into the initial similarity transformation model, and calculate the projection error of each point pair under the similarity transformation model.

[0051] Step S1045. Select point pairs with projection errors less than a preset projection error threshold as interior points, and select point pairs with projection errors greater than or equal to the preset projection error threshold as exterior points.

[0052] Step S1046. Remove the outside points from the target pair of points with the same name.

[0053] Step S1047. Select multiple sets of point pairs from the target corresponding point pairs to solve the model parameters of the similarity transformation model, and calculate the projection error of the remaining point pairs after removing outliers in the target corresponding point pairs under the similarity transformation model until the iteration stopping condition is met. Then, use the remaining interior points to calculate the model parameters of the similarity transformation model to obtain the latest similarity transformation model.

[0054] The iteration stopping condition can be either reaching the preset maximum number of iterations or the number of interior points remaining unchanged.

[0055] Step S1048. Substitute all point pairs in the target pair of points into the latest similarity transformation model, and calculate the projection error of each point pair under the latest similarity transformation model.

[0056] Step S1049. Select point pairs with projection errors less than the preset projection error threshold as interior points, and select point pairs with projection errors greater than or equal to the preset projection error threshold as exterior points and discard them.

[0057] Step S105. Use the feature points in the first image corresponding to the target corresponding point pairs after gross error removal as control points, fix the exterior orientation elements of the camera of the first UAV, and adjust the exterior orientation elements of the camera on the target UAV by bundle adjustment so that the field of view center of the camera on the target UAV faces the target object.

[0058] The camera's external orientation elements include its pitch angle, roll angle, and yaw angle.

[0059] Bundle adjustment (BA) is an algorithm that optimizes multi-view geometry parameters by minimizing reprojection errors. It is widely used in photogrammetry, computer vision and other fields, and its principles will not be elaborated here.

[0060] Step S106. Switch the monitoring image to the image captured by the camera on the target drone.

[0061] In one or more embodiments, after switching the monitoring image to an image captured by a camera on the target drone, the image captured by the camera on the target drone can be scaled or the focal length of the camera on the target drone can be adjusted to scale the image captured by the camera on the target drone. When scaling the image captured by the camera on the target drone, the target object in the image captured by the camera on the target drone can be identified by a target recognition method. Based on the size of the target object in the image captured by the first drone and the size of the target object in the image captured by the camera on the target drone, the image captured by the camera on the target drone is scaled.

[0062] The present invention provides a multi-drone dynamic monitoring method based on UAVs. When the monitored target object deviates from the field of view of the first UAV's camera, SIFT features are extracted from a first image captured by the first UAV containing the target object and second images captured by multiple co-monitoring second UAVs. The SIFT features of the first image are matched with the SIFT features of the second images captured by each of the second UAVs to obtain corresponding point pairs between the second images captured by each of the second UAVs and the first image. If a corresponding point pair between a second image captured by a certain second UAV and the first image contains the target object from the first image... The key point is to use a second drone as the target drone to be switched to, and to use the corresponding point pairs between the second image captured by the second drone and the first image as target corresponding point pairs. The RANSAC algorithm is used to remove gross errors from the target corresponding point pairs, where gross errors are abnormal point pairs. The feature points in the first image corresponding to the target corresponding point pairs after gross error removal are used as control points. The exterior orientation elements of the camera on the first drone are fixed, and the exterior orientation elements of the camera on the target drone are adjusted using bundle adjustment so that the field of view center of the camera on the target drone faces the target object. The monitoring image is then switched to the image captured by the camera on the target drone. Thus, when the monitored target object leaves the monitoring range of the drone, the target drone covering the target object can be determined based on SIFT feature matching. Then, the exterior orientation elements of the camera on the target drone are adjusted by bundle adjustment so that the field of view center of the camera on the target drone faces the target object. Finally, the monitoring image is switched to the image captured by the camera on the target drone. This allows for a very natural and smooth transition from the monitoring screen to the monitoring screen of another drone, realizing dynamic tracking and monitoring of the target object. In this process, there is no need to perform real-time target recognition and analysis on the images collected by each drone. This can reduce the consumption of computing resources while realizing dynamic monitoring of the target, making it convenient for practical application and promotion.

[0063] Please see Figure 2 The second aspect of this application provides a multi-aircraft dynamic monitoring system based on a drone, the multi-aircraft dynamic monitoring system based on a drone includes: The feature extraction unit is used to perform SIFT feature extraction on a first image captured by the first drone that contains the target object and a second image captured by multiple second drones that are monitored in cooperation when the monitored target object deviates from the camera field of view of the first drone. The feature matching unit is used to match the SIFT features of the first image with the SIFT features of the second images captured by each second UAV to obtain the corresponding point pairs between the second images captured by each second UAV and the first image. The first switching unit is configured to, if a second image captured by a certain second drone contains key points of a target object in the first image in a pair of corresponding points, then use the second drone as the target drone to be switched, and use the pair of corresponding points of the second image captured by the second drone and the first image as the target pair of corresponding points. The elimination unit is used to perform gross error elimination on the target homonymous point pairs using the RANSAC algorithm, wherein the gross error refers to the abnormal point pairs among the target homonymous point pairs; The adjustment unit is used to take the feature points in the first image corresponding to the target homonym point pair after gross error removal as control points, fix the exterior orientation elements of the camera of the first UAV, and adjust the exterior orientation elements of the camera on the target UAV by bundle adjustment so that the field of view center of the camera on the target UAV faces the target object. The exterior orientation elements of the camera include the camera's pitch angle, roll angle and yaw angle. The second switching unit is used to switch the monitoring image to the image captured by the camera on the target drone.

[0064] The working process, working details and technical effects of the multi-aircraft mutual dynamic monitoring system based on UAVs provided in the second aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0065] Please see Figure 3 The third aspect of this application provides an electronic device, including a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the multi-machine mutual dynamic monitoring method based on a drone as described in the first aspect of the application.

[0066] Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or last-in-first-out (FILO) memory, etc.; the processor may not be limited to microprocessors of the STM32F105 series, ARM (Advanced RISC Machines), x86 architecture processors, or processors with integrated NPU (neural-network processing units); the transceiver may be, but is not limited to, WiFi (Wireless Fidelity) wireless transceivers, Bluetooth wireless transceivers, General Packet Radio Service (GPRS) wireless transceivers, ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard), 3G transceivers, 4G transceivers, and / or 5G transceivers, etc.

[0067] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions containing the instructions for the multi-drone dynamic monitoring method based on a drone as described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the multi-drone dynamic monitoring method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0068] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the multi-machine mutual dynamic monitoring method based on a drone as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0069] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dynamic monitoring of multiple machines based on a UAV, characterized in that, The method comprises: When the monitored target object deviates from the camera field of view range of the first unmanned aerial vehicle, SIFT feature extraction is performed on a first image currently photographed by the first unmanned aerial vehicle and containing the target object and a second image photographed by a second unmanned aerial vehicle in cooperative monitoring; The SIFT features of the first image are matched with the SIFT features of the second image photographed by each second unmanned aerial vehicle, to obtain corresponding point pairs of the second image photographed by each second unmanned aerial vehicle and the first image; If the corresponding point pairs of the second image photographed by a certain second unmanned aerial vehicle and the first image contain key points of the target object in the first image, the certain second unmanned aerial vehicle is taken as a target unmanned aerial vehicle to be switched, and the corresponding point pairs of the second image photographed by the certain second unmanned aerial vehicle and the first image are taken as target corresponding point pairs; RANSAC algorithm is used to remove gross errors in the target corresponding point pairs, wherein the gross errors are abnormal point pairs in the target corresponding point pairs; The feature points in the first image corresponding to the target corresponding point pairs after the gross errors are removed are taken as control points, the exterior orientation elements of the camera of the first unmanned aerial vehicle are fixed, the exterior orientation elements of the camera on the target unmanned aerial vehicle are adjusted by bundle adjustment, so that the field of view center of the camera on the target unmanned aerial vehicle is directed towards the target object, and the exterior orientation elements of the camera include the pitch angle, roll angle and yaw angle of the camera; The monitoring image is switched to an image photographed by the camera on the target unmanned aerial vehicle. 2.The UAV-mounted multi-machine interaction dynamic monitoring method according to claim 1, characterized in that, The method further comprises: The target object in the image photographed by the first unmanned aerial vehicle is identified by a target recognition method; The distance between the target object and the center point or edge of a plurality of continuous images photographed by the first unmanned aerial vehicle is calculated; If the distance between the target object and the center point of the plurality of continuous images exceeds a first preset distance, or the distance between the target object and one edge of the plurality of continuous images is lower than a second preset distance, it is determined that the monitored target object deviates from the camera field of view range of the first unmanned aerial vehicle. 3.The UAV-mounted multi-machine interaction dynamic monitoring method of claim 2, wherein, After the monitoring image is switched to the image photographed by the camera on the target unmanned aerial vehicle, the method further comprises: The image photographed by the camera on the target unmanned aerial vehicle is scaled or the focal length of the camera on the target unmanned aerial vehicle is adjusted to scale the image photographed by the camera on the target unmanned aerial vehicle. 4.The UAV-mounted multi-machine interaction dynamic monitoring method of claim 3, wherein, The scaling of the image photographed by the camera on the target unmanned aerial vehicle comprises: The target object in the image photographed by the camera on the target unmanned aerial vehicle is identified by a target recognition method; The image photographed by the camera on the target unmanned aerial vehicle is scaled based on the size of the target object in the image photographed by the first unmanned aerial vehicle and the size of the target object in the image photographed by the camera on the target unmanned aerial vehicle. 5.The UAV-mounted multi-machine interaction dynamic monitoring method of claim 1, wherein, The matching of the SIFT features of the first image with the SIFT features of the second image photographed by the second unmanned aerial vehicle to obtain the corresponding point pairs of the second image photographed by the second unmanned aerial vehicle and the first image comprises: The descriptors corresponding to each key point in the first image are matched with the descriptors corresponding to each key point in the second image photographed by the second unmanned aerial vehicle, to obtain the matching degrees of each key point in the first image and each key point in the second image; determine, based on matching degrees of each key point in the first image and each key point in the second image, initial corresponding point pairs of the second image captured by the second unmanned aerial vehicle and the first image; remove abnormal corresponding point pairs in the initial corresponding point pairs based on geometric constraints to obtain corresponding point pairs of the second image captured by the second unmanned aerial vehicle and the first image. 6.The UAV-mounted multi-machine interaction dynamic monitoring method according to claim 5, wherein, match the descriptors corresponding to each key point in the first image with the descriptors corresponding to each key point in the second image to obtain the matching degrees of each key point in the first image and each key point in the second image, including: calculating cosine similarities of the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image; obtaining the matching degrees of each key point in the first image and each key point in the second image based on the cosine similarities of the descriptors corresponding to each key point in the first image and the descriptors corresponding to each key point in the second image. 7.The UAV-mounted multi-machine interaction dynamic monitoring method of claim 1, wherein, performing gross error removal on the target corresponding point pairs through a RANSAC algorithm, including: constructing a similarity transformation model; selecting multiple groups of point pairs from the target corresponding point pairs; solving model parameters of the similarity transformation model based on the multiple groups of point pairs to obtain an initial similarity transformation model; substituting all point pairs in the target corresponding point pairs into the initial similarity transformation model to calculate projection errors of each group of point pairs under the similarity transformation model; regarding point pairs with projection errors less than a preset projection error threshold as inliers and regarding point pairs with projection errors greater than or equal to the preset projection error threshold as outliers; removing outliers in the target corresponding point pairs; reselecting multiple groups of point pairs from the target corresponding point pairs to solve the model parameters of the similarity transformation model and calculating projection errors of the remaining point pairs in the target corresponding point pairs after removing the outliers under the similarity transformation model until an iteration stopping condition is met, using the remaining inliers to calculate the model parameters of the similarity transformation model to obtain a latest similarity transformation model; substituting all point pairs in the target corresponding point pairs into the latest similarity transformation model to calculate projection errors of each group of point pairs under the latest similarity transformation model; regarding point pairs with projection errors less than a preset projection error threshold as inliers and regarding point pairs with projection errors greater than or equal to the preset projection error threshold as outliers and removing the outliers.

8. An unmanned aerial vehicle-mounted multi-machine mutual dynamic monitoring system, characterized in that, including: a feature extraction unit configured to perform SIFT feature extraction on a first image currently captured by a first unmanned aerial vehicle and containing a target object and second images captured by multiple second unmanned aerial vehicles for cooperative monitoring when the target object deviates from a camera field of view range of the first unmanned aerial vehicle; a feature matching unit configured to match SIFT features of the first image with SIFT features of the second images captured by the second unmanned aerial vehicles to obtain corresponding point pairs of the second images and the first image. The first switching unit is configured to, if a second image captured by a certain second unmanned aerial vehicle contains a key point of a target object in the first image in a homonym point pair of the first image, take the certain second unmanned aerial vehicle as a target unmanned aerial vehicle to be switched, and take the homonym point pair of the first image and the second image captured by the certain second unmanned aerial vehicle as a target homonym point pair; The rejection unit is configured to perform gross error rejection on the target homonym point pair by using a RANSAC algorithm, wherein a gross error is an abnormal point pair in the target homonym point pair; The adjustment unit is configured to take a feature point in the first image corresponding to the target homonym point pair after the gross error rejection as a control point, fix an exterior orientation element of a camera of the first unmanned aerial vehicle, and adjust an exterior orientation element of a camera of the target unmanned aerial vehicle by using a bundle adjustment method, so that a field of view center of the camera of the target unmanned aerial vehicle is directed towards the target object, wherein the exterior orientation element of the camera includes a pitch angle, a roll angle and a yaw angle of the camera; The second switching unit is configured to switch the monitoring image to an image captured by the camera of the target unmanned aerial vehicle.

9. An electronic device, comprising: The computer program or the instruction, when executed by a computer, implements the method for mutual dynamic monitoring of multiple unmanned aerial vehicles based on the unmanned aerial vehicles according to any one of claims 1 to 7.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instruction, when executed by a computer, implements the method for mutual dynamic monitoring of multiple unmanned aerial vehicles based on the unmanned aerial vehicles according to any one of claims 1 to 7.

Images