Real-time video analysis method, device and equipment based on cooperation of multiple unmanned aerial vehicles, and storage medium

Abstract

The invention discloses a real-time video analysis method, device and equipment based on cooperation of multiple unmanned aerial vehicles, and a storage medium, and relates to the technical field of unmanned aerial vehicles, and the method comprises the steps: determining the view image of each unmanned aerial vehicle based on the real-time video data shot by each unmanned aerial vehicle in a target unmanned aerial vehicle group; mapping the view images to a world plane in a unified manner, and constructing a panoramic view image; based on the load image area of each unmanned aerial vehicle, respectively determining a processing image of each unmanned aerial vehicle in the panoramic view image, the processing images at least comprising a non-overlapping view processing image and an overlapping view processing image; and distributing the processed image to the corresponding unmanned aerial vehicle for target detection and cooperative tracking. Through the above mode, the view image is mapped from the pixel plane to the world plane, the panoramic view is created, the workload is allocated to the unmanned aerial vehicles on the panoramic view, the shooting view angles of the multiple unmanned aerial vehicles are cooperated, target detection and tracking are performed, the accuracy is improved, and the time delay is reduced.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a real-time video analysis method, device, equipment and storage medium based on multi-drone collaboration. Background Art

[0002] In recent years, multi-UAV video analysis systems have attracted increasing attention and play a key role in various scenarios such as security surveillance, disaster relief, and environmental monitoring. As the number of drones and their operating time continue to increase, UAV-based video analysis has become more and more data-intensive and computationally intensive. Due to the limited field of view of a single drone, multiple drones are usually required to complete the task collaboratively, but existing methods usually process the video data from each drone independently without considering the association between them, making it difficult to accurately detect and track targets in the scene.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a real-time video analysis method, device, equipment and storage medium based on multi-UAV collaboration, aiming to solve the technical problem in the prior art of independently processing the video data on each UAV and making it difficult to accurately detect and track targets in the scene.

[0005] To achieve the above objectives, the present application provides a real-time video analysis method based on multi-UAV collaboration, the method comprising:

[0006] Based on the real-time video data captured by each drone in the target drone group, determine the field of view image of each drone;

[0007] Mapping the field of view images uniformly to a world plane to construct a panoramic field of view image;

[0008] Based on the load image area of ​​each UAV, respectively determine the processed image of each UAV in the panoramic field of view image, wherein the processed image includes at least a non-overlapping field of view processed image and an overlapping field of view processed image;

[0009] The processed images are distributed to corresponding drones for target detection, and collaborative tracking is performed based on the obtained detection results.

[0010] In one embodiment, the step of uniformly mapping the field of view images to a world plane to construct a panoramic field of view image comprises:

[0011] Acquire a field of view image of each drone in the target drone group before movement, and map the pixel coordinates of the field of view image before movement in the pixel plane to the world plane based on the rotation matrix and the translation matrix obtained by feature matching to obtain the world coordinates of the field of view image before movement in the world plane;

[0012] Generate a panoramic view image based on the world coordinates of the view image of each drone in the target drone group before movement in the world plane;

[0013] After each drone in the target drone group moves, the panoramic field of view image is dynamically updated.

[0014] In one embodiment, the step of dynamically updating the panoramic view image after each drone in the target drone group moves includes:

[0015] After each drone in the target drone group moves, a field of view image of each drone after the movement is obtained, and based on the rotation matrix and the translation matrix, pixel coordinates of the field of view image after the movement in the pixel plane are mapped to the world plane to obtain initial world coordinates of the field of view image after the movement in the world plane;

[0016] Based on the initial world coordinates of the field of view image after the movement in the world plane, determining a scaling matrix in the vertical direction and a new rotation matrix and a new translation matrix in the horizontal direction for each drone;

[0017] Based on the new rotation matrix, the new translation matrix, the scaling matrix and the world coordinates of the field of view image before the movement in the world plane, obtaining the world coordinates of the field of view image of each drone after the movement in the world plane;

[0018] The panoramic view image is updated based on the world coordinates of the field of view images of each drone in the target drone group after movement in the world plane.

[0019] In one embodiment, before the step of determining the processed images of each drone in the panoramic field of view image based on the load image area of ​​each drone, the step further includes:

[0020] Obtaining the inference computing power of each drone in the target drone group;

[0021] Based on the proportional relationship between the inference computing power and the load image area, the load image area of ​​each drone in the target drone group is determined.

[0022] In one embodiment, the step of determining the processed images of each drone in the panoramic field of view image based on the load image area of ​​each drone includes:

[0023] Dividing the panoramic field of view image into a plurality of sub-field of view images;

[0024] Based on the non-overlapping field of view of each drone in the target drone group, respectively determine the non-overlapping field of view processed image of each drone in the multiple sub-field of view images;

[0025] Based on the payload image area of ​​each UAV and the pixel plane area of ​​the non-overlapping field of view processed image, the pixel plane area of ​​the overlapping field of view processed image of each UAV is determined respectively;

[0026] Based on the overlapping field of view of the target drone group, determining a sub-overlapping field of view image from the multiple sub-field of view images;

[0027] The pixel plane area of ​​the overlapping view processing image of each drone is converted into the world plane area of ​​the overlapping view processing image of each drone, and based on the world plane area of ​​the overlapping view processing image of each drone, the overlapping view processing image of each drone is respectively determined in the sub-overlapping view image.

[0028] In one embodiment, the step of determining the pixel plane area of ​​the overlapping view processing image of each drone based on the payload image area of ​​each drone and the pixel plane area of ​​the non-overlapping view processing image comprises:

[0029] When the pixel plane area of ​​the non-overlapping field of view processed image of the drone is greater than or equal to the payload image area, determining that the pixel plane area of ​​the overlapping field of view processed image of the drone is zero;

[0030] When the pixel plane area of ​​the non-overlapping field of view processed image of the drone is smaller than the load image area, the pixel plane area of ​​the overlapping field of view processed image of the drone is determined as the difference between the load image area and the non-overlapping field of view processed image area.

[0031] In one embodiment, the method further comprises:

[0032] Acquiring the confidence of the detection result, determining a low-confidence result based on the confidence of the detection result, and using a processed image corresponding to the low-confidence result as a low-confidence region image;

[0033] The low confidence area image is reallocated to other UAVs for target detection to obtain new detection results that meet the confidence conditions, and the new detection results are collaboratively tracked.

[0034] In addition, to achieve the above purpose, the present application also proposes a real-time video analysis device based on multi-UAV collaboration, and the real-time video analysis device based on multi-UAV collaboration includes:

[0035] The field of view alignment module is used to determine the field of view image of each drone based on the real-time video data shot by each drone in the target drone group;

[0036] The field of view alignment module is further used to uniformly map the field of view images to a world plane to construct a panoramic field of view image;

[0037] A load distribution module, configured to determine, based on the load image area of ​​each UAV, a processed image of each UAV in the panoramic field of view image, wherein the processed image includes at least a non-overlapping field of view processed image and an overlapping field of view processed image;

[0038] The collaborative tracking module is used to distribute the processed image to the corresponding UAV for target detection and perform collaborative tracking based on the obtained detection results.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a real-time video analysis device based on multi-UAV collaboration. The real-time video analysis device based on multi-UAV collaboration includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the real-time video analysis method based on multi-UAV collaboration as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the real-time video analysis method based on multi-UAV collaboration as described above are implemented.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the real-time video analysis method based on multi-UAV collaboration as described above.

[0042] The present application provides a real-time video analysis method based on multi-drone collaboration, based on the real-time video data shot by each drone in the target drone group, the field of view image of each drone is determined; the field of view image is uniformly mapped to the world plane to construct a panoramic field of view image; based on the load image area of ​​each drone, the processing image of each drone is determined in the panoramic field of view image, and the processing image includes at least a non-overlapping field of view processing image and an overlapping field of view processing image; the processed image is assigned to the corresponding drone for target detection, and collaborative tracking is performed based on the obtained detection results. The present application maps the field of view images of all drones in the drone group from the pixel plane to the world plane to create a panoramic field of view, thereby allocating workloads to drones on the panoramic field of view based on the field of view association between drones, and coordinates the shooting perspectives of multiple drones for target detection and tracking, improving accuracy and reducing latency. At the same time, by sharing reasoning tasks and reasoning results between drones, the computing efficiency is improved, solving the technical problem of independently processing video data on each drone and difficulty in accurately detecting and tracking targets in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 This is a flow chart of Embodiment 1 of the real-time video analysis method based on multi-UAV collaboration of the present application;

[0046] Figure 2 A schematic diagram of mapping from a pixel plane to a world plane of a real-time video analysis method based on multi-UAV collaboration provided in Example 1 of the present application;

[0047] Figure 3 This is a flow chart of Embodiment 2 of the real-time video analysis method based on multi-UAV collaboration of the present application;

[0048] Figure 4 A schematic diagram of an overlapping field of view segmentation algorithm for a real-time video analysis method based on multi-UAV collaboration provided in Example 1 of the present application;

[0049] Figure 5 A schematic diagram of the overall architecture of a real-time video analysis method based on multi-UAV collaboration provided in Example 2 of the present application;

[0050] Figure 6 This is a schematic diagram of the module structure of a real-time video analysis device based on multi-UAV collaboration in an embodiment of the present application;

[0051] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the real-time video analysis method based on multi-UAV collaboration in the embodiment of the present application.

[0052] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: based on the real-time video data taken by each drone in the target drone group, determine the field of view image of each drone; uniformly map the field of view images to the world plane to construct a panoramic field of view image; based on the load image area of ​​each drone, respectively determine the processed image of each drone in the panoramic field of view image, the processed images include at least non-overlapping field of view processed images and overlapping field of view processed images; distribute the processed images to the corresponding drones for target detection, and perform collaborative tracking based on the detection results obtained.

[0056] Currently, the video data from each drone is usually processed independently without considering the correlation between them, resulting in low accuracy.

[0057] The present application provides a solution, which maps the field of view images of all drones in a drone swarm from the pixel plane to the world plane to create a panoramic field of view, thereby distributing workloads to drones on the panoramic field of view based on the field of view association between drones, and coordinating the shooting perspectives of multiple drones to perform target detection and tracking, thereby improving accuracy and reducing latency. At the same time, by sharing reasoning tasks and reasoning results among drones, computing efficiency is improved, and the technical problem of independently processing video data on each drone and difficulty in accurately detecting and tracking targets in the scene is solved.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, a real-time video analysis device based on multi-UAV collaboration, etc., and this embodiment does not specifically limit this. The following takes the real-time video analysis device based on multi-UAV collaboration as an example to illustrate this embodiment and the following embodiments.

[0059] The present application embodiment provides a real-time video analysis method based on multi-UAV collaboration, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the real-time video analysis method based on multi-UAV collaboration of the present application.

[0060] In this embodiment, the real-time video analysis method based on multi-UAV collaboration includes steps S10 to S40:

[0061] Step S10, determining the field of view image of each drone based on the real-time video data shot by each drone in the target drone group;

[0062] It should be noted that the moving target can be tracked by analyzing the video footage taken by the drone, and the target can be a person, a vehicle, etc. on the ground, without specific limitation. The target drone group refers to a drone group that needs to perform video analysis to track the target, usually including multiple drones, and the specific number is determined according to the actual situation. The drone in this embodiment has a video shooting function, which can shoot videos within the field of view, and the shooting function can be achieved by setting a camera.

[0063] In addition, it should be noted that the real-time video data refers to the images within the field of view taken by the drone in real time, and the field of view image refers to the images within the field of view of the drone, which can be captured from the real-time video data, and each frame of the real-time video data can be used as the field of view image. Each drone in the target drone group needs to provide a field of view image in order to form a panoramic field of view image.

[0064] Step S20, uniformly mapping the field of view images to a world plane to construct a panoramic field of view image;

[0065] It should be noted that in order to accurately share the inference results later, it is necessary to ensure that the fields of view between all drones in the target drone group are aligned. Generally speaking, when the camera is stationary (such as a surveillance camera), the field of view mapping relationship between cameras is fixed and only needs to be established once. However, drones are mobile, that is, the field of view mapping relationship between the cameras of drones will change over time. Therefore, this embodiment adopts a multi-drone field of view alignment algorithm (MDFA) to align the fields of view between multiple drones. The multi-drone field of view alignment algorithm is a lightweight algorithm that can be updated in real time. It aligns the fields of view between drones by uniformly mapping the field of view images of all drones to the world plane. The field of view mapping relationship will be dynamically updated to ensure that the fields of view can be continuously aligned when the drone moves.

[0066] In a feasible implementation, step S20 may include steps S201 to S203:

[0067] Step S201, obtaining a field of view image of each drone in the target drone group before movement, and mapping the pixel coordinates of the field of view image before movement in the pixel plane to the world plane based on the rotation matrix and the translation matrix obtained by feature matching, so as to obtain the world coordinates of the field of view image before movement in the world plane;

[0068] It should be noted that in order to align the fields of view of multiple drones, a common approach is to establish a pixel-to-pixel correspondence between every two drones. For example, feature point matching is used to derive the homography matrix between the two drones for alignment, or a re-identification (ReID) model is used for target matching to establish an accurate mapping relationship between the perspectives of the two drones. However, in scenarios involving multiple drones, assuming there are n drones in total, traditional methods require calculating n(n-1) / 2 mapping relationships at any time. In addition, these mapping relationships need to be continuously updated when the drone moves, which significantly increases the computational overhead of maintaining the mapping relationship.

[0069] To this end, this embodiment introduces a unified world plane. The pixel plane is the plane where the 2D image is located. The 2D position of each pixel in the 2D image is the pixel coordinate. Under normal circumstances, the pixel coordinates of the pixel in the 2D image can be mapped to the 3D world coordinate. For drones that usually fly from a bird's-eye view, the height difference of targets on the ground (such as people and vehicles) can usually be ignored. Therefore, this embodiment ignores the height difference of the target and only focuses on the coordinates of the horizontal plane to align the field of view. This horizontal plane is the world plane, and the world coordinates are the 2D position of the pixel on the horizontal plane of the real world. Reference Figure 2,This embodiment maps the fields of view of all UAVs’ pixel planes to a unified world plane, and then creates a panoramic field of view (PFoV), which can reduce the number of mapping relationships to n, thereby simplifying the maintenance of the mapping relationships.

[0070] It can be understood that the transformation of the field of view image from the pixel plane to the unified world plane depends on the rotation matrix and the translation matrix. Let the pixel coordinate be p i =[x,y] T (where i represents the i-th drone), the world coordinate is P w =[x w ,y w ] T , we can build p i and P w The mapping relationship between them is as follows:

[0071]

[0072] In the formula, C i represents the internal parameter matrix of the i-th UAV, R i and T i Respectively represent the rotation matrix and translation matrix of the i-th drone relative to the world plane. The internal parameter matrix C i The rotation matrix R can be determined using the Tsai camera calibration method. i and the translation matrix T i It can be obtained through feature matching, and illustratively, estimated using a RANSAC (Random Sample Consensus) algorithm, which is not specifically limited.

[0073] It should be understood that the image of the field of view before the drone moves, that is, the image of the field of view when the drone is not moving, can select the most representative image to reflect the real situation within the field of view of the drone. This embodiment uses the rotation matrix and translation matrix obtained by feature matching to preliminarily establish a mapping relationship between the pixel plane and the world plane, maps the image of the field of view before the drone moves to the world plane, and obtains the world coordinates of the field of view in the world plane.

[0074] Step S202, generating a panoramic view image based on the world coordinates of the view image of each drone in the target drone group before movement in the world plane;

[0075] It should be noted that by integrating the fields of view of all drones in the target drone group into the world plane, a panoramic image, i.e., a panoramic field of view image, can be obtained, thus realizing the association of the fields of view of multiple drones. Figure 2,Panoramic vision images can usually show the overlapping relationship between the ,fields of UAV, so that the overlapping field of vision ,area in the panoramic vision image can be determined. ,Using the mapping relationship between the pixel plane and the world plane, the ,overlapping field of vision area can be converted to the pixel plane to obtain the ,pixel coordinates of the overlapping field of vision area.

[0076] In the specific implementation, there is no need to actually stitch together the field of view images captured by each drone to create a panoramic field of view image. Only the relative positions can be recorded for workload distribution between drones in the subsequent stage. This can ensure efficient collaboration between drones without incurring the computing and network overhead caused by panoramic image stitching.

[0077] Step S203: After each drone in the target drone group moves, the panoramic view image is dynamically updated.

[0078] It is understandable that as the drones move, their rotation matrix R i and the translation matrix T i It will keep changing, so the panoramic view image needs to be updated in real time according to the movement of the drone.

[0079] It should be understood that the rotation matrix R is recalculated for each frame of the field of view image. i and the translation matrix T i It will consume a lot of computing power. Therefore, in this embodiment, the movement of the drone is decomposed into rotation and translation in the horizontal direction and scaling in the vertical direction, and the mapping relationship between the pixel plane and the world plane is updated.

[0080] In a feasible implementation, step S203 may include steps S2031 to S2033:

[0081] Step S2031, after each drone in the target drone group moves, obtain the field of view image of each drone after the movement, and map the pixel coordinates of the field of view image after the movement in the pixel plane to the world plane based on the rotation matrix and the translation matrix to obtain the initial world coordinates of the field of view image after the movement in the world plane;

[0082] It should be noted that the field of view image after the drone moves, that is, the field of view image when the drone moves, first needs to use the original mapping relationship between the pixel plane and the world plane (the mapping relationship determined in step S201) to map the field of view image after the drone moves to the world plane and determine the preliminary world coordinates, that is, the initial world coordinates. The original mapping relationship uses the original rotation matrix and translation matrix (the rotation matrix R determined in step S201) i and translation matrix).

[0083] Step S2032, based on the initial world coordinates of the field of view image after the movement in the world plane, determining the scaling matrix of each drone in the vertical direction and the new rotation matrix and new translation matrix in the horizontal direction;

[0084] It should be noted that the scaling matrix of the drone in the vertical direction is the matrix corresponding to the scaling of the drone in the vertical direction when it moves, the new rotation matrix of the drone in the horizontal direction is the matrix corresponding to the rotation of the drone in the horizontal direction when it moves, and the new translation matrix of the drone in the horizontal direction is the matrix corresponding to the translation of the drone in the horizontal direction when it moves.

[0085] It can be understood that the rotation matrix and translation matrix in the horizontal direction can be estimated by the optical flow method, which can track the pixel movement between consecutive frames, and the scaling matrix in the vertical direction can be estimated by the size ratio of the target in consecutive frames.

[0086] For example, the Shi-Tomasi algorithm is used to extract corner points with obvious texture or intensity changes from the image of the world plane. After identifying these corner points, the Lucas-Kanade optical flow method is used to track the movement of the corner points between consecutive frames, thereby estimating the change ΔR′ of the rotation matrix generated by the horizontal movement of the drone i and the change of the translation matrix ΔT′ i , and then calculate the new rotation matrix R′ i and the new translation matrix T′ i , taking into account the change in target size caused by height change, the average ratio of the bounding box size of the same target in the x direction and the y direction in consecutive frames is calculated to generate the corresponding scaling matrix. The final scaling matrix, new rotation matrix and new translation matrix can be expressed as:

[0087] R′ i ←ΔR′ i ·R i , T′ i ←ΔT′ i +T i , S′ i ←ΔS′ i ·S i

[0088] In the formula, R′ i represents the new rotation matrix of the i-th UAV, ΔR′ i represents the change of the rotation matrix of the i-th UAV, R i Represents the initial rotation matrix of the i-th UAV, T′ i represents the new translation matrix of the i-th UAV, ΔT′ i represents the change of the translation matrix of the i-th UAV, Ti represents the initial translation matrix of the i-th UAV, S′ i represents the scaling matrix of the i-th UAV, ΔS′ i represents the change of the scaling matrix of the i-th UAV, S i Represents the initial scaling matrix of the i-th drone.

[0089] Step S2033, obtaining the world coordinates of the field of view image of each drone after the movement in the world plane based on the new rotation matrix, the new translation matrix, the scaling matrix and the world coordinates of the field of view image before the movement in the world plane;

[0090] It should be noted that the world coordinates of the field of view image of the drone before movement in the world plane, the new rotation matrix of the drone, the new translation matrix of the drone and the scaling matrix of the drone can be calculated. The calculation relationship is as follows:

[0091]

[0092] In the formula, Represents the world coordinates of the field of view image of drone i after movement in the world plane, R′ i represents the new rotation matrix of drone i, T′ i represents the new translation matrix of UAV i, S′ i represents the scaling matrix of UAV i, P w Represents the initial world coordinates of the field of view image of the drone i after the movement in the world plane. Using the above calculation relationship, the world coordinates of the field of view image of the drone after the movement in the world plane are calculated.

[0093] Step S2034: updating the panoramic view image based on the world coordinates of the view images of each drone in the target drone group after movement in the world plane.

[0094] It can be understood that the panoramic view image is dynamically updated by utilizing the world coordinates of the view images of each drone in the target drone group after movement in the world plane.

[0095] In this implementation, dynamic mapping of current pixel coordinates to world coordinates is implemented, and the panoramic field of view image can be updated accurately and in real time without continuously calculating the rotation matrix and the translation matrix using feature matching, which can significantly reduce the computational overhead.

[0096] Step S30, based on the load image area of ​​each drone, determining the processed image of each drone in the panoramic field of view image, wherein the processed image at least includes a non-overlapping field of view processed image and an overlapping field of view processed image;

[0097] It should be noted that after aligning the fields of view between multiple drones, it is necessary to reasonably distribute the workload (including target detection and tracking within the field of view of each drone) among the drones to improve the overall efficiency. In order to reduce the computational overhead, the reasoning task on the overlapping field of view should be performed only once, and the results should be shared with other drones, while the non-overlapping fields of view should be processed separately. Therefore, in this embodiment, the overlapping field of view area and the non-overlapping field of view area are respectively selected from the panoramic field of view image and assigned to each drone for target detection and tracking as the workload that each drone needs to complete, and the regional image assigned to each drone is the processed image. The processed image includes at least a non-overlapping field of view processing image and an overlapping field of view processing image, the non-overlapping field of view processing image is the image of the non-overlapping field of view area assigned to each drone, and the overlapping field of view processing image is the image of the overlapping field of view area assigned to each drone.

[0098] In addition, it should be noted that the conversion of the drone’s field of view from the original pixel plane to the world plane will inevitably distort objects, making it difficult to detect and track targets in the panoramic field of view image. Therefore, the panoramic field of view image is only used to guide the distribution of workloads. After determining the image that each drone needs to be allocated in the panoramic field of view image, it is mapped back to the pixel plane for reasoning.

[0099] It is understandable that the load image area, i.e., the area of ​​the processed image that the drone can bear, can be determined by the computing power of the drone during inference.

[0100] In a feasible implementation, before step S30, the method further includes: obtaining the inference computing power of each drone in the target drone group; and determining the load image area of ​​each drone in the target drone group based on the proportional relationship between the inference computing power and the load image area.

[0101] It should be noted that the reasoning power of a drone is the computing power of the drone when performing reasoning tasks. Assuming that the total number of drones is N, the total area of ​​the panoramic image in the world plane is S, and the reasoning power of the drone is C i , computing power C i Affected by factors such as the available computing resources on the drone, the flight time of each drone, and the network connection between drones, etc. Based on the inference computing power of the drone, the work tasks are divided among multiple drones to achieve better load balancing.

[0102] It can be understood that since the inference is performed in the original pixel plane, the payload image area of ​​the drone is determined in the pixel plane, and the payload image area of ​​the drone is usually proportional to its inference computing power, that is:

[0103]

[0104] In the formula, represents the load image area of ​​the ith UAV, C i represents the inference computing power of the i-th drone, and N represents the total number of drones in the target drone group. It should be understood that after obtaining the load image area of ​​each drone, it is converted into an area in the world plane so as to allocate a processing image for each drone in the panoramic view image.

[0105] Step S40: assigning the processed image to the corresponding UAV for target detection, and performing collaborative tracking based on the obtained detection results.

[0106] It should be noted that each drone performs target detection and tracking on the assigned processing images. Since the camera is located at high altitude, the targets are usually small in size and large in number, and the drone has limited power and computing resources, it is difficult to deploy high-computation models for inference. Therefore, it is necessary to minimize the computing overhead, reduce latency and improve accuracy.

[0107] In addition, it should be noted that a deep learning model can usually be used to perform target detection on processed images. The deep learning model generally uses a rectangular image as input for reasoning, but the processed image assigned to each drone may be an irregular shape in the pixel plane. Therefore, a rectilinear polygonalization algorithm can be used to cut the processed image in the pixel plane into rectangles, and then the Guillotine bin packing algorithm can be used to reassemble the cut area into a rectangle for detection, which can minimize the size of the rectangle and reduce the reasoning delay.

[0108] It is understandable that in the traditional multi-camera multi-target tracking framework, the task is usually divided into two different stages: tracking the target within a single camera and then associating these targets between different cameras. This embodiment uses the world plane coordinate system to directly match the target trajectory between the cameras of multiple drones, thereby avoiding the need to associate targets across drones.

[0109] In the specific implementation, the detection bounding box in the field of view of each drone can be projected from the pixel plane to the world plane, and then the detection results can be used in the world plane for target tracking. Since the movement of the target in the image taken by the drone at high altitude is relatively small, the feature matching component can be removed, and no feature matching is performed. Tracking is only based on the intersection of union (IoU) matching between bounding boxes, which can significantly reduce the computational load and improve the tracking speed.

[0110] Furthermore, in the process of target tracking, the problem of target occlusion usually occurs. Therefore, this embodiment introduces a multi-view target re-detection mechanism, which uses multiple perspectives of the drone cluster to further improve the accuracy while reducing latency.

[0111] In a feasible implementation, the confidence of the detection result is obtained, and based on the confidence of the detection result, a low-confidence result is determined, and a processed image corresponding to the low-confidence result is used as a low-confidence area image; the low-confidence area image is reallocated to other UAVs for target detection to obtain a new detection result that meets the confidence condition, and the new detection result is collaboratively tracked.

[0112] It should be noted that a low-confidence result is a detection result with a low confidence level, and correspondingly, a high-confidence result is a detection result with a high confidence level. A confidence threshold can be set to distinguish between low-confidence results and high-confidence results. If the confidence of the detection result is lower than the confidence threshold, the detection result is considered to be a low-confidence result. If the confidence of the detection result is greater than or equal to the confidence threshold, the detection result is considered to be a high-confidence result. The processed image corresponding to the low-confidence result is the low-confidence area image, that is, the processed image of the low-confidence result. Meeting the confidence condition means that the confidence of the detection result reaches a high-confidence level, that is, the new detection result needs to meet the high-confidence result.

[0113] It is understandable that an obscured target in the perspective of one drone may be visible from the perspective of another drone. This embodiment divides the detection results into two groups based on the confidence level: high confidence results and low confidence results. For low confidence area images corresponding to low confidence results, they can be assigned to other drones for re-detection until they are detected as high confidence or no additional drones are available.

[0114] In addition, for tracks that cannot be matched with any target in the overlapping field of view, they are usually not deleted immediately. All other drones whose fields of view overlap with the track can be used to detect the target again to avoid missed detection. If the track can be matched with a target detected by at least one drone, it can be retained, thereby reducing the possibility of ID switching (i.e., assigning multiple ID identifiers to the same target). If the track cannot be matched with the target of any other drone, it is deleted.

[0115] This embodiment provides a real-time video analysis method based on multi-drone collaboration, which determines the field of view image of each drone based on the real-time video data shot by each drone in the target drone group; uniformly maps the field of view image to the world plane to construct a panoramic field of view image; based on the load image area of ​​each drone, determines the processing image of each drone in the panoramic field of view image, and the processing image includes at least a non-overlapping field of view processing image and an overlapping field of view processing image; distributes the processed image to the corresponding drone for target detection, and performs collaborative tracking based on the obtained detection results. The field of view images of all drones in the drone group are mapped from the pixel plane to the world plane to create a panoramic field of view, so as to allocate workloads to drones on the panoramic field of view based on the field of view association between drones, and coordinate the shooting perspectives of multiple drones for target detection and tracking, thereby improving accuracy and reducing latency, and at the same time improving computing efficiency by sharing reasoning tasks and reasoning results between drones.

[0116] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 3 , step S30 may include steps S301 to S303:

[0117] Step S301, dividing the panoramic field of view image into a plurality of sub-field of view images;

[0118] It should be noted that the sub-view image is a sub-image obtained by segmenting the panoramic view image, and the size of the sub-image is usually the same, for example: 32×32, and there is no specific limitation on this. In this embodiment, the panoramic view image is segmented into multiple sub-images, and each sub-image is assigned to a corresponding drone for inference. If the panoramic view image is an irregular shape, it is enlarged to the smallest rectangle containing the panoramic view image, and the blank area is filled with zero pixels to obtain a new standard panoramic view image. Then, this standard panoramic view image is segmented into smaller blocks, each block is of size M×M, and the processed image assigned to each drone is divided in blocks.

[0119] Step S302, based on the non-overlapping field of view of each drone in the target drone group, respectively determine the non-overlapping field of view processed image of each drone in the multiple sub-field of view images;

[0120] It should be noted that the non-overlapping field of view of a drone refers to the area where each drone does not overlap with other drones, and the corresponding overlapping field of view refers to the area where the fields of view between drones overlap. The non-overlapping field of view and the overlapping field of view can be determined when constructing a panoramic field of view image.

[0121] It can be understood that, according to the non-overlapping field of view of the drone, a corresponding image can be found in multiple sub-field of view images as the non-overlapping field of view processed image of the drone, thereby determining the non-overlapping field of view processed image of each drone.

[0122] Step S303, based on the payload image area of ​​each drone and the pixel plane area of ​​the non-overlapping field of view processed image, respectively determine the pixel plane area of ​​the overlapping field of view processed image of each drone;

[0123] It should be noted that the pixel plane area of ​​the non-overlapping view processing image refers to the area of ​​the non-overlapping view processing image in the pixel plane, and the pixel plane area of ​​the overlapping view processing image refers to the area of ​​the overlapping view processing image in the pixel plane. The area of ​​the overlapping view processing image in the pixel plane can be calculated using the area of ​​the non-overlapping view processing image in the pixel plane and the area of ​​the non-overlapping view processing image, so that the area in the pixel plane can be converted into the area in the world plane, and the area of ​​the non-overlapping view processing image in the world plane and the area of ​​the overlapping view processing image in the world plane are obtained.

[0124] In a feasible implementation, step S303 includes: when the pixel plane area of ​​the non-overlapping field of view processed image of the UAV is greater than or equal to the load image area, determining that the pixel plane area of ​​the overlapping field of view processed image of the UAV is zero; when the pixel plane area of ​​the non-overlapping field of view processed image of the UAV is less than the load image area, determining that the pixel plane area of ​​the overlapping field of view processed image of the UAV is the difference between the area of ​​the load image and the area of ​​the non-overlapping field of view processed image.

[0125] It can be understood that, in the pixel plane, if the pixel plane area of ​​the non-overlapping field of view processed image is greater than or equal to the load image area, the pixel plane area of ​​the overlapping field of view processed image is zero, and if the pixel plane area of ​​the non-overlapping field of view processed image is less than the load image area, the pixel plane area of ​​the overlapping field of view processed image is the difference between the load image area and the non-overlapping field of view processed image area, which can be expressed by the following calculation relationship:

[0126]

[0127] In the formula, represents the pixel plane area of ​​the overlapping field of view processed image of the i-th UAV, represents the pixel plane area of ​​the non-overlapping field of view processed image of the i-th UAV, Represents the payload image area of ​​the i-th UAV.

[0128] Step S304, determining a sub-overlapping field of view image from the multiple sub-field of view images based on the overlapping field of view range of the target drone group;

[0129] It can be understood that, according to the overlapping field of view range, a corresponding image, that is, a sub-overlapping field of view image, can be found in a plurality of sub-field of view images.

[0130] Step S305, converting the pixel plane area of ​​the overlapping view processing images of each UAV into the world plane area of ​​the overlapping view processing images of each UAV, and determining the overlapping view processing images of each UAV in the sub-overlapping view images based on the world plane area of ​​the overlapping view processing images of each UAV.

[0131] It should be noted that the pixel plane area of ​​the overlapping view processing image also needs to be converted into the area in the world plane, that is, the world plane area of ​​the overlapping view processing image, which can be calculated using the following formula:

[0132]

[0133] In the formula, represents the world plane area of ​​the overlapping field of view processed images of the i-th drone, represents the pixel plane area of ​​the overlapping field of view processed image of the i-th UAV, α i Represents the mapping coefficient of the i-th UAV. Mapping coefficient α i It can be obtained by dividing the overlapping field of view area of ​​drone i in the world plane by the field of view area in the pixel plane.

[0134] It is understandable that, given that the total area s satisfies The sub-overlapping view images can be further segmented according to the calculated world plane area of ​​the overlapping view processing image to obtain the overlapping view processing images of each UAV. During the segmentation process, the overlapping view processing images are still allocated in blocks.

[0135] It should be understood that after determining the overlapping field of view processing images of each drone, a heuristic overlapping field of view segmentation algorithm (OFP) can be used to specify the exact area assigned to each drone. Figure 4, for each drone i, a demand value is introduced and initialized as This requirement value is used to represent the overlap area that has been allocated (initially zero) and the expected overlap area. The difference between them. First, the drone i with the highest demand is selected as the predator (line 2 of the algorithm), that is, the drone waiting for block allocation; all unallocated views of the predator are considered defenders (line 4), that is, blocks that can be used for allocation; in each round of iteration, the predator is assigned a block with the lowest overlap level (line 6) and adjacent to its allocated block (line 8), where the overlap level is defined as the number of drones to which an overlapping block belongs (for example, for three drones, the maximum overlap level is three); after allocating a block, continue to allocate it to the next drone with the highest demand value, and so on.

[0136] During the segmentation of panoramic images, an object may be segmented into multiple sub-images. Since the size of objects in the drone's field of view is usually small, the sub-view images can be enlarged by a fixed ratio to ensure that objects on the segmentation boundary can be correctly detected. In addition, the allocation strategy for processing images is only valid for specific panoramic images and computing power ratios, so a new allocation strategy needs to be generated for each scheduling. Due to the relatively high efficiency of PFWA, the computational overhead is negligible.

[0137] In addition, a central device is usually required for management and coordination when allocating processed images. In this embodiment, a master drone can be designated to construct a panoramic view image and generate an allocation strategy for processing images. If a drone completes reasoning, its reasoning results will be sent to the master drone for aggregation and shared with other drones. Instead of using a fixed master drone, the master drone can be dynamically determined based on the current actual situation. For example, the drone that completes reasoning first will become the master drone for the next round. By distributing workloads on idle drones, lower latency and higher throughput can be achieved.

[0138] This embodiment provides a real-time video analysis method based on multi-drone collaboration, which maps the field of view images of all drones in a drone group from the pixel plane to the world plane to create a panoramic field of view, thereby distributing workloads to drones on the panoramic field of view based on the field of view association between drones, and coordinating the shooting perspectives of multiple drones to perform target detection and tracking, thereby improving accuracy and reducing latency. At the same time, by sharing reasoning tasks and reasoning results among drones, computing efficiency is improved.

[0139] For example, in order to help understand the implementation process of the real-time video analysis method based on multi-UAV collaboration obtained by combining this embodiment with the above-mentioned embodiment 2, please refer to Figure 5 , Figure 5A schematic diagram of the overall architecture of a real-time video analysis method based on multi-UAV collaboration is provided, specifically:

[0140] The overall architecture mainly consists of three parts: dynamic field of view alignment and update, workload distribution based on PFoV (panoramic field of view), and multi-UAV collaborative tracking. First, the field of view image from the drone is converted from the pixel plane to a unified world plane. Then, the field of view (FoV) of each drone on the world plane is used to create a PFoV, and the inference task load is distributed on the panoramic field of view. Finally, by coordinating the shooting perspectives of multiple drones, target detection and tracking are performed in the image area, thereby improving accuracy and reducing latency.

[0141] Dynamic field of view alignment and update: The conversion of images from the pixel plane to the unified world plane depends on the rotation matrix and the translation matrix. In order to avoid frequent feature matching due to the movement of the drone, a lightweight multi-drone field of view alignment algorithm (MDFA) is adopted. MDFA decomposes the motion of the drone into horizontal and vertical components. In the horizontal direction, the motion of the drone is represented by the new rotation matrix and the new translation matrix estimated by the optical flow method. In the vertical direction, the motion of the drone is represented by the scaling matrix estimated by scaling. Using these mapping matrices, the field of view of the drone's original pixel plane is mapped to the world plane, and then merged to create PFoV.

[0142] PFoV-based workload distribution: After mapping the fields of view of multiple drones to the same plane, PFoV can easily capture the field of view correlation between drones. In order to balance the load of inference work in a drone cluster, a fine-grained workload distribution strategy (PFWA) based on PFoV is adopted. According to the available computing resources, each drone will be assigned a field of view of non-overlapping areas and a field of view of part of the overlapping area. In order to allocate the field of view of the overlapping area to each drone, a heuristic overlapping field of view segmentation algorithm (OFP) is adopted. The allocation result of OFP (i.e., the partially overlapping area) and the non-overlapping area will be mapped to the original pixel plane to obtain a sub-image on the pixel plane, which will be used for reasoning.

[0143] Multi-UAV Cooperative Tracking: To further reduce computational overhead, we slice and reassemble sub-images on the pixel plane for detection. Low-confidence detection areas of a certain UAV will be re-detected with the assistance of other UAVs, and the repaired detection results will be mapped to PFoV for tracking on the world plane. In addition, if a target disappears from the field of view of one UAV, the field of view of other UAVs will be used for tracking to avoid target occlusion.

[0144] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the real-time video analysis method based on multi-UAV collaboration of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0145] This application also provides a real-time video analysis device based on multi-UAV collaboration, please refer to Figure 6 , the real-time video analysis device based on multi-UAV collaboration includes:

[0146] The field of view alignment module 10 is used to determine the field of view image of each drone based on the real-time video data shot by each drone in the target drone group.

[0147] The field of view alignment module 10 is further used to uniformly map the field of view images to a world plane to construct a panoramic field of view image.

[0148] The load distribution module 20 is used to determine the processed images of each UAV in the panoramic field of view image based on the load image area of ​​each UAV, and the processed images at least include a non-overlapping field of view processed image and an overlapping field of view processed image.

[0149] The collaborative tracking module 30 is used to distribute the processed image to the corresponding UAV for target detection, and perform collaborative tracking based on the obtained detection results.

[0150] In a feasible implementation manner, the field of view alignment module 10 is further used to obtain the field of view image of each drone in the target drone group before the movement, and based on the rotation matrix and the translation matrix obtained by feature matching, map the pixel coordinates of the field of view image before the movement in the pixel plane to the world plane to obtain the world coordinates of the field of view image before the movement in the world plane;

[0151] Generate a panoramic view image based on the world coordinates of the view image of each drone in the target drone group before movement in the world plane;

[0152] After each drone in the target drone group moves, the panoramic field of view image is dynamically updated.

[0153] In a feasible implementation manner, the field of view alignment module 10 is further used to obtain a field of view image of each drone after the movement of each drone in the target drone group, and map the pixel coordinates of the field of view image after the movement in the pixel plane to the world plane based on the rotation matrix and the translation matrix to obtain the initial world coordinates of the field of view image after the movement in the world plane;

[0154] Based on the initial world coordinates of the field of view image after the movement in the world plane, determining a scaling matrix in the vertical direction and a new rotation matrix and a new translation matrix in the horizontal direction for each drone;

[0155] Based on the new rotation matrix, the new translation matrix, the scaling matrix and the world coordinates of the field of view image before the movement in the world plane, obtaining the world coordinates of the field of view image of each drone after the movement in the world plane;

[0156] The panoramic view image is updated based on the world coordinates of the field of view images of each drone in the target drone group after movement in the world plane.

[0157] In a feasible implementation manner, the load distribution module 20 is further used to obtain the inference computing power of each drone in the target drone group;

[0158] Based on the proportional relationship between the inference computing power and the load image area, the load image area of ​​each drone in the target drone group is determined.

[0159] In a feasible implementation manner, the load distribution module 20 is further configured to divide the panoramic field of view image into a plurality of sub-field of view images;

[0160] Based on the non-overlapping field of view of each drone in the target drone group, respectively determine the non-overlapping field of view processed image of each drone in the multiple sub-field of view images;

[0161] Based on the payload image area of ​​each UAV and the pixel plane area of ​​the non-overlapping field of view processed image, the pixel plane area of ​​the overlapping field of view processed image of each UAV is determined respectively;

[0162] Based on the overlapping field of view of the target drone group, determining a sub-overlapping field of view image from the multiple sub-field of view images;

[0163] The pixel plane area of ​​the overlapping view processing image of each drone is converted into the world plane area of ​​the overlapping view processing image of each drone, and based on the world plane area of ​​the overlapping view processing image of each drone, the overlapping view processing image of each drone is respectively determined in the sub-overlapping view image.

[0164] In a feasible implementation manner, the load distribution module 20 is further configured to determine that the pixel plane area of ​​the overlapping view processing image of the drone is zero when the pixel plane area of ​​the non-overlapping view processing image of the drone is greater than or equal to the load image area;

[0165] When the pixel plane area of ​​the non-overlapping field of view processed image of the drone is smaller than the load image area, the pixel plane area of ​​the overlapping field of view processed image of the drone is determined as the difference between the load image area and the non-overlapping field of view processed image area.

[0166] In a feasible implementation manner, the collaborative tracking module 30 is further used to obtain the confidence of the detection result, determine a low-confidence result based on the confidence of the detection result, and use the processed image corresponding to the low-confidence result as a low-confidence area image;

[0167] The low confidence area image is reallocated to other UAVs for target detection to obtain new detection results that meet the confidence conditions, and the new detection results are collaboratively tracked.

[0168] The real-time video analysis device based on multi-UAV collaboration provided by the present application adopts the real-time video analysis method based on multi-UAV collaboration in the above embodiment, which can solve the technical problem of independently processing the video data on each UAV and being difficult to accurately detect and track the target in the scene. Compared with the prior art, the beneficial effects of the real-time video analysis device based on multi-UAV collaboration provided by the present application are the same as the beneficial effects of the real-time video analysis method based on multi-UAV collaboration provided by the above embodiment, and the other technical features in the real-time video analysis device based on multi-UAV collaboration are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0169] The present application provides a real-time video analysis device based on multi-UAV collaboration, and the real-time video analysis device based on multi-UAV collaboration includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time video analysis method based on multi-UAV collaboration in the above-mentioned embodiment one.

[0170] Reference below Figure 7 , which shows a schematic diagram of the structure of a real-time video analysis device based on multi-UAV collaboration suitable for implementing the embodiment of the present application. The real-time video analysis device based on multi-UAV collaboration in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The real-time video analysis device based on multi-UAV collaboration is shown as an example only and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0171] like Figure 7 As shown, the real-time video analysis device based on multi-UAV collaboration may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the real-time video analysis device based on multi-UAV collaboration are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the real-time video analysis device based on multi-drone collaboration to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a real-time video analysis device based on multi-drone collaboration with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0172] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0173] The real-time video analysis device based on multi-UAV collaboration provided by the present application adopts the real-time video analysis method based on multi-UAV collaboration in the above embodiment, which can solve the technical problem of independently processing the video data on each UAV and being difficult to accurately detect and track the target in the scene. Compared with the prior art, the beneficial effects of the real-time video analysis device based on multi-UAV collaboration provided by the present application are the same as the beneficial effects of the real-time video analysis method based on multi-UAV collaboration provided by the above embodiment, and the other technical features of the real-time video analysis device based on multi-UAV collaboration are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0174] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0175] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0176] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the real-time video analysis method based on multi-UAV collaboration in the above-mentioned embodiment.

[0177] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0178] The above-mentioned computer-readable storage medium may be included in the real-time video analysis device based on multi-UAV collaboration; or it may exist independently without being assembled into the real-time video analysis device based on multi-UAV collaboration.

[0179] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the real-time video analysis device based on multi-UAV collaboration, the real-time video analysis device based on multi-UAV collaboration: determines the field of view image of each UAV based on the real-time video data shot by each UAV in the target UAV group; uniformly maps the field of view image to the world plane to construct a panoramic field of view image; based on the load image area of ​​each UAV, determines the processed image of each UAV in the panoramic field of view image, the processed image includes at least a non-overlapping field of view processed image and an overlapping field of view processed image; distributes the processed image to the corresponding UAV for target detection, and performs collaborative tracking based on the obtained detection results.

[0180] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0183] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned real-time video analysis method based on multi-UAV collaboration, and can solve the technical problem of independently processing the video data on each UAV and being difficult to accurately detect and track targets in the scene. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the real-time video analysis method based on multi-UAV collaboration provided by the above-mentioned embodiment, and will not be elaborated here.

[0184] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned real-time video analysis method based on multi-UAV collaboration.

[0185] The computer program product provided by the present application can solve the technical problem that it is difficult to accurately detect and track targets in the scene by independently processing the video data on each drone. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the real-time video analysis method based on multi-drone collaboration provided by the above embodiment, and will not be elaborated here.

[0186] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A real-time video analysis method based on multi-UAV collaboration, characterized in that: The method comprises: Based on the real-time video data captured by each drone in the target drone group, determine the field of view image of each drone; Mapping the field of view images uniformly to a world plane to construct a panoramic field of view image; Based on the load image area of ​​each UAV, respectively determine the processed image of each UAV in the panoramic field of view image, wherein the processed image includes at least a non-overlapping field of view processed image and an overlapping field of view processed image; The processed images are distributed to corresponding drones for target detection, and collaborative tracking is performed based on the obtained detection results.

2. The method according to claim 1, characterized in that The step of uniformly mapping the field of view images to the world plane to construct a panoramic field of view image comprises: Acquire a field of view image of each drone in the target drone group before movement, and map the pixel coordinates of the field of view image before movement in the pixel plane to the world plane based on the rotation matrix and the translation matrix obtained by feature matching to obtain the world coordinates of the field of view image before movement in the world plane; Generate a panoramic view image based on the world coordinates of the view image of each drone in the target drone group before movement in the world plane; After each drone in the target drone group moves, the panoramic field of view image is dynamically updated.

3. The method according to claim 2, characterized in that After each drone in the target drone group moves, the step of dynamically updating the panoramic view image comprises: After each drone in the target drone group moves, a field of view image of each drone after the movement is obtained, and based on the rotation matrix and the translation matrix, pixel coordinates of the field of view image after the movement in the pixel plane are mapped to the world plane to obtain initial world coordinates of the field of view image after the movement in the world plane; Based on the initial world coordinates of the field of view image after the movement in the world plane, determining a scaling matrix in the vertical direction and a new rotation matrix and a new translation matrix in the horizontal direction for each drone; Based on the new rotation matrix, the new translation matrix, the scaling matrix and the world coordinates of the field of view image before the movement in the world plane, obtaining the world coordinates of the field of view image of each drone after the movement in the world plane; The panoramic view image is updated based on the world coordinates of the field of view images of each drone in the target drone group after movement in the world plane.

4. The method according to claim 1, characterized in that Before the step of respectively determining the processed images of each drone in the panoramic field of view image based on the load image area of ​​each drone, the step further includes: Obtaining the inference computing power of each drone in the target drone group; Based on the proportional relationship between the inference computing power and the load image area, the load image area of ​​each drone in the target drone group is determined.

5. The method according to claim 1, characterized in that The step of determining the processed images of each drone in the panoramic field of view image based on the load image area of ​​each drone includes: Dividing the panoramic field of view image into a plurality of sub-field of view images; Based on the non-overlapping field of view of each drone in the target drone group, respectively determine the non-overlapping field of view processed image of each drone in the multiple sub-field of view images; Based on the payload image area of ​​each UAV and the pixel plane area of ​​the non-overlapping field of view processed image, the pixel plane area of ​​the overlapping field of view processed image of each UAV is determined respectively; Based on the overlapping field of view of the target drone group, determining a sub-overlapping field of view image from the multiple sub-field of view images; The pixel plane area of ​​the overlapping view processing image of each drone is converted into the world plane area of ​​the overlapping view processing image of each drone, and based on the world plane area of ​​the overlapping view processing image of each drone, the overlapping view processing image of each drone is respectively determined in the sub-overlapping view image.

6. The method according to claim 5, characterized in that The step of determining the pixel plane area of ​​the overlapping view processing image of each drone based on the payload image area of ​​each drone and the pixel plane area of ​​the non-overlapping view processing image comprises: When the pixel plane area of ​​the non-overlapping field of view processed image of the drone is greater than or equal to the payload image area, determining that the pixel plane area of ​​the overlapping field of view processed image of the drone is zero; When the pixel plane area of ​​the non-overlapping field of view processed image of the drone is smaller than the load image area, the pixel plane area of ​​the overlapping field of view processed image of the drone is determined as the difference between the load image area and the non-overlapping field of view processed image area.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Acquiring the confidence of the detection result, determining a low-confidence result based on the confidence of the detection result, and using a processed image corresponding to the low-confidence result as a low-confidence region image; The low confidence area image is reallocated to other UAVs for target detection to obtain new detection results that meet the confidence conditions, and the new detection results are collaboratively tracked.

8. A real-time video analysis device based on multi-UAV collaboration, characterized in that: The device comprises: The field of view alignment module is used to determine the field of view image of each drone based on the real-time video data shot by each drone in the target drone group; The field of view alignment module is further used to uniformly map the field of view images to a world plane to construct a panoramic field of view image; A load distribution module, configured to determine, based on the load image area of ​​each UAV, a processed image of each UAV in the panoramic field of view image, wherein the processed image includes at least a non-overlapping field of view processed image and an overlapping field of view processed image; The collaborative tracking module is used to distribute the processed image to the corresponding UAV for target detection and perform collaborative tracking based on the obtained detection results.

9. A real-time video analysis device based on multi-UAV collaboration, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the real-time video analysis method based on multi-UAV collaboration as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the real-time video analysis method based on multi-UAV collaboration as described in any one of claims 1 to 7 are implemented.

Images