Multi-unmanned aerial vehicle cooperative target recognition system

Abstract

The invention relates to a multi-unmanned aerial vehicle cooperative target recognition system, belongs to the technical field of unmanned aerial vehicle systems, and solves the problems that an existing unmanned aerial vehicle system is low in recognition accuracy, large in model calculation requirement and low in communication efficiency. The system comprises an unmanned aerial vehicle platform which comprises a plurality of unmanned aerial vehicles, and each unmanned aerial vehicle comprises a three-light pod, an airborne calculation processing module and an airborne communication module; the airborne calculation processing module is used for carrying out target detection and identification on the image based on an improved target detection model; the airborne communication module is used for communicating with a ground control station through an asynchronous message passing library ZeroMQ and sending an original bounding box corresponding to a target in the image and attribute information of the bounding box to the ground control station; and the ground control station converts each original bounding box, performs cyclic fusion processing on the converted bounding box to obtain a bounding box after cyclic fusion processing, and performs detection and discrimination on the target based on the bounding box after fusion processing.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle systems, and in particular to a multi-unmanned aerial vehicle collaborative target recognition system. Background Art

[0002] The rapid development of drone technology has promoted its widespread application in the field of aerial image target detection, such as agricultural monitoring, traffic monitoring, environmental assessment, etc. However, the challenges faced by drone target detection systems are becoming increasingly prominent. Due to its fixed shooting angle and limited field of view, a single drone system is difficult to fully capture target features, especially in complex environments such as urban canyons and forests, where the detection performance of small targets is limited.

[0003] Factors such as the scale variation of the target, background complexity, occlusion and illumination variation further increase the difficulty of target detection. In practical applications, these factors often lead to the lack of significant target features in the images taken by drones, and traditional target detection models are difficult to accurately identify and locate targets. At the same time, the limited onboard computing resources and energy supply of drones restrict their ability to run target detection models with high computing requirements. The target detection model needs to be improved to adapt to the computing power of drones. At the same time, traditional communication methods are difficult to meet the needs of efficient and real-time data transmission between drones and ground control stations. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a multi-UAV collaborative target recognition system to solve the problems of low recognition accuracy, large model computing resource requirements, and low communication efficiency of existing UAV systems.

[0005] An embodiment of the present invention provides a multi-UAV cooperative target recognition system, comprising:

[0006] UAV platforms and ground control stations;

[0007] The UAV platform includes multiple UAVs, each of which includes a three-light pod, an airborne computing and processing module, and an airborne communication module;

[0008] The three-light pod of each drone is used to obtain images of the same target;

[0009] The onboard computing and processing module is used to perform target detection and recognition on each image acquired by the three-light pod based on the improved target detection model, and obtain the original bounding box corresponding to the target in each image and the attribute information of the bounding box;

[0010] The airborne communication module is used to communicate with the ground control station through the asynchronous message transmission library ZeroMQ, and send the original bounding box corresponding to the target in the image and the attribute information of the bounding box to the ground control station;

[0011] The ground control station is used to convert each original bounding box, perform cyclic fusion processing on the converted bounding box to obtain a bounding box after cyclic fusion processing, and detect and identify the target based on the bounding box after fusion processing.

[0012] As a further improvement of the present application, the improved target detection model is an improved YoloV5s model; wherein the darknet-53 backbone network in the YoloV5s model is replaced with the MobileNet backbone network, downsampled, and the activation function of the MobileNet backbone network is replaced with the h-swish hard activation function; an SE module is added to the bottleneck layer of the MobileNet backbone network to obtain the backbone network of the improved target detection model.

[0013] As a further improvement of the present application, the backbone network of the improved target detection model includes an input layer, a depth-separable convolution layer, a bottleneck layer, an activation function layer, and an output layer connected in sequence; the bottleneck layer includes an inverse residual module and an SE module connected in sequence, and the output of the inverse residual module serves as the input of the SE module. As a further improvement of the present application, the ground control station includes:

[0014] A first conversion unit, configured to convert the original bounding box to obtain a real bounding box;

[0015] The second conversion unit converts all real bounding boxes into the same map to obtain bounding boxes of all images in the same map;

[0016] A fusion processing unit, used for performing fusion processing on all the converted bounding boxes based on the attribute information of the bounding boxes to obtain a bounding box after cyclic fusion processing;

[0017] The detection and discrimination module is used to detect and discriminate the target based on the bounding box after the loop fusion processing.

[0018] As a further improvement of the present application, the airborne communication module includes:

[0019] A data encapsulation unit is used to encapsulate the original bounding box corresponding to the target in the image and the attribute information of the bounding box into a structure, where the attribute information of the bounding box includes the center coordinates, width, height of the bounding box, and the target type, confidence and ID corresponding to the bounding box;

[0020] The protocol adapter unit is used to convert the structure into JSON format;

[0021] The information transmission unit is used to transmit the structure converted into JSON format to the ground control station through the asynchronous messaging library ZeroMQ.

[0022] As a further improvement of the present application, the communication mode of the asynchronous messaging library ZeroMQ includes:

[0023] Request-response REQ / REP mode, publish-subscribe PUB / SUB mode, and push-pull PUSH / PULL mode.

[0024] As a further improvement of the present application, all the converted bounding boxes are fused based on the attribute information of the bounding boxes, and the bounding boxes obtained after the cyclic fusion processing include:

[0025] S71, selecting the bounding box with the highest confidence as the reference box, and the remaining bounding boxes as detection boxes;

[0026] S72, traversing all detection frames, performing cyclic fusion processing on the detection frames through a preset target fusion algorithm based on attribute information of the detection frames and the reference frames, and identifying redundant detection frames and abnormal detection frames;

[0027] S73, merge each redundant detection frame identified with the reference frame to obtain a new bounding box, and obtain attribute information of the new bounding box based on the attribute information of the redundant detection frame and the reference frame, and delete the redundant detection frame and the abnormal detection frame to obtain a fused bounding box; return to step S71 until no redundant detection frame can be identified, and obtain a bounding box after the cyclic fusion process.

[0028] As a further improvement of the present application, based on the attribute information of the detection frame and the reference frame, the detection frame is subjected to a cyclic fusion process through a preset target fusion algorithm, and redundant detection frames and abnormal detection frames are identified, including:

[0029] Calculate the overlapping area of ​​the current detection frame and the reference frame;

[0030] If the overlapping area between the current detection frame and the reference frame is greater than or equal to the overlapping area threshold, the current detection frame is marked as a redundant detection frame;

[0031] If the overlapping area between the current detection frame and the reference frame is less than the overlapping area threshold, the target types of the current detection frame and the reference frame are compared; if the target types corresponding to the current detection frame and the reference frame are inconsistent, the detection frame is retained;

[0032] If the target type corresponding to the current detection frame is consistent with that of the reference frame, the distance between the center point of the current detection frame and the center point of the reference frame is calculated; if the distance between the center point of the current detection frame and the center point of the reference frame is less than or equal to the distance threshold, the current detection frame is marked as a redundant detection frame;

[0033] If the distance between the center point of the current detection frame and the center point of the reference frame is greater than the distance threshold, check whether the ID information of the current detection frame is consistent with the ID information of the reference frame; if the ID information of the current detection frame is consistent with the ID information of the reference frame, the detection frame is an abnormal detection frame, and the current detection frame is removed;

[0034] If the ID information of the current detection box is inconsistent with the ID information of the reference box, the detection box is retained.

[0035] As a further improvement of the present application, each drone includes an inertial measurement unit IMU, which is used to obtain real-time attitude information of the drone, and the attitude information includes the pitch angle, roll angle and heading angle of the drone. The airborne communication module is also used to send the attitude information of the drone to a ground station.

[0036] As a further improvement of the present application, all real bounding boxes are converted into the same map, and the bounding boxes of all images in the same map include:

[0037] Convert all generated real bounding boxes into the image space auxiliary coordinate system to generate image space auxiliary coordinate system coordinates of all real bounding boxes;

[0038] Convert all generated image space auxiliary coordinate system coordinates into body coordinate system coordinates;

[0039] Convert all generated body coordinate system coordinates into station center horizontal coordinate system coordinates;

[0040] The generated geocentric horizontal coordinate system coordinates of all stations are converted into the same map, which is a geocentric rectangular coordinate system.

[0041] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0042] 1. The drone platform of the multi-drone collaborative target recognition system includes multiple drones. Through the collaborative work of multiple drones, the same target is photographed from different angles, which improves the accuracy of target detection and reduces the impact of occlusion and perspective changes on the detection results.

[0043] 2. The onboard computing processing module replaces the darknet-53 backbone network in the YoloV5s model with the MobileNet backbone network, the standard convolution layer with a depth-separable convolution layer, and the global average pooling layer with a residual unit, which significantly reduces the amount of computation and the number of parameters of the model, making the model more suitable for running on drones while maintaining efficient target detection performance.

[0044] 3. The airborne communication module communicates with the ground control station through the asynchronous messaging library ZeroMQ, ensuring the reliability and real-time performance of data transmission and improving the overall response speed of the system and user experience.

[0045] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0047] Figure 1 A hardware structure diagram of a multi-UAV cooperative target recognition system provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a three-light pod provided in one embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a depth-wise separable convolutional layer provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of size conversion of an original bounding box provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of a body coordinate system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0053] A specific embodiment of the present invention discloses a multi-UAV cooperative target recognition system, such as Figure 1 shown.

[0054] UAV platforms and ground control stations;

[0055] The UAV platform includes multiple UAVs, each of which includes a three-light pod, an airborne computing and processing module, and an airborne communication module;

[0056] The three-light pod of each drone is used to obtain images of the same target;

[0057] The onboard computing and processing module is used to perform target detection and recognition on each image acquired by the three-light pod based on the improved target detection model, and obtain the original bounding box corresponding to the target in each image and the attribute information of the bounding box;

[0058] The airborne communication module is used to communicate with the ground control station through the asynchronous message transmission library ZeroMQ, and send the original bounding box corresponding to the target in the image and the attribute information of the bounding box to the ground control station;

[0059] The ground control station is used to convert each original bounding box, perform cyclic fusion processing on the converted bounding box to obtain a bounding box after cyclic fusion processing, and detect and identify the target based on the bounding box after fusion processing.

[0060] The UAV platform consists of multiple UAVs, each of which is equipped with a three-light pod, an airborne computing and processing module, and an airborne communication module. Figure 2 As shown. The three-light pod is a device that integrates visible light, infrared and laser radar, and is used to obtain target images from different spectrums. The onboard computing processing module is used to run the target detection model and perform target detection and recognition on the image, and the onboard communication module is used to exchange data with the ground control station.

[0061] When the drones in the drone platform are performing a mission, multiple drones are deployed to surround or cover the same target area from different angles, and three-light pods are used to collect images of the target from different positions and angles to enhance the ability to fully capture the target's features.

[0062] The onboard computing and processing module performs target detection and recognition on each image acquired by the three-light pod based on the improved target detection model. The improved target detection model is an improved YoloV5s model. The darknet-53 backbone network in the YoloV5s model is replaced by the MobileNet backbone network, and the activation function of the MobileNet backbone network is replaced by the h-swish hard activation function. The SE module is added to the bottleneck layer of the MobileNet backbone network to obtain the backbone network of the improved target detection model.

[0063] The YoloV5s model is a target detection model that consists of four parts, including the input layer, the backbone network (Backbone), the neck network (Neck), and the detection head (Head). The input layer is used to receive the original image data. The backbone network is used to extract features from the original image data received by the input layer. The neck network receives the feature map from the backbone network, fuses the feature maps of different scales, and enhances the detection ability of the model. The detection head of the YoloV5s model includes multiple scales, corresponding to feature maps of different resolutions, for detecting targets of different sizes. The detection head receives the feature map fused by the neck network and performs target detection.

[0064] The backbone network of the YoloV5s model is darknet-53, which extracts image features through multi-layer convolution operations. The darknet-53 network structure has a large number of parameters and computing requirements, which is not suitable for running on drones with limited computing resources. In contrast, the MobileNet network greatly reduces the number of parameters and computing load of the model, making it more suitable for mobile and embedded devices.

[0065] In some embodiments, the backbone network of the improved target detection model includes an input layer, a depth-separable convolutional layer, a bottleneck layer, an activation function layer and an output layer connected in sequence; the bottleneck layer includes an inverted residual module and a SE module connected in sequence, and the output of the inverted residual module serves as the input of the SE module.

[0066] The input layer is used to receive the raw image data and pass it to the depthwise separable convolutional layer.

[0067] like Figure 3 As shown in the figure, the depthwise separable convolution layer in the MobileNet backbone network decomposes the standard convolution into two steps: depthwise convolution and pointwise convolution. The depthwise convolution operation is performed independently on each input channel, and the pointwise convolution combines the output channels of the depthwise convolution, thereby reducing the amount of computation and model size.

[0068] The bottleneck layer includes an inverted residual block and an SE module. The output of the inverted residual block is used as the input of the SE module. The inverted residual block expands the number of channels through 1x1 convolution, then performs depthwise convolution, and finally reduces the number of channels through 1x1 convolution, reducing the number of parameters and computational complexity while maintaining the effectiveness of the model. The SE module (Squeeze-and-Excitation Module) dynamically adjusts channel weights by learning the relationship between channels to enhance the representation ability of the model and improve the model's perception of key features, thereby improving classification performance. The h-swish hard activation function is shown in the following calculation formula:

[0069]

[0070] Wherein, x is the output of the SE module, ReLU6(x)=min(max(0,x),6). In some embodiments, the airborne communication module includes:

[0071] A data encapsulation unit is used to encapsulate the original bounding box corresponding to the target in the image and the attribute information of the bounding box into a structure, where the attribute information of the bounding box includes the center coordinates, width, height of the bounding box, and the target type, confidence and ID corresponding to the bounding box;

[0072] The protocol adapter unit is used to convert the structure into JSON format;

[0073] The information transmission unit is used to transmit the structure converted into JSON format to the ground control station through the asynchronous messaging library ZeroMQ.

[0074] It should be noted that the original bounding box is the predicted target position on the image. The attribute information of the bounding box is detailed information related to each bounding box, including the center coordinates, width, height of the bounding box, and the target type, confidence, and ID corresponding to the bounding box. The target type is the category to which the detected target belongs. The confidence is the probability that the bounding box contains the target, usually a value between 0 and 1. The ID is the identifier of each bounding box, which is used to distinguish the bounding boxes in images taken by different drones. The IDs of all bounding boxes in images taken by the same drone are the same.

[0075] The data encapsulation unit packages the original bounding box corresponding to the target in the image and the attribute information of the bounding box into a structured data packet, i.e., a structure. For example, the center coordinates of the bounding box are encapsulated as a substructure containing x and y values, the width and height are encapsulated as integers or floating point numbers, the confidence is encapsulated as a floating point number between 0 and 1, and the ID can be a string or a number.

[0076] The protocol adapter unit converts the encapsulated structure data into JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format that is easy for machines to parse and generate.

[0077] Specifically, the protocol adapter unit maps each field in the structure to a key-value pair in the JSON format according to a predefined template or rule. For example, the center coordinates of the bounding box are converted to the "center" key in the JSON object, whose value is a sub-object containing "x" and "y".

[0078] Finally, the information transmission unit transmits the structure converted into JSON format to the ground control station through the asynchronous messaging library ZeroMQ. The asynchronous messaging library ZeroMQ is a high-performance asynchronous messaging library that supports multiple communication modes, including request-response REQ / REP mode, publish-subscribe PUB / SUB mode, and push-pull PUSH / PULL mode.

[0079] In the request-response REQ / REP mode, in the communication between the drone platform and the ground control station, the ground control station acts as the server to receive requests from the drone and returns corresponding responses based on the request content. This mode is suitable for scenarios where the ground station needs to send instructions to the drone or update parameter configurations, ensuring the timely communication and execution of instructions. In the publish-subscribe PUB / SUB mode, the ground control station is the subscriber. According to the subscription settings, the drone transmits the structure to the ground control station in real time or at preset time intervals. In the push-pull PUSH / PULL mode, the drone continuously collects image data and performs target detection. The ground control station pulls data when needed and receives the structure transmitted by the drone.

[0080] The ground control station is the ground-end processing center of the multi-UAV collaborative target recognition system. It receives the original bounding box and corresponding attribute information transmitted by the UAV platform, and detects and identifies the target after further processing of the original bounding box.

[0081] In some embodiments, the ground control station includes:

[0082] A first conversion unit, configured to convert the original bounding box to obtain a real bounding box;

[0083] The second conversion unit converts all real bounding boxes into the same map to obtain bounding boxes of all images in the same map;

[0084] A fusion processing unit, used for performing fusion processing on all the converted bounding boxes based on the attribute information of the bounding boxes to obtain a bounding box after cyclic fusion processing;

[0085] The detection and discrimination module is used to detect and discriminate the target based on the bounding box after the loop fusion processing.

[0086] like Figure 4 The original bounding box is transformed to obtain the real bounding box. The conversion relationship of the original bounding box for size conversion is shown in formula (1):

[0087]

[0088] Among them, μ is the pixel size of the original bounding box, f is the focal length of the three-light pod camera, D is the distance between the drone and the target, and d is the real bounding box size.

[0089] Convert all true bounding boxes to the same map, and get the bounding boxes of all images in the same map including:

[0090] Convert all generated real bounding boxes into the image space auxiliary coordinate system to generate image space auxiliary coordinate system coordinates of all real bounding boxes;

[0091] Convert all generated image space auxiliary coordinate system coordinates into body coordinate system coordinates;

[0092] Convert all generated body coordinate system coordinates into station center horizontal coordinate system coordinates;

[0093] The generated geocentric horizontal coordinate system coordinates of all stations are converted into the same map, which is a geocentric rectangular coordinate system.

[0094] The image space auxiliary coordinate system is a local coordinate system relative to the camera lens, which is used to describe the position of the target in the image. The origin of the image space auxiliary coordinate system is located at the optical center of the camera lens, that is, the projection center S. The image space auxiliary coordinate system and the body coordinate system can be defined as a left-handed system, with the X axis pointing to the top of the image, the Y axis pointing to the right of the image, and the Z axis pointing upward.

[0095] The body coordinate system is a coordinate system relative to the drone body, used to describe the position and direction of the camera on the drone. The origin of the body coordinate system is located at the center of mass of the drone, the X-axis points to the nose, the Y-axis points to the right side of the drone plane, and the Z-axis is perpendicular to the drone and upward. Figure 5 shown.

[0096] The station-centered horizontal coordinate system is a local coordinate system relative to the current position of the drone. The origin of the station-centered horizontal coordinate system is the current position of the drone, and it is used to describe the geographical location of the drone's surroundings. The station-centered horizontal coordinate system is defined as a left-handed system, with the X-axis pointing in the direction of the meridian, with the north being positive; the Z-axis pointing in the direction of the ellipsoid normal through the origin, with the zenith being positive; and the Y-axis pointing perpendicular to the X- and Z-axis planes, with the east being positive.

[0097] The geocentric rectangular coordinate system is a globally unified coordinate system with the center of mass of the earth as the origin, which is used to describe the precise position of any location on the earth. The geocentric rectangular coordinate system is defined as a right-hand system, with the center of the reference ellipsoid as the origin, the intersection of the starting meridian plane and the equatorial plane as the X-axis, the direction orthogonal to the X-axis on the equatorial plane as the Y-axis, and the rotation axis of the ellipsoid as the Z-axis.

[0098] Specifically, the image space auxiliary coordinate system coordinates of the real bounding box are:

[0099]

[0100] The calculation formula for converting the image space auxiliary coordinate system coordinates into the body coordinate system coordinates is:

[0101]

[0102] Among them, X1, Y1, Z1 are the coordinates of the auxiliary coordinate system of the image space; Dis is the distance from the pod to the target; X2, Y2, Z2 are the coordinates of the auxiliary coordinate system of the target's body; X1, Y1, Z1 are the coordinates of the auxiliary coordinate system of the target's image space; X0, Y0, Z0 are the coordinates of the optoelectronic pod in the body coordinate system; K is the positive rotation matrix of the pod placement angle; A is the rotation matrix.

[0103] The positive rotation matrix K of the pod placement angle is the rotation matrix that describes the fixed position and orientation of the drone pod relative to the drone body. The rotation matrix A is a universal rotation matrix that is used to describe the rotation transformation from one coordinate system to another in three-dimensional space.

[0104] Specifically, the calculation formula of the pod placement angle positive rotation matrix K is:

[0105]

[0106] In the formula, y is the heading angle of the pod, and the right deviation is positive; p is the pitch angle of the pod, the head is positive when it is raised, and negative when it is lowered; r is the roll angle of the pod, and the right side is positive when it is raised.

[0107] The calculation formula of rotation matrix A is:

[0108]

[0109] In the formula, Yaw is the azimuth angle of the pod relative to the nose of the drone, with right deviation being positive; P is the pitch angle of the pod relative to the aircraft, with head up being positive and head down being negative. R is the roll angle of the pod relative to the drone, and K is the rotation matrix for correcting the pod placement angle.

[0110] Each drone includes an inertial measurement unit (IMU) to obtain the drone's real-time attitude information, including pitch angle, roll angle, and heading angle. The airborne communication module sends the drone's attitude information to the ground control station. The pitch angle is the tilt of the drone relative to the horizontal plane, the roll angle is the rotation of the drone body around the forward direction, and the heading angle is the rotation of the drone relative to the geographic north direction.

[0111] The calculation formula for converting the body coordinate system coordinates into the station center horizontal coordinate system coordinates is:

[0112]

[0113] Among them, X3, Y3, and Z3 are the coordinates of the target station's center-horizontal coordinate system; X2, Y2, and Z2 are the coordinates of the target's body coordinate system; θ is the UAV's heading angle, ψ is the UAV's pitch angle, and φ is the UAV's roll angle.

[0114] The calculation formula for converting the station center geocentric coordinate system coordinates into the geocentric rectangular coordinate system coordinates is:

[0115]

[0116] Among them, X4, Y4, and Z4 are the coordinates of the target in the geocentric rectangular coordinate system; X3, Y3, and Z3 are the coordinates of the target in the station-centered horizontal coordinate system; B is the latitude of the drone, and L is the longitude of the drone; XA, YA, and ZA are the coordinates of the drone in the geocentric rectangular coordinate system.

[0117] All the transformed bounding boxes are fused based on the attribute information of the bounding boxes, and the bounding boxes obtained after the cyclic fusion processing include:

[0118] S71, selecting the bounding box with the highest confidence as the reference box, and the remaining bounding boxes as detection boxes;

[0119] S72, traversing all detection frames, performing cyclic fusion processing on the detection frames through a preset target fusion algorithm based on attribute information of the detection frames and the reference frames, and identifying redundant detection frames and abnormal detection frames;

[0120] S73, merge each redundant detection frame identified with the reference frame to obtain a new bounding box, and obtain attribute information of the new bounding box based on the attribute information of the redundant detection frame and the reference frame, and delete the redundant detection frame and the abnormal detection frame to obtain a fused bounding box; return to step S71 until no redundant detection frame can be identified, and obtain a bounding box after the cyclic fusion process.

[0121] When the redundant detection frame and the reference frame are merged, the center point coordinates of the new bounding box are weighted according to the confidence of the redundant detection frame and the reference frame, and the center point coordinates are the weighted average of the redundant detection frame and the reference frame. The width and height are the width and height of the minimum bounding rectangle that contains the width and height of the redundant detection frame and the reference frame, respectively, to ensure that the new bounding box can cover the target area corresponding to the redundant detection frame and the reference frame.

[0122] The confidence of the new bounding box can be obtained by weighted average method or Soft NMS algorithm. The Soft NMS algorithm assigns weights to the redundant detection box and the reference box by calculating the intersection of the redundant detection box and the reference box, and generates the confidence of the new bounding box. The ID and attribute information of the new bounding box are the same as those of the original redundant detection box.

[0123] Furthermore, based on the attribute information of the detection frame and the reference frame, the detection frame is subjected to a cyclic fusion process through a preset target fusion algorithm, and redundant detection frames and abnormal detection frames are identified, including:

[0124] Calculate the overlapping area of ​​the current detection frame and the reference frame;

[0125] If the overlapping area between the current detection frame and the reference frame is greater than or equal to the overlapping area threshold, the current detection frame is marked as a redundant detection frame;

[0126] If the overlapping area between the current detection frame and the reference frame is less than the overlapping area threshold, the target types of the current detection frame and the reference frame are compared; if the target types corresponding to the current detection frame and the reference frame are inconsistent, the detection frame is retained;

[0127] If the target type corresponding to the current detection frame is consistent with that of the reference frame, the distance between the center point of the current detection frame and the center point of the reference frame is calculated; if the distance between the center point of the current detection frame and the center point of the reference frame is less than or equal to the distance threshold, the current detection frame is marked as a redundant detection frame;

[0128] If the distance between the center point of the current detection frame and the center point of the reference frame is greater than the distance threshold, check whether the ID information of the current detection frame is consistent with the ID information of the reference frame; if the ID information of the current detection frame is consistent with the ID information of the reference frame, the detection frame is an abnormal detection frame, and the current detection frame is removed;

[0129] If the ID information of the current detection box is inconsistent with the ID information of the reference box, the detection box is retained.

[0130] The overlapping area of ​​the current detection frame and the reference frame is the overlapping area of ​​the detection frame and the reference frame. Calculating the overlapping area of ​​the detection frame and the reference frame includes:

[0131] Obtain the coordinates of all detection frames and reference frames in the geocentric rectangular coordinate system; calculate the coordinates of the intersection area of ​​each detection frame and reference frame; calculate the area of ​​the intersection area based on the coordinates of the intersection area, and then obtain the overlapping area.

[0132] The overlap area threshold can be set based on experience or determined by analyzing data in a specific application scenario. For example, the overlap area threshold can be set to 0.5, which means that if the overlap area between the detection box and the reference box reaches or exceeds 50%, the current detection box is marked as a redundant detection box. The overlap area threshold can be adaptively adjusted according to the complexity of the scene and the size of the target. In scenes with dense targets, the overlap area threshold needs to be lowered to reduce misjudgment; in scenes with sparse targets, the overlap area threshold can be increased to improve the accuracy of the de-redundancy process.

[0133] The distance between the center point of the current detection frame and the center point of the reference frame can be calculated using the Euclidean distance formula. The center point is the geometric center of the detection frame and the reference frame. The distance threshold can be set based on the size of the target and the specific conditions of the scene.

[0134] Compared with the prior art, the multi-UAV collaborative target recognition system provided in this embodiment improves the accuracy of target detection and reduces the impact of occlusion and perspective changes on the detection results by using multiple UAVs to work together to shoot the same target from different angles. The improved YOLOv5s model is used to significantly reduce the amount of calculation and the number of parameters of the model, making the model more suitable for running on UAVs while maintaining efficient target detection performance. The asynchronous messaging library ZeroMQ is used to achieve efficient communication between the UAV and the ground control station, ensuring the reliability and real-time performance of data transmission and improving the overall response speed of the system and user experience.

[0135] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-UAV cooperative target recognition system, characterized in that: include: UAV platforms and ground control stations; The UAV platform includes multiple UAVs, each of which includes a three-light pod, an airborne computing and processing module, and an airborne communication module; The three-light pod of each drone is used to obtain images of the same target; The onboard computing and processing module is used to perform target detection and recognition on each image acquired by the three-light pod based on the improved target detection model, and obtain the original bounding box corresponding to the target in each image and the attribute information of the bounding box; The airborne communication module is used to communicate with the ground control station through the asynchronous message transmission library ZeroMQ, and send the original bounding box corresponding to the target in the image and the attribute information of the bounding box to the ground control station; The ground control station is used to convert each original bounding box, perform cyclic fusion processing on the converted bounding box to obtain a bounding box after cyclic fusion processing, and detect and identify the target based on the bounding box after fusion processing.

2. The system according to claim 1, characterized in that The improved target detection model is an improved YoloV5s model; wherein the darknet-53 backbone network in the YoloV5s model is replaced with the MobileNet backbone network, downsampled, and the activation function of the MobileNet backbone network is replaced with the h-swish hard activation function; an SE module is added to the bottleneck layer of the MobileNet backbone network to obtain the backbone network of the improved target detection model.

3. The system according to claim 2, characterized in that The backbone network of the improved target detection model includes an input layer, a depth-separable convolutional layer, a bottleneck layer, an activation function layer and an output layer connected in sequence; the bottleneck layer includes an inverted residual module and an SE module connected in sequence, and the output of the inverted residual module serves as the input of the SE module.

4. The system according to claim 1, characterized in that The ground control station comprises: A first conversion unit, configured to convert the original bounding box to obtain a real bounding box; The second conversion unit converts all real bounding boxes into the same map to obtain bounding boxes of all images in the same map; A fusion processing unit, used for performing fusion processing on all the converted bounding boxes based on the attribute information of the bounding boxes to obtain a bounding box after cyclic fusion processing; The detection and discrimination module is used to detect and discriminate the target based on the bounding box after the loop fusion processing.

5. The system according to claim 1, characterized in that The onboard communication module comprises: A data encapsulation unit is used to encapsulate the original bounding box corresponding to the target in the image and the attribute information of the bounding box into a structure, where the attribute information of the bounding box includes the center coordinates, width, height of the bounding box, and the target type, confidence and ID corresponding to the bounding box; The protocol adapter unit is used to convert the structure into JSON format; The information transmission unit is used to transmit the structure converted into JSON format to the ground control station through the asynchronous messaging library ZeroMQ.

6. The system according to claim 5, characterized in that The communication mode of the asynchronous messaging library ZeroMQ includes: Request-response REQ / REP mode, publish-subscribe PUB / SUB mode, and push-pull PUSH / PULL mode.

7. The system according to claim 4, characterized in that All the transformed bounding boxes are fused based on the attribute information of the bounding boxes, and the bounding boxes obtained after the cyclic fusion processing include: S71, selecting the bounding box with the highest confidence as the reference box, and the remaining bounding boxes as detection boxes; S72, traversing all detection frames, performing cyclic fusion processing on the detection frames through a preset target fusion algorithm based on attribute information of the detection frames and the reference frames, and identifying redundant detection frames and abnormal detection frames; S73, merge each redundant detection frame identified with the reference frame to obtain a new bounding box, and obtain attribute information of the new bounding box based on the attribute information of the redundant detection frame and the reference frame, and delete the redundant detection frame and the abnormal detection frame to obtain a fused bounding box; return to step S71 until no redundant detection frame can be identified, and obtain a bounding box after the cyclic fusion process.

8. The system according to claim 7, characterized in that Based on the attribute information of the detection frame and the reference frame, the detection frame is cyclically fused through the preset target fusion algorithm to identify redundant detection frames and abnormal detection frames, including: Calculate the overlapping area of ​​the current detection frame and the reference frame; If the overlapping area between the current detection frame and the reference frame is greater than or equal to the overlapping area threshold, the current detection frame is marked as a redundant detection frame; If the overlapping area between the current detection frame and the reference frame is less than the overlapping area threshold, the target types of the current detection frame and the reference frame are compared; if the target types corresponding to the current detection frame and the reference frame are inconsistent, the detection frame is retained; If the target type corresponding to the current detection frame is consistent with that of the reference frame, the distance between the center point of the current detection frame and the center point of the reference frame is calculated; if the distance between the center point of the current detection frame and the center point of the reference frame is less than or equal to the distance threshold, the current detection frame is marked as a redundant detection frame; If the distance between the center point of the current detection frame and the center point of the reference frame is greater than the distance threshold, check whether the ID information of the current detection frame is consistent with the ID information of the reference frame; if the ID information of the current detection frame is consistent with the ID information of the reference frame, the detection frame is an abnormal detection frame, and the current detection frame is removed; If the ID information of the current detection box is inconsistent with the ID information of the reference box, the detection box is retained.

9. The system according to claim 1, characterized in that Each UAV includes an inertial measurement unit (IMU) for acquiring real-time attitude information of the UAV, wherein the attitude information includes the pitch angle, roll angle and heading angle of the UAV. The airborne communication module is also used to send the attitude information of the UAV to a ground control station.

10. The system according to claim 4, characterized in that Convert all true bounding boxes to the same map, and get the bounding boxes of all images in the same map including: Convert all generated real bounding boxes into the image space auxiliary coordinate system to generate image space auxiliary coordinate system coordinates of all real bounding boxes; Convert all generated image space auxiliary coordinate system coordinates into body coordinate system coordinates; Convert all generated body coordinate system coordinates into station center horizontal coordinate system coordinates; The generated geocentric horizontal coordinate system coordinates of all stations are converted into the same map, which is a geocentric rectangular coordinate system.

Images