Method for transmitting information among multiple unmanned aerial vehicles based on visual instruction

Abstract

The invention provides a method for transmitting information among multiple unmanned aerial vehicles based on visual instructions. The method comprises the following steps: step 1, determining a visual instruction library based on light signals; the visual instruction library is used for transmitting information between the unmanned aerial vehicles; the unmanned aerial vehicle is divided into a leader and a follower; step 2, the leader unmanned aerial vehicle continuously transmits a light instruction; the leader unmanned aerial vehicle is sequentially switched to a corresponding light state based on the visual instruction library, and continuously transmits the light instruction; step 3, the following unmanned aerial vehicle uses unmanned aerial vehicle detection and identification based on appearance and motion information under a complex background to realize detection of the leading unmanned aerial vehicle; through a global-local combined unmanned aerial vehicle detector, unmanned aerial vehicle identification is carried out to lead an unmanned aerial vehicle; and step 4, the following unmanned aerial vehicle uses a vision-based light instruction identification algorithm to realize identification of the light instruction transmitted by the leader unmanned aerial vehicle. The method and the device can realize cluster formation cooperation without depending on inter-machine communication, and can be suitable for a cluster combat scene in which communication is rejected or communication needs to be kept silent.

Description

Technical Field

[0001] The present application relates to the technical field of multi-agent systems, and in particular to a method for transmitting information between multi-agent machines based on visual instructions. Background Art

[0002] With the development of autonomous and networked communication technology for drones, the combat mode of drones is also constantly evolving. Single drones are limited by detection capabilities, payload field of view and other factors, making it difficult to complete complex combat missions. Compared with single drone systems, drone swarms can take advantage of their scale to complete complex tasks and have stronger survivability. However, most of the current research on drone swarm combat relies on wireless communication to achieve autonomous coordination between drones. In the future, as electromagnetic interference technology becomes more mature, the combat capability of drone swarms based on wireless communication will be greatly reduced. UAV swarms based on visual commands are expected to achieve highly intelligent clusters in denied environments or communication silence conditions because they do not rely on inter-machine communication and are "passive perception".

[0003] At present, some teams at home and abroad have conducted relevant research on the realization of UAV cluster formation coordination based on pure vision. Zhang Yongwei's team proposed a vision-based method for UAV landing and collision avoidance between UAVs, which takes reducing the difference between expected images and actual images as the control target, and uses an external multi-camera positioning system to measure the control input; Soria et al. analyzed the impact of the field of view size and installation position of the visual sensor on the cluster safety, orderliness, aggregation and other indicators, which helps the real UAV cluster system to determine the selection and installation direction of the visual sensor; Yang et al. subdivided the vision of the intelligent agent into visual perception lines to perceive obstacle information, and set the field of view matrix to store obstacle information, and finally let the intelligent agent move towards the center of the blank area of ​​the obstacle.

[0004] These studies only focus on certain technical points such as drone landing, collision avoidance, and the selection and installation of visual sensors, and have not formed a complete method and strategy for realizing cluster formations based on vision. Shen Xiaodong's team studied how to realize the formation transformation of drones with only visual perception, established a mathematical model to prevent visual occlusion and a mathematical model to avoid collisions during formation switching, designed an optimization objective function, and decomposed the formation transformation problem into the problem of target point allocation and the problem of path planning from the original position to the target position. Finally, this optimization problem was solved and the algorithm for solving the problem was improved. This research is still in the theoretical research stage. The problem is abstractly modeled as a multi-objective optimization problem with multiple constraints and solved by a planning method. It does not consider the formation strategy setting and transmission, drone identification, formation strategy identification and other issues that need to be solved in the actual scene to realize cluster formation coordination based on pure vision, and cannot adapt to the real combat environment.

[0005] In summary, there are few studies on the realization of UAV cluster formation coordination based on pure vision at home and abroad, and there is still no efficient, complete, and vision-based method and strategy for cluster formation that can adapt to real combat scenarios. Summary of the invention

[0006] The present application provides a method for transmitting information between multiple UAVs based on visual instructions, which can be used to solve the technical problem that there is little research on realizing UAV cluster formation coordination based on pure vision.

[0007] The present application provides a method for transmitting information between multiple drones based on visual instructions, the method comprising:

[0008] Step 1: Determine the visual command library based on light signals; the visual command library is used to transmit information between drones; drones are divided into leaders and followers;

[0009] Step 2: The Leader UAV continuously transmits the light command. The Leader UAV switches to the corresponding light state in turn based on the visual command library and continuously transmits the light command, so that the Follower UAV can recognize the light command information issued by the Leader UAV.

[0010] Step 3: The Follower UAV detects the Leader UAV using the UAV detection and recognition based on appearance and motion information under complex backgrounds; the Leader UAV is identified by the UAV detector combining global and local features;

[0011] Step 4: The Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV.

[0012] Step 5: The Follower UAV matches the instruction library based on the detected and identified Leader UAV and the light instruction signal it transmits, and responds to the formation strategy immediately according to the matching result. It adjusts its relative position with the Leader based on the virtual structure method to achieve autonomous formation execution.

[0013] Furthermore, the presentation of the visual instruction library is completed by lights, and the presentation of lights includes two aspects: the color of the light and the flashing mode of the light. The color of the light includes multiple colors, and the flashing mode of the light includes two consecutive flashing modes, a single flashing mode, a flashing mode with an interval of one time unit, and a flashing mode with an interval of two time units.

[0014] Furthermore, the communication mechanism between drones is that the Leader drone identifies itself by flashing twice in succession to transmit the command information downward, while the Follower drone responds by flashing once to indicate that it has received the information;

[0015] The light flashing modes and colors are divided into four categories, namely:

[0016] Startup and mission end categories, including departure, i.e. starting the drone and preparing to enter the mission state; return, i.e. returning the drone to the starting point or the predetermined return point;

[0017] Assembly type, including gathering, that is, drones fly to a designated assembly point to form a cluster;

[0018] Distribution and execution, including scattering, i.e. cluster dispersion, distribution according to task requirements; following, i.e. following the target to execute tasks;

[0019] Position adjustment and holding categories, adjustments include rising, falling, left translation, left translation and the corresponding adjusted unit distance and number of unit distances; holding position means that the drone remains stationary in the current air position.

[0020] Furthermore, in step 3, the Follower UAV detects the Leader UAV using UAV detection and recognition based on appearance and motion information under complex backgrounds; including:

[0021] Step 31, determining a global detector, which is used to search for drone targets in the full-size image; the global detector includes an appearance-based detection module and a motion-based detection module; drone detection based on motion features includes three parts: motion compensation, motion target detection, and drone classification;

[0022] Step 32, determining a local detector;

[0023] After the global detector detects the drone target, the local detector is activated to perform subsequent detection in a local search area cropped from the neighboring area around the target; the local detector includes a local appearance-based module and a local motion-based module;

[0024] Step 33, coordinating the global and local detectors through a switch;

[0025] When the local detector always fails to detect the target for a certain period of time, that is, the target is definitely lost, the global detector is called again through the switch.

[0026] Furthermore, step 31, determining a global detector, includes:

[0027] Step 311, motion compensation: using a homography matrix to describe the motion relationship between consecutive frame images, and using perspective transformation to align consecutive frame images;

[0028] First, evenly distributed grid points are selected in the previous frame as the initial key points, and then the pyramid Lucas-Kanade optical flow method is used to calculate the corresponding key points in the current frame to obtain the matched feature point pairs;

[0029] Then, the homography matrix is ​​calculated based on the matched feature point pairs, and iterative RANSAC is used to remove the interference caused by abnormal points;

[0030] Based on the homography matrix, align adjacent frame images through perspective transformation;

[0031] Step 312, moving target detection: using a multi-frame difference method to obtain candidate moving targets;

[0032] The three-frame difference method is used to improve the effect of moving target detection. For three consecutive frames of pictures f_(t-1)(x,y), f_t(x,y), and f_(t+1)(x,y), median filtering is first used to eliminate salt and pepper noise, and then the middle frame and the other two frames are differentially operated:

[0033] FD_12(x,y)=|f_t(x,y)-f_(t-1)(x,y)|

[0034] FD_23(x,y)=|f_t(x,y)-f_(t+1)(x,y)|

[0035] Among them, FD_12(x,y) is the absolute difference between time t and t-1; FD_23(x,y) is the absolute difference between time t and t+1;

[0036] Next, the difference image is segmented and binarized using a dynamic threshold, and a logical AND operation is performed on the difference image to suppress the target ghosting phenomenon:

[0037] D_1(x,y)=FD_12(x,y)∩FD_23(x,y)

[0038] D_1(x,y) is the binary image data of three consecutive frames of moving targets at time t; the candidate moving targets are obtained by processing;

[0039] Step 313, using appearance feature classification to classify drones and obtain drone targets;

[0040] Through a shallow classification network architecture, the first three layers of the network are used to extract the high-dimensional features of the drone, and finally a binary classification is performed through the fully connected layer and Softmax.

[0041] Furthermore, in step 4, the Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV, including:

[0042] Step 41, extracting light color based on edge detection and color feature segmentation: extracting the back panel area of ​​the image, image denoising preprocessing, canny edge detection, coordinate mapping, and color recognition;

[0043] Step 42, dynamic lighting command recognition based on time-space combination: time-space combination, that is, combining the lighting rules of multiple lighting signal images in time sequence and the position distribution in space to recognize the lighting command signal.

[0044] Furthermore, step 41, extracting light color based on edge detection and color feature segmentation, includes:

[0045] Firstly, by using the pixel value binarization function, the pixel values ​​within the color threshold range of the traffic light are set to white 255, and the pixel values ​​outside the threshold range are set to black 0, and the area with black backplane is initially extracted;

[0046] Then, the extracted image is pre-processed for denoising, and a low-pass filter is used to eliminate Gaussian noise, so that each pixel on the output function is the weighted sum of the corresponding pixel on the original image and the surrounding pixels, and the original image is smoothed; then, a median filter is used for secondary denoising to remove the speckle noise and salt and pepper noise of the image, eliminate the tiny spikes on the black backplane, disconnect the narrow connections, separate objects, eliminate small areas, and deal with the large amount of white noise that has not been processed due to binarization;

[0047] Then, we perform canny edge detection: we use the first-order partial derivative finite difference to calculate the gradient amplitude and direction, then suppress the gradient amplitude for non-maximum, and use the double threshold algorithm to detect and obtain the edge, finally extract the black backplane rectangle, detect the object contour and merge it with the original image, and draw the detected rectangular contour on the original image;

[0048] Then coordinate mapping is carried out: the outline is drawn on the initial image, the corner points of the outline are detected, the point coordinates are defined, and the coordinate mapping is carried out in the new image. At this time, the new image only contains the traffic light, the black backboard and a small part of the background;

[0049] Finally, the light color is recognized based on color feature segmentation; HSV color space is used for color recognition;

[0050] By converting the image to the HSV color space and then traversing each pixel of the image, we determine in turn whether the H, S, and V values ​​meet the constraints of different color lights; when the thresholds of each channel of the pixel point simultaneously meet the constraints of a certain color light, set its pixel mask to 255, otherwise set it to 0, so that binary images of different color lights can be obtained respectively;

[0051] Then, the binary image is searched for outer contours to obtain candidate image regions of all lights; finally, color attributes are assigned to candidate regions of each lamp body to complete color recognition.

[0052] Furthermore, step 42, based on the spatiotemporal joint dynamic lighting instruction recognition, includes:

[0053] Firstly, based on the static recognition results of a single frame image, the lighting rules of the light signal image in time sequence are extracted through the inter-frame target matching method;

[0054] Then, spatial distribution matching is performed based on the spatial position relationship;

[0055] Finally, based on the template matching method, the light signal that satisfies both the lighting timing and spatial distribution matching is matched with the instruction library.

[0056] Compared with the prior art, the present invention has the following significant advantages:

[0057] The present invention proposes a method for realizing UAV cluster formation based on vision, which can achieve cluster formation coordination without relying on inter-machine communication, and can be applied to cluster combat scenarios where communication denial or communication silence is required.

[0058] The present invention proposes a global-local combined drone detector.

[0059] The present invention proposes a light instruction recognition algorithm based on vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the distribution of swarm drone agents in a simulation system provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a method flow chart provided in an embodiment of the present application;

[0062] Figure 3 A light visual instruction design diagram provided for an embodiment of the present application;

[0063] Figure 4 A task instruction classification diagram provided for an embodiment of the present application;

[0064] Figure 5 A mapping relationship diagram of light signals to task instructions provided in an embodiment of the present application;

[0065] Figure 6 A Leader drone state switching declaration diagram provided in an embodiment of the present application;

[0066] Figure 7 A diagram of the YOLOv3 neural network structure provided in an embodiment of the present application;

[0067] Figure 8 A processing flow chart of a UAV detection algorithm based on motion features provided in an embodiment of the present application;

[0068] Fig. 9 This is a test result diagram of a small target drone dataset recognition test under a complex background provided by an embodiment of the present application;

[0069] Fig.10 A flow chart of a light color extraction algorithm based on edge detection and color feature segmentation provided in an embodiment of the present application;

[0070] Fig.11 A flow chart of a dynamic lighting instruction recognition algorithm based on spatiotemporal combination provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0072] The object of the present invention is to provide a method and system for transmitting information between multiple drones based on visual instructions.

[0073] The technical solution to achieve the purpose of the present invention is: a method for transmitting information between multiple drones based on visual instructions, comprising:

[0074] Step 1: Determine the visual command library based on light signals; build a command library with obvious features and complete commands to realize the mapping of light commands to cluster flight action commands.

[0075] The visual instructions are presented by lights, which include two aspects: the color of the light and the flashing mode of the light. The color of the light includes a variety of colors, such as yellow, red, blue, green and purple. The flashing mode of the light includes two consecutive flashing modes, a single flashing mode, a flashing mode with a time interval of one unit and a flashing mode with a time interval of two units. Figure 3 In addition, the light command mode can also be expanded as needed.

[0076] In addition, the lighting command mode can also be expanded as needed;

[0077] Drones are divided into leaders and followers.

[0078] A cylindrical light device is installed on the Leader to continuously broadcast light instructions to all directions, so that the Leader image seen from all directions is similar;

[0079] The Follower drones are equipped with four wide-angle cameras in a cross-staggered manner to obtain a 360-degree field of view, enabling real-time observation of the status of the cluster head drones throughout the process, and using a drone detection and recognition algorithm based on appearance and motion information under complex backgrounds to identify the Leader drone;

[0080] The Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV;

[0081] Follower drones match the command library based on light command signals to achieve autonomous formation execution.

[0082] Furthermore, based on the changes in multi-dimensional state information such as the color and flashing frequency of multi-color LED lights, a command library with obvious features and complete instructions is constructed to achieve the mapping of light instructions to cluster flight action instructions.

[0083] In order to expand the usage scenarios of the UAV formation coordination method based on light commands, a propagation mechanism is specially determined, that is, the Leader UAV identifies itself by flashing twice in succession for propagating command information downward, and the Follower UAV responds by flashing once to indicate that it has received the information. This mode can effectively avoid the situation when the scale of the UAV cluster is large, the Follower UAV is far away from the Leader UAV and cannot effectively identify the command information, or the Follower UAV cannot directly observe and receive the Leader UAV when transmitting the light command information.

[0084] Ten unique visual commands are generated based on the light flashing pattern and color of the light command design method. Figure 4 The light flashing modes and colors are divided into four categories:

[0085] (1) Startup and mission completion, including departure, i.e. starting the drone and preparing to enter the mission state; return, i.e. returning the drone to the starting point or the predetermined return point;

[0086] (2) Assembly type, including gathering, that is, drones fly to a designated assembly point to form a cluster.

[0087] (3) Distribution and execution, including dispersion, i.e., cluster dispersion, distribution according to task requirements; and following, i.e., following the target to execute the task.

[0088] (4) Position adjustment and maintenance category, including rising, falling, left translation, left translation and corresponding adjusted unit distance and number of unit distances; maintaining position means that the drone remains stationary in the current air position. The unit distance is determined according to the actual situation. For example, 1 meter, then the height of 1 meter is one unit distance of height; the specific instructions are similar to: rising 1 meter, the drone rises 1 meter; falling 1 meter, the drone falls 1 meter; left translation 1 meter, the drone moves 1 meter to the left; right translation 1 meter, the drone moves 1 meter to the right.

[0089] Step 2: The Leader drone continues to broadcast light instructions;

[0090] in accordance with Figure 6 The Leader UAV state switching statement shown in the figure shows that the Leader UAV switches to the corresponding light state in sequence based on the visual command library and continuously transmits the light command so that the Follower UAV can recognize the light command information issued by the Leader UAV.

[0091] Step 3: The Follower UAV detects the Leader UAV using UAV detection and recognition based on appearance and motion information in a complex background.

[0092] Train a global-local drone detector to identify the leader drone based on appearance and motion information in complex backgrounds.

[0093] The global detector includes an appearance-based detection module and a motion-based detection module. YOLOv3 is used as the appearance-based detection module, and its neural network structure is as follows: Figure 7 The UAV detection algorithm based on motion features is used as the motion-based detection module, which includes motion compensation, moving target detection, and UAV classification. The processing flow is as follows: Figure 8 shown.

[0094] After the global detector detects the drone target, the local detector is activated to perform subsequent detection in the local search area cropped from the neighboring area around the target. Finally, a switcher is designed to coordinate the global and local detectors. When the local detector always fails to detect the target for a certain period of time, that is, when the target is determined to be lost, the global detector is recalled through the switcher.

[0095] Finally, the algorithm recognition effect is verified using a small target drone dataset under complex background. The recognition results are as follows: Fig. 9 shown.

[0096] Furthermore, in view of the current situation that the target image is blurred and small in size due to the movement of the platform and the long distance in complex scenes, the real-time and accuracy of target detection are seriously reduced. A drone detection method based on appearance and motion information under complex backgrounds is adopted, and a drone detector based on global-local combination is designed to achieve effective recognition of air-to-air targets in motion, providing the prerequisite for the issuance of cluster formation tasks. The specific method is as follows:

[0097] Step 31, design a global detector, which is used to search for drone targets in full-size images; the global detector includes an appearance-based detection module and a motion-based detection module; when the appearance features are unreliable, the motion-based detection module serves as a good auxiliary tool. YOLOv3 is used as the appearance-based detection module, and the drone detection algorithm based on motion features is used as the motion-based detection module. The drone detection algorithm based on motion features includes three parts: motion compensation, moving target detection, and drone classification.

[0098] Step 311, motion compensation

[0099] Since the observation camera and the target drone are moving at the same time, camera motion estimation is required to separate the background motion and the drone motion. The homography matrix is ​​used to describe the motion relationship between consecutive frame images, and the perspective transformation is used to align consecutive frame images; the perspective transformation accurately models the relative motion relationship from a 2D plane to a specified plane in the 3D world. First, uniformly distributed grid points are selected in the previous frame as the initial key points, and then the pyramid Lucas-Kanade optical flow method is used to calculate the corresponding key points in the current frame to obtain the matched feature point pairs;

[0100] Then, the homography matrix is ​​calculated based on the matched feature point pairs, and iterative RANSAC is used to remove the interference caused by abnormal points;

[0101] Based on the homography matrix, adjacent frame images are aligned through perspective transformation.

[0102] Step 312, moving target detection: using a multi-frame difference method to obtain candidate moving targets;

[0103] The multi-frame difference method is a commonly used moving target detection algorithm. The algorithm performs a difference operation on two or more frames of images that are continuous in time, subtracts the pixels corresponding to different frames, and determines the absolute value of the grayscale difference. When the absolute value exceeds a certain threshold, it can be determined as a moving target, thereby realizing the detection function of moving targets.

[0104] The commonly used two-frame difference method has problems such as severe ghosting when the target moves quickly, discontinuous target contours, and much background noise interference. Therefore, the three-frame difference method is used to improve the moving target detection effect. For the three consecutive frames of pictures f_(t-1)(x,y), f_t(x,y), and f_(t+1)(x,y), first use median filtering to eliminate salt and pepper noise, and then perform difference operations on the middle frame and the other two frames respectively:

[0105] FD_12(x,y)=|f_t(x,y)-f_(t-1)(x,y)|

[0106] FD_23(x,y)=|f_t(x,y)-f_(t+1)(x,y)|

[0107] Among them, FD_12(x,y) is the absolute difference between time t and t-1; FD_23(x,y) is the absolute difference between time t and t+1;

[0108] Next, the difference image is segmented and binarized using a dynamic threshold, and a logical AND operation is performed on the difference image to suppress the target ghosting phenomenon:

[0109] D_1(x,y)=FD_12(x,y)∩FD_23(x,y)

[0110] D_1(x,y) is the binary image data of three consecutive frames of moving targets at time t;

[0111] At this time, the binary image still contains a lot of noise and small holes. Next, morphological opening and closing operations, connected domain analysis, minimum envelope box and envelope box merging are used to obtain candidate moving targets.

[0112] Step 313, using appearance feature classification to classify drones and obtain drone targets;

[0113] The candidate moving targets still include a large number of non-UAV targets and interferences, such as moving cars, pedestrians, shaking leaves, water surface, and image alignment errors. In order to separate the UAV from the candidate moving targets, appearance feature classification is used to obtain the final moving UAV target.

[0114] Since the size of candidate moving targets is small and the classification network needs to be frequently called to eliminate the influence of non-UAV targets, using common classification neural networks such as ResNet, VGG, GoogleNet, DenseNet, etc. will consume a lot of computing resources, which is not conducive to real-time deployment on mobile terminals. Therefore, a shallow classification network architecture is designed, using the first three layers of the network to extract the high-dimensional features of the UAV, and finally performing binary classification through the fully connected layer and Softmax;

[0115] Step 32, designing a local detector;

[0116] Since the global detector needs to search for targets in the full-size image, there are many interferences and large amount of calculation. After the global detector detects the drone target, it will activate the local detector and perform subsequent detection in the local search area cropped from the neighboring area around the target. The local detector includes a local appearance-based module and a local motion-based module. Compared with the global detector, the local detector occupies significantly less computing power and has significantly improved computing efficiency.

[0117] Step 33, design a switch to coordinate the global and local detectors;

[0118] Since target occlusion and violent camera motion are unavoidable in real-world tasks, an effective method is needed to prevent the detector from falling into the wrong local search domain when the target is lost. Therefore, a switcher is designed to coordinate the global and local detectors. When the local detector always fails to detect the target for a certain period of time, that is, when the target is determined to be lost, the global detector is re-called through the switcher; the switcher should be able to adaptively adjust its search area based on the detection results of the previous frame, so as to avoid erroneously searching the local search area when the local detector temporarily fails.

[0119] Step 4: The Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV.

[0120] Step 4 consists of two parts:

[0121] The first part is to extract light color based on edge detection and color feature segmentation. The method flow chart is as follows: Fig.10 Through the steps of backplane area extraction, image preprocessing (denoising), canny edge detection, coordinate mapping, color recognition, etc., the extraction and recognition of various light colors can be achieved.

[0122] The second part is the dynamic lighting command recognition based on spatiotemporal combination. The method flow is as follows: Fig.11 Based on the static recognition results of a single-frame image, the lighting rules of the light signal image in time sequence are extracted through the inter-frame target matching method; then the spatial distribution matching is performed based on the spatial position relationship, and finally the light signal that satisfies both the lighting timing and spatial distribution matching is matched with the instruction library based on the template matching method to complete the recognition of the light instruction.

[0123] Specifically: Step 4 includes:

[0124] Step 41, extracting light color based on edge detection and color feature segmentation: extracting the back panel area of ​​the image, image denoising preprocessing, canny edge detection, coordinate mapping, and color recognition;

[0125] In order to address the interference of external environment such as blue sky and sunlight, the black drone back panel area of ​​the signal light is extracted by taking advantage of the fact that the back panel of the LED signal light is quite different from the environmental background. Then, the color of the light is recognized by taking advantage of the fact that the HSV color space is less affected by light.

[0126] Firstly, by using the pixel value binarization function, the pixel values ​​within the color threshold range of the traffic light are set to white 255, and the pixel values ​​outside the threshold range are set to black 0, and the area with black backplane is initially extracted;

[0127] Then, the extracted image is pre-processed for denoising, and a low-pass filter is used to eliminate Gaussian noise, so that each pixel on the output function is the weighted sum of the corresponding pixel on the original image and the surrounding pixels, and the original image is smoothed; then, a median filter is used for secondary denoising to remove the speckle noise and salt and pepper noise of the image, eliminate the tiny spikes on the black backplane, disconnect the narrow connections, separate objects, eliminate small areas, and deal with the large amount of white noise that has not been processed due to binarization;

[0128] Then, Canny edge detection is performed: the gradient amplitude and direction are calculated using the first-order partial derivative finite difference, the gradient amplitude is non-maximum suppressed, and the edge is detected using the double threshold algorithm to achieve the final extraction of the black backplane rectangular frame, the object contour is detected and fused with the original image, and the detected rectangular contour is drawn on the original image.

[0129] Then coordinate mapping is carried out: the outline is drawn on the initial image, the corner points of the outline are detected, the point coordinates are defined, and the coordinate mapping is carried out in the new image. At this time, the new image only contains the traffic light, the black backboard and a small part of the background;

[0130] Finally, the color of the light is recognized based on color feature segmentation; the commonly used color models are RGB color space and HSV color space. R in RGB color space represents red, G represents green, and B represents blue. The three channels of HSV color space are hue (H), saturation (S), and brightness (V). However, due to the low correlation between the three components of RGB, it cannot be used to separate the similar areas of two colors. The HSV color space eliminates the correlation between the three components and can more intuitively express the brightness, hue, and vividness of the color, making it easier to process similar colors. Therefore, the HSV color space is used for color recognition.

[0131] By converting the image to the HSV color space and then traversing each pixel of the image, we determine in turn whether the H, S, and V values ​​meet the constraints of different color lights; when the thresholds of each channel of the pixel point simultaneously meet the constraints of a certain color light, set its pixel mask to 255, otherwise set it to 0, so that binary images of different color lights can be obtained respectively;

[0132] Then, the binary image is searched for outer contours to obtain candidate image regions of all lights. Finally, color attributes are assigned to candidate regions of each light body to complete color recognition.

[0133] Step 42, dynamic lighting instruction recognition based on spatiotemporal combination:

[0134] Spatiotemporal union, that is, combining the lighting rules of multiple light signal images in time sequence with their position distribution in space to identify light command signals. First, based on the static recognition results of a single frame image, the lighting rules of the light signal image in time sequence are extracted through the inter-frame target matching method; then spatial distribution matching is performed based on the spatial position relationship, and finally, the light signal that satisfies both the lighting sequence and spatial distribution matching is matched with the command library based on the template matching method.

[0135] Step 5: The Follower UAV matches the instruction library based on the detected and identified Leader UAV and the light instruction signal it transmits, and responds to the formation strategy immediately according to the matching result. It adjusts its relative position with the Leader based on the virtual structure method to achieve autonomous formation execution.

[0136] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A method for transmitting information between multiple drones based on visual instructions, characterized in that: The method comprises: Step 1: Determine the visual command library based on light signals; the visual command library is used to transmit information between drones; drones are divided into leaders and followers; Step 2: The Leader UAV continuously transmits the light command. The Leader UAV switches to the corresponding light state in turn based on the visual command library and continuously transmits the light command, so that the Follower UAV can recognize the light command information issued by the Leader UAV. Step 3: The Follower UAV detects the Leader UAV using the UAV detection and recognition based on appearance and motion information under complex backgrounds; the Leader UAV is identified by the UAV detector combining global and local features; Step 4: The Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV. Step 5: The Follower UAV matches the instruction library based on the detected and identified Leader UAV and the light instruction signal it transmits, and responds to the formation strategy immediately according to the matching result. It adjusts its relative position with the Leader based on the virtual structure method to achieve autonomous formation execution.

2. The method according to claim 1, characterized in that The presentation of the visual instruction library is completed by lights. The presentation of lights includes two aspects: the color of the lights and the flashing mode of the lights. The color of the lights includes multiple colors. The flashing modes of the lights include two consecutive flashing modes, a single flashing mode, a flashing mode with an interval of one time unit, and a flashing mode with an interval of two time units.

3. The method according to claim 2, characterized in that The communication mechanism between drones is that the Leader drone identifies itself by flashing twice in succession to transmit command information downward, while the Follower drone responds by flashing once to indicate that it has received the information; The light flashing modes and colors are divided into four categories, namely: Startup and mission end categories, including departure, i.e. starting the drone and preparing to enter the mission state; return, i.e. returning the drone to the starting point or the predetermined return point; Assembly type, including gathering, that is, drones fly to a designated assembly point to form a cluster; Distribution and execution, including scattering, i.e. cluster dispersion, distribution according to task requirements; following, i.e. following the target to execute tasks; Position adjustment and holding categories, adjustments include rising, falling, left translation, left translation and the corresponding adjusted unit distance and number of unit distances; holding position means that the drone remains stationary in the current air position.

4. The method according to claim 1, characterized in that: Step 3: The Follower UAV detects the Leader UAV using UAV detection and recognition based on appearance and motion information in a complex background; including: Step 31, determining a global detector, which is used to search for drone targets in the full-size image; the global detector includes an appearance-based detection module and a motion-based detection module; drone detection based on motion features includes three parts: motion compensation, motion target detection, and drone classification; Step 32, determining a local detector; After the global detector detects the drone target, the local detector is activated to perform subsequent detection in a local search area cropped from the neighboring area around the target; the local detector includes a local appearance-based module and a local motion-based module; Step 33, coordinating the global and local detectors through a switch; When the local detector always fails to detect the target for a certain period of time, that is, the target is definitely lost, the global detector is called again through the switch.

5. The method according to claim 4, characterized in that Step 31, determining a global detector, includes: Step 311, motion compensation: using a homography matrix to describe the motion relationship between consecutive frame images, and using perspective transformation to align consecutive frame images; First, evenly distributed grid points are selected in the previous frame as the initial key points, and then the pyramid Lucas-Kanade optical flow method is used to calculate the corresponding key points in the current frame to obtain the matched feature point pairs; Then, the homography matrix is ​​calculated based on the matched feature point pairs, and iterative RANSAC is used to remove the interference caused by abnormal points; Based on the homography matrix, align adjacent frame images through perspective transformation; Step 312, moving target detection: using a multi-frame difference method to obtain candidate moving targets; The three-frame difference method is used to improve the effect of moving target detection. For three consecutive frames of pictures f_(t-1)(x,y), f_t(x,y), and f_(t+1)(x,y), median filtering is first used to eliminate salt and pepper noise, and then the middle frame and the other two frames are differentially operated: FD_12(x,y)=|f_t(x,y)-f_(t-1)(x,y)| FD_23(x,y)=|f_t(x,y)-f_(t+1)(x,y)| Among them, FD_12(x,y) is the absolute difference between time t and t-1; FD_23(x,y) is the absolute difference between time t and t+1; Next, the difference image is segmented and binarized using a dynamic threshold, and a logical AND operation is performed on the difference image to suppress the target ghosting phenomenon: D_1(x,y)=FD_12(x,y)∩FD_23(x,y) D_1(x,y) is the binary image data of three consecutive frames of moving targets at time t; the candidate moving targets are obtained by processing; Step 313, using appearance feature classification to classify drones and obtain drone targets; Through a shallow classification network architecture, the first three layers of the network are used to extract the high-dimensional features of the drone, and finally a binary classification is performed through the fully connected layer and Softmax.

6. The method according to claim 1, characterized in that Step 4: The Follower UAV uses a vision-based light command recognition algorithm to recognize the light command transmitted by the Leader UAV, including: Step 41, extracting light color based on edge detection and color feature segmentation: extracting the back panel area of ​​the image, image denoising preprocessing, canny edge detection, coordinate mapping, and color recognition; Step 42, dynamic lighting command recognition based on time-space combination: time-space combination, that is, combining the lighting rules of multiple lighting signal images in time sequence and the position distribution in space to recognize the lighting command signal.

7. The method according to claim 6, characterized in that Step 41, extracting light color based on edge detection and color feature segmentation, includes: Firstly, by using the pixel value binarization function, the pixel values ​​within the color threshold range of the traffic light are set to white 255, and the pixel values ​​outside the threshold range are set to black 0, and the area with black backplane is initially extracted; Then, the extracted image is pre-processed for denoising, and a low-pass filter is used to eliminate Gaussian noise, so that each pixel on the output function is the weighted sum of the corresponding pixel on the original image and the surrounding pixels, and the original image is smoothed; then, a median filter is used for secondary denoising to remove the speckle noise and salt and pepper noise of the image, eliminate the tiny spikes on the black backplane, disconnect the narrow connections, separate objects, eliminate small areas, and deal with the large amount of white noise that has not been processed due to binarization; Then, we perform canny edge detection: we use the first-order partial derivative finite difference to calculate the gradient amplitude and direction, then suppress the gradient amplitude for non-maximum, and use the double threshold algorithm to detect and obtain the edge, finally extract the black backplane rectangle, detect the object contour and merge it with the original image, and draw the detected rectangular contour on the original image; Then coordinate mapping is carried out: the outline is drawn on the initial image, the corner points of the outline are detected, the point coordinates are defined, and the coordinate mapping is carried out in the new image. At this time, the new image only contains the traffic light, the black backboard and a small part of the background; Finally, the light color is recognized based on color feature segmentation; HSV color space is used for color recognition; By converting the image to the HSV color space and then traversing each pixel of the image, we determine in turn whether the H, S, and V values ​​meet the constraints of different color lights; when the thresholds of each channel of the pixel point simultaneously meet the constraints of a certain color light, set its pixel mask to 255, otherwise set it to 0, so that binary images of different color lights can be obtained respectively; Then, the binary image is searched for outer contours to obtain candidate image regions of all lights; finally, color attributes are assigned to candidate regions of each lamp body to complete color recognition.

8. The method according to claim 6, characterized in that Step 42, dynamic lighting instruction recognition based on spatiotemporal combination, includes: Firstly, based on the static recognition results of a single frame image, the lighting rules of the light signal image in time sequence are extracted through the inter-frame target matching method; Then, spatial distribution matching is performed based on the spatial position relationship; Finally, based on the template matching method, the light signal that satisfies both the lighting timing and spatial distribution matching is matched with the instruction library.

Images