Congested environment-oriented pure visual distributed cluster flight emergence method and system

Abstract

The invention discloses a pure vision distributed cluster flight emergence method and system for a crowded environment, and belongs to the technical field of unmanned aerial vehicle cluster flight, and the method comprises the following steps: S1, obtaining an observed image; s2, analyzing the image to obtain an environment depth image and an RGB image; s3, analyzing the RGB image to obtain the position and speed information of the neighbor unmanned aerial vehicle; s4, generating a grid map and an ESDF map from the environment depth map; s5, the airborne intelligent end machine synthesizes the task waypoint and the information in the S3 and the S4, and performs cluster decision making by using a cluster model to obtain a speed decision amount; s6, the airborne intelligent end machine sends the speed decision quantity to the unmanned aerial vehicle autopilot; and S7, feeding the flight state information back to the airborne intelligent terminal machine. The invention aims to solve the problems that an unmanned aerial vehicle cluster system excessively depends on inter-vehicle communication and cannot deal with a complex crowded environment, and individuals are low in adaptability to the complex crowded environment.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) swarm flight technology, and in particular to a pure visual distributed swarm flight emergence method and system for crowded environments. Background Art

[0002] As drone technology shines in battlefields, aerial photography, search and rescue, surveying and mapping and other fields, distributed drone clusters can handle more complex tasks and have higher fault tolerance than single drones, and have therefore received widespread attention in both civil and military fields.

[0003] In order for fully autonomous drone swarms to achieve higher mission performance, swarm collaborative decision-making is very necessary. So far, research on swarm collaborative decision-making and control related technologies has achieved rich results. From the perspective of the core technical ideas of decision-making and control, current research can be mainly divided into methods based on bionic behavior rules, artificial potential field methods, and methods based on collaborative planning.

[0004] However, most current studies still have the following problems: 1) It is assumed that there is efficient communication between drones, which makes the cluster system unable to be effectively applied in complex environments with limited communication; 2) The default usage scenario is relatively open, or there are only a few obstacles, which cannot cope with crowded indoor complex environments; 3) Although the collaborative planning-based method can avoid obstacles and perform cluster flight under effective communication conditions, the nature of trajectory planning increases the probability that it cannot be solved in complex environments, and cluster individuals are prone to fall into local optimality, which in turn affects the entire cluster. Summary of the invention

[0005] The purpose of the present invention is to provide a pure visual distributed cluster flight emergence method and system for crowded environments, in which individual drones can obtain environmental obstacle information and neighbor individual location information through visual sensors, thereby solving the problem that the drone cluster system is overly dependent on inter-machine communication, cannot cope with complex crowded environments, and has low adaptability of individuals to complex crowded environments.

[0006] To achieve the above object, the present invention provides a pure visual distributed cluster flight emergence method for crowded environments, comprising the following steps:

[0007] S1, the rotorcraft drone obtains the observed visible light image through the onboard depth camera;

[0008] S2, obtains the environment depth map and RGB map by analyzing the image;

[0009] S3, analyzes the RGB image, performs Kalman filtering on the location information of the neighboring drone, and estimates the location and speed information of the neighboring drone;

[0010] S4, generating a raster map and an ESDF map from the environment depth map;

[0011] S5, the airborne intelligent terminal weights the cluster mission waypoints of the cluster ground station and the position and speed information of the neighboring UAVs, and after weighting, uses the cluster model to make cluster decisions in conjunction with the information of the grid map and the ESDF map to obtain the speed decision amount;

[0012] S6, the airborne intelligent terminal sends the speed decision amount to the UAV autopilot, and the UAV autopilot controls the rotor UAV to perform cluster flight according to the speed decision amount;

[0013] S7, the rotorcraft drone feeds back the flight status information to the onboard intelligent terminal.

[0014] Preferably, in S2, the specific process of parsing the image is: first, the friendly drone in the image is identified, the identification frame coordinates and size of the friendly drone are obtained, an RGB image is generated, and then the neighbor individual image coordinates identified in the RGB image are removed from the image to obtain an environmental depth map.

[0015] Preferably, the specific steps of S3 are:

[0016] S31, use yolov8 to identify individuals, and then perform data association on the identification results to distinguish different neighbor individuals in the field of view;

[0017] S32, obtain the pixel coordinates (u c ,v c ), and identification box parameters (s b ,r b ), where s b is the detection box area, r b is the detection box aspect ratio;

[0018] S33, project the pixel coordinates and the identification frame parameters into the image to obtain the distance information d of the observed neighboring drone. The local coordinates of the neighboring drone in the inertial coordinate system are

[0019] M1M2[x,y,z,1] T =d[u,v,1] T ;

[0020] Where M1 is the camera internal parameter, and M2 is the rotation matrix of the camera relative to the world system;

[0021] S34, performing Kalman filtering on the observed neighboring UAV position information to estimate the position and speed information of the neighboring UAV.

[0022] Preferably, the specific process of S4 is to open source the environment depth map using an open source method to generate a raster map and an ESDF map.

[0023] Preferably, the specific steps of S5 are:

[0024] S51, establish cluster model;

[0025] S51, using the ESDF map, obtaining obstacle position information and obtaining obstacle avoidance behavior as the first speed decision quantity;

[0026] S52, using the position and speed information of the neighboring drone in S3, obtains the distance keeping behavior as the second speed decision quantity;

[0027] S53, using the cluster mission waypoints in the cluster ground station, obtain the mission behavior as the speed decision quantity three;

[0028] S54, weighting the speed decision quantity 2 and the speed decision quantity 3, with the weight being 1, to obtain the speed decision quantity 4;

[0029] S55, combining the speed decision quantity 4 with the speed decision quantity 1 to perform cluster decision making and obtain the speed decision quantity.

[0030] Preferably, the specific process of establishing the cluster model is:

[0031] Using an undirected graph Represents a drone cluster with N individuals, where For drone collection, is the interaction edge between UAVs, where R max is the maximum sensing range of the drone, d ij is the displacement vector of UAV i relative to j, and the detected neighbor set of UAV i is defined as Each drone i in the swarm model follows the following motion model:

[0032]

[0033] where p i and v i is the position and speed of UAV i, k is a control parameter greater than 0, v d,i It is the input of the UAV speed controller and also the speed decision variable, which satisfies the maximum speed limit of UAV i.

[0034] Preferably, the obstacle avoidance behavior is obtained as follows:

[0035] Obtain obstacle information from ESDF maps Where l is the number of obstacles obtained, αk is the confidence of obstacle information, d obs,k is the shortest vector pointing from UAV i to the surface of obstacle k, k∈l;

[0036] The position coordinates of drone i are p i (x i ,y i ,z i ), the obstacle position coordinates are p g (x g ,y g ,z g ), the relative position relationship between UAV i and the obstacle is With the current speed v of the drone i i Predict the next n time steps t step The position p i,pred (t setp ), the relative position relationship between each predicted position and the nearest obstacle point is d i,pred (t setp );

[0037] Introducing the predicted obstacle avoidance parameter R safe , that is, UAV i predicts that there will be obstacles less than R in a certain step length according to the current speed. safe , then the obstacle avoidance rule needs to be executed, R reach is the threshold for the UAV to determine whether it has reached the target area, R avoid The distance threshold at which the drone determines that it needs to start avoiding obstacles, where R avoid =R0+kv, R0 is set to 3 meters, k is 1; the following conditions are met

[0038]

[0039] The expected velocity vector generated according to the obstacle avoidance rule is:

[0040]

[0041] Preferably, the distance keeping behavior is obtained in the following manner:

[0042] Assume there are N individuals in the cluster, and the position coordinates of drone i are p i (x i ,y i ,z i ), and the relative position relationship with the jth UAV j is d i,j =p i -p j , where the position of UAV j is obtained by visual observation, and its credibility is α nei,j , and 0≤α nei,j≤1, according to the distance keeping rule, the expected velocity component generated by UAV j on UAV i is:

[0043]

[0044] Where R align Represents the safe distance between drones. The distance between drone j and drone i is less than the safe distance R between drones. align When , the desired velocity components moving away from each other are generated, and vice versa, the desired velocity components moving toward each other are generated;

[0045] The task behavior is obtained as follows:

[0046] The position coordinates of drone i are p i (x i ,y i ,z i ), the position coordinate of the target is p g (x g ,y g ,z g ), the relative position relationship between UAV i and the target is d goal,i =p g -p i , according to the task behavior rules, the expected velocity vector generated by UAV i is:

[0047]

[0048] in is a unit vector; when the distance d from UAV i to the target goal,i Less than R mission As the distance to the target gets closer, the expected speed gradually decreases. When the distance is greater than R mission When the desired speed is maintained at V mmax , where R mission The size of is related to the number of individuals in the cluster, N. k is the coefficient;

[0049] The speed decision quantity is:

[0050] v cmd =k1v obs +k2v align +k3v mission ;

[0051] Among them, k1, k2, and k3 represent the weight of speed decision quantity one, the weight of speed decision quantity two, and the weight of speed decision quantity three, respectively, and the sum of k2 and k3 is 1.

[0052] Preferably, in S6, the optimization process of cluster flight is as follows:

[0053] Setting Φobs represents the proportion of individuals in the cluster colliding with obstacles,

[0054]

[0055] where Θ(·) is the Heaviside step function, For p i (t) in symbolic form, when p i (t) When in an obstacle, Take positive value, otherwise take negative value;

[0056] Setting Φ col represents the proportion of collisions between individuals in the cluster,

[0057]

[0058] Setting Φ mis Indicates the completion of the cluster task, the time when the centroid reaches the target point, and the number of surviving individuals.

[0059]

[0060] According to the above metrics, a single objective optimization function is constructed.

[0061]

[0062] Use open source tools to solve the algorithm and obtain the optimal control parameters.

[0063] The present invention also provides a pure visual distributed cluster flight emergence system for crowded environments, which adopts the above-mentioned pure visual distributed cluster flight emergence method for crowded environments, including:

[0064] A visual perception module, used to obtain visible light images;

[0065] Cluster ground station, used to issue mission instructions;

[0066] The cluster decision module is used to process and extract information from visible light images and task instructions to obtain speed decision quantities;

[0067] An unmanned driving module, used for receiving and executing speed decision quantities to control the movement of the unmanned aerial vehicle; and a unmanned aerial vehicle body.

[0068] Therefore, the pure visual distributed cluster flight emergence method and system for crowded environments using the above structure in the present invention has the following advantages:

[0069] 1. Most existing cluster systems rely on inter-machine communication, but in the present invention, thanks to the design in the visual perception module, only the onboard depth camera module is needed to obtain obstacle information and neighbor individual information in the environment. The present invention uses visible light to classify visual information into neighbor individual information and environmental obstacle information, and further uses depth information to obtain accurate estimation of the location information of the drone individual on the neighbor individual and obstacles in the environment.

[0070] 2. In the existing research on cluster systems based on behavioral rules, most of them assume that the flight environment is relatively empty or there are only a few obstacles. In this invention, by using the perceived neighbor and obstacle information as input, considering the perception error and credibility, reliable obstacle avoidance, distance keeping, and task rules are designed to drive individuals to perform reasonable behaviors, thereby realizing a distributed drone cluster based on behavioral rules that can cope with crowded obstacle environments.

[0071] 3. In order to solve the problem that it is difficult to select behavioral weights and balance multiple indicators,

[0072] The present invention quantifies the task execution capability of the cluster system and designs a target optimization function.

[0073] The optimal weight parameters are obtained through genetic algorithm.

[0074] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A block diagram of a cluster of individual drones in an embodiment of a purely visual distributed cluster flight emergence method and system for crowded environments according to the present invention;

[0076] Figure 2 An ESDF map of an embodiment of a purely visual distributed cluster flight emergence method and system for a crowded environment according to the present invention;

[0077] Figure 3 A schematic diagram of obstacle avoidance rules of a purely visual distributed cluster flight emergence method and system embodiment for crowded environments of the present invention;

[0078] Figure 4 It is a schematic diagram of distance maintenance of a purely visual distributed cluster flight emergence method and system embodiment for crowded environments of the present invention;

[0079] Figure 5 It is a task behavior diagram of an embodiment of a pure visual distributed cluster flight emergence method and system for crowded environments of the present invention;

[0080] Figure 6The first is a cluster flight simulation effect in a typical crowded scene of the pure visual distributed cluster flight emergence method and system embodiment of the present invention for crowded environments;

[0081] Figure 7 This is the second cluster flight simulation effect in a typical crowded scene of the pure visual distributed cluster flight emergence method and system embodiment of the present invention for crowded environments. DETAILED DESCRIPTION

[0082] Example

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0084] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0086] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0087] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "setting", "installation" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0088] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0089] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as “upper”, “lower”, “left”, “right”, “inside” and “outside” are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they cannot be understood as limitations on the present invention.

[0090] The specific model specifications need to be selected and determined based on the actual specifications of the device, and the specific selection calculation method adopts the existing technology in this field, so it will not be described in detail.

[0091] like Figure 1 As shown, a purely visual distributed cluster flight emergence method for crowded environments includes the following steps:

[0092] S1, the rotorcraft drone obtains the observed visible light image through the onboard depth camera;

[0093] S2, obtains the environment depth map and RGB map by analyzing the image;

[0094] In S2, the specific process of parsing the image is as follows: first, the friendly drone in the image is identified, the coordinates and size of the identification frame of the friendly drone are obtained, and an RGB image is generated. Then, the coordinates of the neighbor individual images identified in the RGB image are removed from the image to obtain an environmental depth map.

[0095] S3, analyzes the RGB image, performs Kalman filtering on the location information of the neighboring drone, and estimates the location and speed information of the neighboring drone;

[0096] The specific steps of S3 are:

[0097] S31, use yolov8 to identify individuals, and then perform data association on the identification results to distinguish different neighbor individuals in the field of view;

[0098] Taking a drone individual in the field of view as an example, the location information of the drone is further estimated. S32, obtaining the pixel coordinates (u c ,v c ), and identification box parameters (s b ,r b ), where s b is the detection box area, r b is the detection box aspect ratio;

[0099] S33, project the pixel coordinates and the identification frame parameters into the image to obtain the distance information d of the observed neighboring drone. The local coordinates of the neighboring drone in the inertial coordinate system are

[0100] M1M2[x,y,z,1] T =d[u,v,1] T ;

[0101] Where M1 is the camera internal parameter, and M2 is the rotation matrix of the camera relative to the world system.

[0102] S34, performing Kalman filtering on the observed neighboring UAV position information to estimate the position and speed information of the neighboring UAV.

[0103] S4, generating a raster map and an ESDF map from the environment depth map;

[0104] like Figure 2 As shown in the figure, the specific process of S4 is to obtain the environment depth map from the depth camera, remove the neighbor individual pixel coordinates identified in the RGB map from the depth map, only build a map of the environment information in the depth map, and use open source methods such as grid_map and orb_slam3 to generate a grid map and ESDF map. The subsequent obstacle avoidance behavior directly uses the ESDF map to obtain the obstacle location information.

[0105] S5, the airborne intelligent terminal weights the cluster mission waypoints of the cluster ground station and the position and speed information of the neighboring UAVs, and after weighting, uses the cluster model to make cluster decisions in conjunction with the information of the grid map and the ESDF map to obtain the speed decision amount;

[0106] The S5 process includes:

[0107] S51, establish cluster model;

[0108] The specific process of establishing the cluster model is as follows:

[0109] Using an undirected graph Represents a drone cluster with N individuals, where For drone collection, is the interaction edge between UAVs, where R max is the maximum sensing range of the drone, d ij is the displacement vector of UAV i relative to j.

[0110] All drones use mature commercial quad-rotors and autopilots that can provide speed controller functions. The detected neighbor set of drone i is defined as Each drone i in the swarm model follows the following motion model:

[0111]

[0112] where p i and v i is the position and speed of UAV i, k is a control parameter greater than 0, v d,i It is the input of the UAV speed controller and also the speed decision variable, which satisfies the maximum speed limit of UAV i.

[0113] S51, using the ESDF map, obtaining obstacle position information and obtaining obstacle avoidance behavior as the first speed decision quantity;

[0114] Obstacle avoidance behavior means that when each individual in the rotor cluster approaches an obstacle, under the action of the obstacle avoidance rule, it obtains the distance and direction of the nearest obstacle in the environment through the perception method, and obtains the expected velocity vector required for obstacle avoidance.

[0115] Obstacle avoidance behavior is obtained as follows:

[0116] Obtain obstacle information from ESDF maps Where l is the number of obstacles obtained, α k is the confidence of obstacle information, d obs,k is the shortest vector pointing from UAV i to the surface of obstacle k, k∈l;

[0117] The position coordinates of drone i are p i (x i ,y i ,z i ), the obstacle position coordinates are p g (x g ,y g ,z g ), the relative position relationship between UAV i and the obstacle is With the current speed v of the drone i i Predict the next n time steps t step The position p i,pred (t setp ), the relative position relationship between each predicted position and the nearest obstacle point is d i,pred (t setp );

[0118] like Figure 3 As shown, the predicted obstacle avoidance parameter R is introduced safe , that is, UAV i predicts that there will be obstacles less than R in a certain step length according to the current speed. safe , then the obstacle avoidance rule needs to be executed, R reach The threshold for the drone to determine whether it has reached the target area, that is, when the distance between the drone and the target center is Less than R reach When , it is considered that the target has been reached. avoid The distance threshold for the drone to determine whether to start avoiding obstacles, that is, when the distance between the drone and the obstacle is d obs,k Less than R avoid When R avoid =R0+kv, R0 is set to 3 meters, k is 1; the following conditions are met

[0119]

[0120] The expected velocity vector generated according to the obstacle avoidance rule is:

[0121]

[0122] If the above conditions are not met, the obstacle avoidance decision amount is 0. Under the action of this rule, the UAV will simultaneously generate a decision amount to move away from the obstacle and a decision amount to move parallel to the obstacle surface for the obstacle k, driving the individual to perform avoidance and exploration maneuvers respectively.

[0123] S52, using the position and speed information of the neighboring drone in S3, obtains the distance keeping behavior as the second speed decision quantity;

[0124] The distance keeping behavior means that each individual in the rotor cluster obtains the position information of other individuals in the cluster through inter-machine communication under the action of the distance keeping rule to maintain the clustered state, calculates the relative position vector with other individuals as input, and obtains the expected velocity vector to maintain the clustered state.

[0125] The way to obtain the distance keeping behavior is:

[0126] like Figure 4 As shown, suppose there are N individuals in the cluster, and the position coordinate of drone i is p i (x i ,y i ,z i ), and the relative position relationship with the jth UAV j is d i,j =p i -p j , where the position of UAV j is obtained by visual observation, and its credibility is α nei,j , and 0≤α nei,j ≤1, according to the distance keeping rule, the expected velocity component generated by UAV j on UAV i is:

[0127]

[0128] Where R align Represents the safe distance between drones. The distance between drone j and drone i is less than the safe distance R between drones.align When , the desired velocity components moving away from each other are generated, and vice versa, the desired velocity components moving toward each other are generated;

[0129] S53, using the cluster mission waypoints in the cluster ground station, obtain the mission behavior as the speed decision quantity three;

[0130] Mission behavior means that each individual in the rotor cluster obtains the target point position information released by the ground station as input under the action of the mission behavior rules to fly to the target waypoint, and obtains the expected speed vector to fly to the target waypoint.

[0131] The task behavior is obtained as follows:

[0132] like Figure 5 As shown, the position coordinates of drone i are p i (x i ,y i ,z i ), the position coordinate of the target is p g (x g ,y g ,z g ), the relative position relationship between UAV i and the target is d goal,i =p g -p i , according to the task behavior rules, the expected velocity vector generated by UAV i is:

[0133]

[0134] in is a unit vector. When the distance d between UAV i and the target goal,i Less than R mission As the distance to the target gets closer, the expected speed gradually decreases. When the distance is greater than R mission When the desired speed is maintained at V mmax , where R mission The size of is related to the number of individuals in the cluster, N. k is the coefficient;

[0135] S54, weighting the speed decision quantity 2 and the speed decision quantity 3, with the weight being 1, to obtain the speed decision quantity 4;

[0136] S55, combining the speed decision quantity 4 with the speed decision quantity 1 to perform cluster decision making and obtain the speed decision quantity.

[0137] Therefore, the speed decision quantity is

[0138] v cmd =k1v obs +k2v align +k3v mission ;

[0139] Among them, k1, k2, and k3 represent the weight of speed decision quantity one, the weight of speed decision quantity two, and the weight of speed decision quantity three, respectively, and the sum of k2 and k3 is 1.

[0140] S6, the airborne intelligent terminal sends the speed decision amount to the UAV autopilot, and the UAV autopilot controls the rotor UAV to perform cluster flight according to the speed decision amount;

[0141] The optimization process of cluster flight in S6 is as follows:

[0142] Setting Φ obs represents the proportion of individuals in the cluster colliding with obstacles,

[0143]

[0144] where Θ(·) is the Heaviside step function, For p i (t) in symbolic form, when p i (t) When in an obstacle, Take positive value, otherwise take negative value;

[0145] Setting Φ col represents the proportion of collisions between individuals in the cluster,

[0146]

[0147] Setting Φ mis Indicates the completion of the cluster task, that is, the time when the centroid reaches the target point and the number of surviving individuals.

[0148]

[0149] According to the above metrics, a single objective optimization function is constructed.

[0150]

[0151] Use open source tools such as MATLAB's GA tool to solve the algorithm and obtain the optimal control parameters.

[0152] S7, the rotorcraft drone feeds back the flight status information to the onboard intelligent terminal.

[0153] The flight status information is fed back to the onboard intelligent terminal and participates in cluster decision-making as an important input of the cluster algorithm: the aircraft's current position information is used in the obstacle avoidance rule to calculate the two components parallel to the obstacle and perpendicular to the obstacle; in the distance keeping behavior, the aircraft's current position and estimated neighbor position information are used to calculate the decision value of the distance keeping item.

[0154] During the flight, the speed decision of the cluster decision is sent to the UAV autopilot in real time at a frequency of 20hz. The autopilot calculates the motor speed control value based on the speed decision and controls the UAV to perform cluster flight. Figure 6 Figure 7 shown.

[0155] The present invention also provides a pure visual distributed cluster flight emergence system for crowded environments, which adopts the above-mentioned pure visual distributed cluster flight emergence method for crowded environments, including:

[0156] A visual perception module, used to obtain visible light images;

[0157] Cluster ground station, used to issue mission instructions;

[0158] The cluster decision module is used to process and extract information from visible light images and task instructions to obtain speed decision quantities;

[0159] An unmanned driving module, used for receiving and executing speed decision quantities to control the movement of the unmanned aerial vehicle; and a unmanned aerial vehicle body.

[0160] Therefore, the present invention adopts the above-mentioned pure visual distributed cluster flight emergence method and system for crowded environments, in which individual drones can obtain environmental obstacle information and neighbor individual location information through visual sensors. This solves the problem that the drone cluster system is overly dependent on inter-machine communication, cannot cope with complex crowded environments, and has low individual adaptability to complex crowded environments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A purely visual distributed cluster flight emergence method for crowded environments, characterized by: The following steps are involved: S1, the rotorcraft drone obtains the observed visible light image through the onboard depth camera; S2, obtains the environment depth map and RGB map by analyzing the image; S3, analyzes the RGB image, performs Kalman filtering on the location information of the neighboring drone, and estimates the location and speed information of the neighboring drone; S4, generating a raster map and an ESDF map from the environment depth map; S5, the airborne intelligent terminal weights the cluster mission waypoints of the cluster ground station and the position and speed information of the neighboring UAVs, and after weighting, uses the cluster model to make cluster decisions in conjunction with the information of the grid map and the ESDF map to obtain the speed decision amount; S6, the airborne intelligent terminal sends the speed decision amount to the UAV autopilot, and the UAV autopilot controls the rotor UAV to perform cluster flight according to the speed decision amount; S7, the rotorcraft drone feeds back the flight status information to the onboard intelligent terminal.

2. The purely visual distributed cluster flight emergence method for crowded environments according to claim 1 is characterized by: In S2, the specific process of parsing the image is as follows: first, the friendly drone in the image is identified, the coordinates and size of the identification frame of the friendly drone are obtained, and an RGB image is generated. Then, the coordinates of the neighbor individual images identified in the RGB image are removed from the image to obtain an environmental depth map.

3. The purely visual distributed cluster flight emergence method for crowded environments according to claim 2 is characterized by: The specific steps of S3 are: S31, use yolov8 to identify individuals, and then perform data association on the identification results to distinguish different neighbor individuals in the field of view; S32, obtain the pixel coordinates (u c ,v c ), and identification box parameters (s b ,r b ), where s b is the detection box area, r b is the detection box aspect ratio; S33, project the pixel coordinates and the identification frame parameters into the image to obtain the distance information d of the observed neighboring drone. The local coordinates of the neighboring drone in the inertial coordinate system are M1M2[x,y,z,1] T =d[u,v,1] T ; Where M1 is the camera internal parameter, and M2 is the rotation matrix of the camera relative to the world system; S34, performing Kalman filtering on the observed neighboring UAV position information to estimate the position and speed information of the neighboring UAV.

4. The purely visual distributed cluster flight emergence method for crowded environments according to claim 3 is characterized by: The specific process of S4 is: use open source methods to open source the environmental depth map and generate raster maps and ESDF maps.

5. The purely visual distributed cluster flight emergence method for crowded environments according to claim 4 is characterized by: The specific steps of S5 are: S51, establish cluster model; S51, using the ESDF map, obtaining obstacle position information and obtaining obstacle avoidance behavior as the first speed decision quantity; S52, using the position and speed information of the neighboring drone in S3, obtains the distance keeping behavior as the second speed decision quantity; S53, using the cluster mission waypoints in the cluster ground station, obtain the mission behavior as the speed decision quantity three; S54, weighting the speed decision quantity 2 and the speed decision quantity 3, with the weight being 1, to obtain the speed decision quantity 4; S55, combining the speed decision quantity 4 with the speed decision quantity 1 to perform cluster decision making and obtain the speed decision quantity.

6. The purely visual distributed cluster flight emergence method for crowded environments according to claim 5 is characterized by: The specific process of establishing the cluster model is as follows: Using an undirected graph Represents a drone cluster with N individuals, where For drone collection, is the interaction edge between UAVs, where R max is the maximum sensing range of the drone, d ij is the displacement vector of UAV i relative to j, and the detected neighbor set of UAV i is defined as Each drone i in the swarm model follows the following motion model: where p i and v i is the position and speed of UAV i, k is a control parameter greater than 0, v d,i It is the input of the UAV speed controller and also the speed decision variable, which satisfies the maximum speed limit of UAV i.

7. The purely visual distributed cluster flight emergence method for crowded environments according to claim 6 is characterized by: Obstacle avoidance behavior is obtained as follows: Obtain obstacle information from ESDF maps Where l is the number of obstacles obtained, α k is the confidence of obstacle information, d obs,k is the shortest vector pointing from UAV i to the surface of obstacle k, k∈l; The position coordinates of drone i are p i (x i ,y i ,z i ), the obstacle position coordinates are p g (x g ,y g ,z g ), the relative position relationship between UAV i and the obstacle is With the current speed v of the drone i i Predict the next n time steps t step The position p i,pred (t setp ), the relative position relationship between each predicted position and the nearest obstacle point is d i,pred (t setp ); Introducing the predicted obstacle avoidance parameter R safe , UAV i predicts that there is an obstacle less than R in a certain step length according to the current speed. safe , then the obstacle avoidance rule needs to be executed, R reach is the threshold for the UAV to determine whether it has reached the target area, R avoid The distance threshold at which the drone determines that it needs to start avoiding obstacles, where R avoid =R0+kv, R0 is set to 3 meters, k is 1; The following conditions are met: The expected velocity vector generated according to the obstacle avoidance rule is:

8. The purely visual distributed cluster flight emergence method for crowded environments according to claim 7 is characterized by: The way to obtain the distance keeping behavior is: Assume there are N individuals in the cluster, and the position coordinates of drone i are p i (x i ,y i ,z i ), and the relative position relationship with the jth UAV j is d i,j =p i -p j , where the position of UAV j is obtained by visual observation, and its credibility is α nei,j , and 0≤α nei,j ≤1, according to the distance keeping rule, the expected velocity component generated by UAV j on UAV i is: Where R align Represents the safe distance between drones. The distance between drone j and drone i is less than the safe distance R between drones. align When , the desired velocity components moving away from each other are generated, and vice versa, the desired velocity components moving toward each other are generated; The task behavior is obtained as follows: The position coordinates of drone i are p i (x i ,y i ,z i ), the position coordinate of the target is p g (x g ,y g ,z g ), the relative position relationship between UAV i and the target is d goal,i =p g -p i , according to the task behavior rules, the expected velocity vector generated by UAV i is: in is a unit vector; when the distance d from UAV i to the target goal,i Less than R mission As the distance to the target gets closer, the expected speed gradually decreases. When the distance is greater than R mission When the desired speed is maintained at V m max , where R mission The size of is related to the number of individuals in the cluster, N. k is the coefficient; The speed decision quantity is: in cmd =k1v obs +k2v align +k3v mission ; Among them, k1, k2, and k3 represent the weight of speed decision quantity one, the weight of speed decision quantity two, and the weight of speed decision quantity three, respectively, and the sum of k2 and k3 is 1.

9. The purely visual distributed cluster flight emergence method for crowded environments according to claim 8 is characterized by: The optimization process of cluster flight in S6 is as follows: Setting Φ obs represents the proportion of individuals in the cluster colliding with obstacles, where Θ(·) is the Heaviside step function, For p i (t) in symbolic form, when p i (t) When in an obstacle, Take positive value, otherwise take negative value; Setting Φ col represents the proportion of collisions between individuals in the cluster, Setting Φ mis Indicates the completion of the cluster task, the time when the centroid reaches the target point, and the number of surviving individuals. According to the above metrics, a single objective optimization function is constructed. Use open source tools to solve the algorithm and obtain the optimal control parameters.

10. A purely visual distributed swarm flight emergence system for crowded environments, characterized by: The pure visual distributed cluster flight emergence method for crowded environment described in any one of claims 1 to 9 is adopted. include, A visual perception module, used to obtain visible light images; Cluster ground station, used to issue mission instructions; The cluster decision module is used to process and extract information from visible light images and task instructions to obtain speed decision quantities; An unmanned driving module, used for receiving and executing speed decision quantities to control the movement of the unmanned aerial vehicle; and a unmanned aerial vehicle body.

Images