Unmanned aerial vehicle swarm obstacle crossing method and control system, electronic equipment, and storage medium
By collecting and analyzing the array information and target area blocking data of the drone cluster, adjusting the drone array to achieve flexible obstacle crossing paths, solving the problem of insufficient flexibility in the existing technology of the drone cluster obstacle crossing method and improving safety Sex and efficiency.
Patent Information
- Application Number
- CN202210484541.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-06
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-05-06
AI Technical Summary
The existing drone group obstacle-over-blocking methods are less flexible, resulting in safety hazards and reduced flight efficiency in the coordinated operation of drone groups.
By collecting the array information of the cluster and the blocking image information of the target area, the blocking boundary data is obtained, the barrier position information is generated, and the drone array is adjusted based on the array information and barrier position information is adjusted to achieve a more flexible barrier path.
It improves the flexibility and safety of the drone swarms passing through obstacles, improves the efficiency of the drone swarm's collaborative operation, and expands the application of the drone swarm in various fields.
Smart Images

Figure CN114924595B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automatic control technology, and in particular to an obstacle surmounting method for a group of unmanned aerial vehicles, a control system, an electronic device, and a storage medium. Background Art
[0002] In recent years, drones have been used more and more widely. Drone swarm formation is an aerial flight system with a large number of members, high flexibility, low cost and high efficiency. The collaborative operation of drone swarms has many advantages over single-machine operation. For this reason, the collaborative operation of drone swarm formations is widely used in military, people's livelihood and other fields. However, in the process of drone swarm flight, certain obstacles often affect the flight operation of the drone swarm, resulting in a decrease in the flight operation efficiency of the entire swarm. How the drone swarm passes through the target area full of obstacles has become an important factor affecting the flight operation of the drone swarm. Among the related technologies, there are mainly methods such as "leader-wingman" method, behavior method, artificial potential energy method, etc., which can achieve the requirements of avoiding obstacles. However, the flexibility of the obstacle crossing method of drone swarms in the related technology is relatively low, which brings safety risks to the collaborative operation of drone swarms. Summary of the invention
[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a drone swarm obstacle crossing method and control system, electronic equipment, and storage medium, which can achieve more flexible drone swarm obstacle crossing.
[0004] A drone swarm obstacle surmounting method according to a first aspect of the present invention includes:
[0005] Collecting fleet array information and blocked image information of the target area, wherein the fleet array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array;
[0006] Acquire blocking boundary data according to the blocking image information, wherein the blocking boundary data reflects the edge contour of the obstacle in the target area;
[0007] Generate obstacle crossing position information based on the blocking boundary data, wherein the obstacle crossing position information reflects an obstacle crossing position in the target area that can cross the obstacle;
[0008] According to the swarm array information and the obstacle crossing position information, the drone array is adjusted to a swarm obstacle crossing array.
[0009] Optionally, according to some embodiments of the present invention, generating obstacle crossing position information based on the blocking boundary data includes:
[0010] Based on the blocking boundary data, construct a target map for the target area, wherein the target map reflects the spatial position relationship of each obstacle in the target area;
[0011] Obstacle crossing position information is generated according to the target map.
[0012] Optionally, according to some embodiments of the present invention, constructing a target map for the target area based on the blocking boundary data includes:
[0013] extracting target blocking features in the blocking image information according to the blocking boundary data;
[0014] Performing pose estimation on the viewing angle camera corresponding to the blocking image information to obtain group pose data, wherein the group pose data reflects the spatial position of the UAV array;
[0015] Obtaining a depth value of a feature point of each obstacle in the target area according to the target blocking feature and the group posture data;
[0016] Based on the depth values of the feature points, the spatial position relationship of the obstacles in the target area is determined to generate the target map.
[0017] Optionally, according to some embodiments of the present invention, generating obstacle crossing position information according to the target map includes:
[0018] Determining the relative position relationship between the drone array and each obstacle according to the group posture data and the target map;
[0019] Obstacle crossing position information is generated based on the relative position relationship between the drone array and the obstacles.
[0020] Optionally, according to some embodiments of the present invention, adjusting the drone array to a swarm obstacle crossing array according to the swarm array information and the obstacle crossing position information includes:
[0021] Generate an expected position of each of the drones according to the fleet array information and the obstacle crossing position information;
[0022] According to the desired position, the drone array is adjusted to the swarm obstacle crossing array.
[0023] Optionally, according to some embodiments of the present invention, generating the expected position of each of the drones according to the fleet array information and the obstacle crossing position information includes:
[0024] According to the position arrangement of each drone in the drone array and the obstacle crossing position information, each expected position corresponding to each drone is obtained through an expected position generation algorithm.
[0025] Optionally, according to some embodiments of the present invention, adjusting the drone array to the swarm obstacle crossing array according to the expected position includes:
[0026] Allocating target positions corresponding to the desired positions to each of the drones in the drone array;
[0027] Based on each of the target positions, the drone array is adjusted to the swarm obstacle crossing array.
[0028] A drone swarm control system according to a second aspect of the present invention includes:
[0029] An information module, used to collect information about the drone array and blocked image information of the target area, wherein the drone array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array;
[0030] A boundary judgment module, used for acquiring blocking boundary data according to the blocking image information, wherein the blocking boundary data reflects the edge contour of the obstacle in the target area;
[0031] A position processing module, configured to generate obstacle crossing position information based on the blocking boundary data, wherein the obstacle crossing position information reflects an obstacle crossing position in the target area that can cross the obstacle;
[0032] An array conversion module is used to adjust the drone array into a swarm obstacle-crossing array according to the swarm array information and the obstacle-crossing position information.
[0033] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the drone swarm obstacle surmounting method as described in any one of the embodiments of the first aspect of the present invention.
[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement a drone swarm obstacle surmounting method as described in any one of the embodiments of the first aspect of the present invention.
[0035] The drone swarm obstacle surmounting method and control system, electronic device, and storage medium of the embodiments of the present invention have at least the following beneficial effects:
[0036] In the drone swarm obstacle crossing method in the embodiment of the present invention, the swarm array information and the blocking image information of the target area are first collected, and the swarm array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array. Then, the blocking boundary data is obtained according to the blocking image information, and the blocking boundary data reflects the edge contour of the obstacle in the target area. Further, the obstacle crossing position information is generated based on the blocking boundary data, and the obstacle crossing position information reflects the obstacle crossing position in the target area that can cross the obstacle. Finally, according to the swarm array information and the obstacle crossing position information, the drone array is adjusted to the swarm obstacle crossing array. According to the drone swarm obstacle crossing method provided by the embodiment of the present invention, the obstacle crossing position information that can cross the obstacle is determined from the target area based on the blocking boundary data of the blocking image information, and then the human-machine array is adjusted to the swarm obstacle crossing array based on the swarm array information and the obstacle crossing position information, which provides a highly flexible obstacle crossing method for the drone swarm to cross the obstacle process, and also provides guarantee for the operation safety of the drone swarm.
[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0039] Figure 1 A flow chart of a drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0040] Figure 2 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0041] Figure 3 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0042] Figure 4 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0043] Figure 5 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0044] Figure 6 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0045] Figure 7 A flow chart of another drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0046] Figure 8 The wedge formula used to calculate the expected position point;
[0047] Fig. 9 A schematic diagram of a drone array of a drone swarm obstacle crossing method provided by an embodiment of the present invention;
[0048] Fig.10 A schematic diagram of another drone array of the drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0049] Fig.11 A schematic diagram of another drone array of the drone swarm obstacle surmounting method provided by an embodiment of the present invention;
[0050] Fig.12 A flowchart of a drone swarm obstacle surmounting method provided by a specific embodiment of the present invention;
[0051] Fig.13 A schematic diagram of a drone swarm obstacle crossing system module provided by an embodiment of the present invention;
[0052] Fig.14 Schematic diagram of an electronic device for executing the drone swarm obstacle surmounting method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0053] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0054] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0055] In the description of the present invention, it should be understood that descriptions involving orientation, such as up, down, left, right, front, back, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0056] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, the identification of specific steps below does not represent a limitation on the order of steps and the execution logic. The execution order and execution logic between the various steps should be understood and inferred with reference to the contents described in the embodiments.
[0057] In recent years, drones have been used more and more widely. Drone swarm formation is an aerial flight system with a large number of members, high flexibility, low cost and high efficiency. The collaborative operation of drone swarms has many advantages over single-machine operation. For this reason, the collaborative operation of drone swarm formations is widely used in military, people's livelihood and other fields. However, in the process of drone swarm flight, certain obstacles often affect the flight operation of the drone swarm, resulting in a decrease in the flight operation efficiency of the entire swarm. How the drone swarm passes through the target area full of obstacles has become an important factor affecting the flight operation of the drone swarm. Among the related technologies, there are mainly methods such as "leader-wingman" method, behavior method, artificial potential energy method, etc., which can achieve the requirements of avoiding obstacles. However, the flexibility of the obstacle crossing method of drone swarms in the related technology is relatively low, which brings safety risks to the collaborative operation of drone swarms.
[0058] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a drone swarm obstacle crossing method and control system, electronic equipment, and storage medium, which can achieve more flexible drone swarm obstacle crossing.
[0059] The following is a further description with reference to the accompanying drawings.
[0060] Reference Figure 1 According to the first aspect of the present invention, a drone swarm obstacle crossing method includes:
[0061] Step S101, collect the swarm array information and the blocking image information of the target area. The swarm array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array; the blocking image information is the target area image collected by the drone swarm, reflecting the blocking situation of various obstacles in the target area to the drone swarm.
[0062] According to some embodiments provided by the present invention, the collection of blocking image information can use the high-definition camera equipped with the drone to take pictures of the target area in front. In some more specific embodiments, five four-wing drones are prepared, one of which is the head drone, two are the first-level follower drones, and the remaining two are the second-level follower drones. The head drone is equipped with a 4k high-definition camera, and it is checked and confirmed that all drones are fault-free and have sufficient power to ensure the normal flight. The five drones are started and formed into a "human" shape to maintain flight, and the camera equipped with the head drone begins to shoot the environment in front. It should be understood that in some more preferred embodiments, the fleet array information reflects the real-time number of drones in the drone array at the current moment and the real-time position arrangement of each drone in the drone array. It should be emphasized that the purpose of collecting the fleet array information is to use the fleet array information to provide a reference benchmark for the position arrangement and number of the drone fleet for the adjustment of the drone array; the purpose of collecting the blocking image information of the target area is to identify the target area that the drone fleet is about to cross and clarify the blocking position of each obstacle in the target area. It should be understood that the collection of the fleet array information and the blocking image information of the target area is not limited to the specific embodiments listed above.
[0063] Step S102, obtaining blocking boundary data according to the blocking image information, where the blocking boundary data reflects the edge contour of the obstacle in the target area;
[0064] It should be noted that the blocking boundary information refers to the information in the blocking image information that reflects the edge contour of the obstacle in the target area. It should be understood that obtaining the blocking boundary information is to provide a data basis for the subsequent generation of obstacle crossing position information. After the edge contour of the obstacle in the target area is clarified, the obstacle crossing position information can be further obtained. According to some embodiments provided by the present invention, obtaining the blocking boundary data according to the blocking image information also includes: first performing color enhancement according to the blocking image information to generate equalized image data, then generating grayscale image data according to the equalized image data, and further, generating the blocking boundary data of the target area according to the grayscale image data generated according to the equalized image data. It should be understood that the generation of the blocking boundary data protects but is not limited to the above-mentioned embodiments.
[0065] Step S103, generating obstacle crossing position information based on the blocking boundary data, where the obstacle crossing position information reflects the obstacle crossing position in the target area that can cross the obstacle;
[0066] According to the embodiments provided by the present invention, after obtaining the blocking boundary data, the Savi map of the target area is constructed based on the blocking boundary data to clarify the position layout of each obstacle in the target area in space, and then obtain the obstacle crossing position of the drone group in the target area that can cross the obstacles. In some more specific embodiments provided by the present invention, a real-time three-dimensional map reconstruction and positioning (Simultaneous Localization And Mapping, SLAM) algorithm is used to estimate the current position, create a map, and analyze the feature points of the drone-taken pictures through an artificial intelligence model of image recognition (such as ResNet50, VGG, AlexNet, GooLeNet and other models) to obtain the ideal obstacle crossing position information. It should be understood that the generation of obstacle crossing position information based on blocking boundary data is not limited to the specific embodiments listed above.
[0067] Step S104, adjusting the drone array to a swarm obstacle crossing array according to the swarm array information and the obstacle crossing position information.
[0068] According to the embodiment provided by the present invention, after obtaining the obstacle crossing position information, combined with the swarm array information, the drone array can be adjusted to the swarm obstacle crossing array. It should be noted that the swarm obstacle crossing array refers to an array composed of several drones that can pass through various obstacles in the target area. For example, based on the perspective faced by the drone swarm, when each obstacle is blocked in a ring shape in the target area, the drone swarm is adjusted to pass through the holes and gaps of the ring-shaped blocking. For another example, based on the perspective faced by the drone swarm, when each obstacle is an irregular blocking surface of several levels in the target area, the drone swarm immediately updates the blocking image information every time it passes through a layer of blocking surface, thereby making adaptive adjustments to the swarm obstacle crossing array to enable the drone swarm to pass through the holes and gaps of each blocking surface accordingly, so as to achieve flexible adjustment of the drone swarm obstacle crossing of the present invention. It should be understood that adjusting the drone array to the swarm obstacle crossing array based on the swarm array information and the obstacle crossing position information is not limited to the specific embodiments listed above.
[0069] In the drone swarm obstacle crossing method in the embodiment of the present invention, the swarm array information and the blocking image information of the target area are first collected, and the swarm array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array. Then, the blocking boundary data is obtained according to the blocking image information, and the blocking boundary data reflects the edge contour of the obstacle in the target area. Further, the obstacle crossing position information is generated based on the blocking boundary data, and the obstacle crossing position information reflects the obstacle crossing position in the target area that can cross the obstacle. Finally, according to the swarm array information and the obstacle crossing position information, the drone array is adjusted to the swarm obstacle crossing array. According to the drone swarm obstacle crossing method provided by the embodiment of the present invention, the obstacle crossing position information that can cross the obstacle is determined from the target area based on the blocking boundary data of the blocking image information, and then the human-machine array is reconstructed based on the swarm array information and the obstacle crossing position information, and adjusted to the swarm obstacle crossing array, which provides a highly flexible obstacle crossing method for the drone swarm to cross obstacles in the target area, and also provides guarantee for the operation safety of the drone swarm. Improving the safety and flexibility of the drone swarm crossing the target area is more conducive to the drone swarm crossing complex target areas, improving the efficiency of the drone swarm collaborative operation, and expanding the application of drone swarms in various fields.
[0070] Reference Figure 2 According to some embodiments of the present invention, generating obstacle crossing position information based on barrier boundary data includes:
[0071] Step S201, constructing a target map for the target area based on the blocking boundary data, wherein the target map reflects the spatial position relationship of each obstacle in the target area;
[0072] Step S202: Generate obstacle crossing position information according to the target map.
[0073] It should be noted that the target map refers to a three-dimensional map that reflects the spatial position relationship of each obstacle in the target area. In some more specific embodiments provided by the present invention, a real-time three-dimensional map reconstruction and positioning (Simultaneous Localization And Mapping, SLAM) algorithm is used to estimate the current position, create a map, and analyze the feature points of the drone's pictures through an image processing model (such as ResNet50, VGG, AlexNet, GooLeNet, etc.) to obtain the ideal obstacle crossing position information.
[0074] Reference Figure 3 According to some embodiments of the present invention, a target map is constructed for a target area based on blocking boundary data, including:
[0075] Step S301, extracting target blocking features in blocking image information according to blocking boundary data;
[0076] Step S302, performing pose estimation on the viewing angle camera corresponding to the blocked image information to obtain group pose data, where the group pose data reflects the spatial position of the UAV array;
[0077] Step S303, obtaining the depth value of the feature points of each obstacle in the target area according to the target blocking features and the group posture data;
[0078] Step S304: Based on the depth values of the feature points, determine the spatial position relationship of the obstacles in the target area and generate a target map.
[0079] It should be noted that the target blocking feature refers to the blocking feature reflected by the obstacle in the blocking boundary data. It should be understood that the blocking feature varies with the edge contour of the obstacle. The swarm posture data refers to the data detected in real time that reflects the spatial position and posture of each drone in the swarm. The feature point depth value is an attribute that reflects the spatial position of the obstacle in the target area. Based on the position difference of the obstacle from the drone swarm image acquisition device, the feature point depth value obtained according to the target blocking feature and the swarm posture data will also be different. In some embodiments of the present invention, it is based on the above reasons that the spatial position relationship of each obstacle in the target area can be determined according to the feature point depth value, thereby generating a target map.
[0080] According to some more specific embodiments provided by the present invention, the SLAM algorithm is generally composed of three parts: image acquisition, front-end process, and back-end optimization. In some more specific embodiments, it includes feature point extraction, camera pose estimation, point cloud map construction, key frame insertion, loop detection, and optimization methods for local, loop, and global optimization of the back-end. The important operations are explained in detail below:
[0081] (1) Feature point extraction
[0082] Feature point extraction is the first step of SLAM algorithm to utilize image information and is the basis of all subsequent work. This paper proposes to combine neural network with SLAM algorithm and use image processing model to extract feature points from pictures taken by drone. The main ideas of this method are as follows:
[0083] In some embodiments, the preprocessed image is input into the ResNet50 network. ResNet is composed of multiple residual networks stacked, each residual network includes a forward neural network and a shortcut. The main branch is composed of two 1x1 convolutional layers plus a single 3x3 convolutional layer, and the ReLU activation function is used to increase the nonlinearity of the network. The connecting line on the right is the shortcut branch. The problem of model degradation is cleverly resolved due to the existence of the shortcut branch. In some more specific embodiments provided by the present invention, some embodiments of the present invention make some adjustments based on the original network structure of ResNet, that is, postpone downsampling, the downsampling module of the original model, Path A (path A) first undergoes 1x1 convolution to complete the contraction of the channel, and the step size is 2 to achieve downsampling; secondly, it undergoes 3x3 convolution, the number of channels remains unchanged, mainly used for feature extraction; finally, it undergoes 1x1 convolution to complete the expansion of the number of channels. Among them, the first convolution is used for downsampling. The convolution with kernel size 1x1 and stride 2 will cause 3 / 4 information loss. Taking the 6x6 feature map as an example, only part of the information can be passed to the next layer after downsampling, and the other part does not participate in the convolution operation. Now postpone the downsampling and put the downsampling process on the 3x3 convolution. Since the convolution kernel width is greater than the stride, the convolution kernel can traverse all the information on the input feature map during the movement, and there is some overlap.
[0084] Through the above convolution operation, the feature point information of the image is obtained, and further, the obstacle position information coordinates are obtained by analyzing the obtained feature point information.
[0085] (2) Pose and spatial point position estimation
[0086] Since some embodiments of the present invention adopt a monocular vision algorithm, a 2D-2D (plane) pose estimation method is used to calculate the camera's pose, and the coordinates of the spatial point are obtained by calculating the depth value of the feature point in the scene. Since the monocular camera cannot directly obtain the depth value, a triangulation method is used to solve it.
[0087] (3) Monocular camera initialization
[0088] The monocular camera cannot obtain the absolute depth value of the point, so it is necessary to manually set a value and then use triangulation to find the relative value of the depth. However, different scenes may appear during the movement of the camera, so only setting an initial value will have a bad impact on subsequent operations, resulting in the failure of SLAM. Some embodiments of the present invention adopt an adaptive initialization method. For different scenes, different types of matrices are decomposed. First, a pair of input grayscale images are read, and a better image (more than 100 groups of matching points) is selected. The matching points are divided into a group of 8 random points, and a total of 200 groups are divided. The basic matrix and homography matrix corresponding to the 200 groups of data are further calculated respectively. Each time a group of data is calculated, the corresponding matrix is calculated according to the scoring model. The score is calculated, and finally the result with the highest score is selected from 200 iterations. The corresponding model is determined by using the proportional relationship as the matrix for initializing the decomposition. Through this method, the problem that traditional initialization can only adapt to one scene or even requires manual intervention is cleverly solved, making the initialization of monocular vision SLAM more accurate.
[0089] (4) Key frame selection strategy
[0090] Because image content is the basis of visual SLAM, determining key frames based on image content is more in line with the principles of visual SLAM.
[0091] The frame rate of an ordinary camera is generally 30fps, which means that the camera takes 30 pictures continuously in one second to form a video stream, that is, 30 frames; for a camera moving at a low speed, it will hardly run out of the area outside the line of sight when starting to shoot in one second. Therefore, some embodiments of the present invention adopt a strategy for screening key frames. The principle is that when the current frame exceeds the previous key frame by more than 20 frames, any current operation is stopped, and it is judged whether the current frame is a key frame. For the current frame that does not exceed 20 frames, it is judged only when the system is idle. The condition for judging that the current frame is a key frame is that the number of feature points extracted from the current frame is greater than 50 and the number of matching feature points in the previous key frame is less than 90%. These two conditions ensure the richness and effectiveness of the image frame content. This method adopted by some embodiments of the present invention not only avoids the situation of excessive calculation caused by too many key frames, but also ensures that there will be no tracking failure in the SLAM process.
[0092] In some more specific embodiments provided by the present invention, some embodiments of the present invention aim to improve the safety of drone swarms crossing a target area. Some embodiments of the present invention use the R-SLAM algorithm to better calculate the obstacle crossing position information of the drone swarm crossing the target area, reduce the probability of the drone swarm colliding with obstacles, improve the ability of the drone swarm to avoid obstacles, and improve the safety of the drone swarm crossing the target area.
[0093] Reference Figure 4According to some embodiments of the present invention, generating obstacle crossing position information according to a target map includes:
[0094] Step S401, determining the relative position relationship between the drone array and each obstacle based on the group posture data and the target map;
[0095] Step S402, generating obstacle crossing position information based on the relative position relationship between the drone array and each obstacle.
[0096] It should be noted that after the relative position relationship between the drone array and each obstacle is determined according to the target map, based on the relative position relationship between the drone array and each obstacle, the obstacle crossing position of the drone swarm in the target area can be obtained through spatial position judgment, so as to provide a basis for the subsequent adjustment of the drone array that drives the drone swarm to the swarm obstacle crossing array.
[0097] Reference Figure 5 According to some embodiments of the present invention, adjusting the drone array to a drone swarm obstacle crossing array according to the drone swarm array information and the obstacle crossing position information includes:
[0098] Step S501, generating the expected position of each UAV according to the fleet array information and obstacle crossing position information;
[0099] Step S502: adjusting the drone array to an obstacle-crossing array according to the desired position.
[0100] Some embodiments of the present invention can further enhance the flexibility of drone swarms in crossing obstacles. According to some embodiments provided by the present invention, an expected position generation algorithm can be used for all members (members refer to drones) in the drone swarm formation to obtain a set of expected position points for all members, with the purpose of generating the coordinates of the positions of each drone after reconstruction. In addition, some embodiments of the present invention can use drone formation reconstruction algorithms such as target position selection algorithms to select different formations to pass through obstacles according to the different characteristics of different obstacles when crossing the target area.
[0101] Reference Figure 6 According to some embodiments of the present invention, generating the expected position of each drone according to the fleet array information and the obstacle crossing position information includes:
[0102] Step S601, according to the position arrangement and obstacle crossing position information of each drone in the drone array;
[0103] Step S602: Obtain each expected position corresponding to each drone through an expected position generation algorithm.
[0104] Reference Figure 7According to some embodiments of the present invention, adjusting the drone array to a swarm obstacle crossing array according to a desired position includes:
[0105] 701, assigning each target position corresponding to the expected position to each drone in the drone array;
[0106] At 702, based on each target position, the drone array is adjusted to a swarm obstacle crossing array.
[0107] In some more specific embodiments provided by the present invention, the expected position generation algorithm: assuming that messages can be transmitted smoothly between drones, in general, the communication topology of the formation is an arbitrary directed graph, and at least one formation member can directly receive information about formation changes, and other members can indirectly obtain the above information by communicating with members who have obtained information related to formation changes, that is, the communication topology at this time has a directed spanning tree with the above-mentioned designated device as the root node. After receiving the above information, each machine solves and calculates according to the geometric constraints of the target formation shape to obtain all the expected position points. Taking the wedge formula as an example, Figure 8 As shown. Among them, Figure 8 The right side of the equal sign (X k ,Y k ,Z k )(k=1,2,...,n) is the initial position point loaded on the onboard computer of the formation team before takeoff. In the XOY plane with the origin of the ground coordinate system as the formation center, it is solved according to the geometric constraints of the target formation shape and flies in the direction indicated by the Y axis; θ is the angle between the velocity vector ζF and the OXY plane; θ is the angle between the projection of the velocity vector ζF to the XOY plane and the positive semi-axis of the Y axis. According to the above formula, the formation center can be transformed to the agreed location ξ F (X F ,Y F ,Z F ), while (X k ,Y k ,Z k ) to Thus, a set Ω of all desired position points in the virtual body is formed.
[0108] The target position selection algorithm is used for all members of the UAV swarm formation, and a coordinate position generated by the expected position algorithm is assigned to each UAV in the formation to complete the one-line formation reconstruction. The formation reconstruction process is as follows: Fig. 9 As shown, it should be understood that Fig. 9 The obstacles shown in the figure are annular obstacles or other obstacles with holes or gaps. The drone swarm can pass through the holes or gaps of the annular obstacles after adjusting to a swarm obstacle crossing array.
[0109] According to some preferred embodiments of the present invention, a target position selection algorithm is used to select different formations to pass through obstacles according to the different characteristics of different obstacles when crossing the target area.
[0110] In some specific embodiments provided by the present invention, the target position selection algorithm: This algorithm mainly selects a desired position point from the desired position for each drone in the formation. Assume that there are n desired position points in the virtual body corresponding to the target formation to form the desired position set Ω, and N = n. The communication range of the drone is the communication distance d with its position as the center. a In the circular area formed by the radius, each drone can only communicate with other members within its communication range to obtain the other party’s identification number, location and expected location point, forming a neighbor set, location set and corresponding tracking object set.
[0111] like Fig.10 As shown, assuming that the neighbor set of drone i in the current state is N′ t ={i3,i4,i5}, position set N′ t* = {ξi3,ξi4,ξi5} and tracking object set T′ t ={⑤,③,③}, which means: UAV i obtains that UAV i3 is at position ξi3 and is flying to the desired position point ⑤, that is, it is tracking the position point ⑤ in the rigid body. UAVs i4 and i5 are at positions ξi4 and ξi5 respectively and are jointly tracking the desired position point ③. Similarly, UAVs i3, i4 and i5 also have their own neighbor sets, position sets and tracking object sets. Furthermore, UAV i obtains their neighbor sets, position sets and tracking object sets by communicating with neighbor members i3, i4 and i5, and uses the information combination method to generate the total neighbor set N t 、Location set L t and the total set of tracked objects T t , which is the local information of the surrounding environment obtained by the formation members through local perception. On this basis, the target position selection algorithm includes the following operations:
[0112] (1) “Approach” operation
[0113] First, let UAV i (i = 1, 2, ..., m) search for the nearest desired position point from Ω; then, the UAV starts to track the desired position point, and at the same time, it relies on the local sensor system to obtain its own neighbor set, position set and tracking object set as well as the neighbor set, position set and tracking object set of its neighbors, and combines the above information to obtain the total neighbor set N t 、Location set L t and the total set of tracked objects T t .
[0114] (2) “Virtual allocation” operation
[0115] First, the drone uses its own T t The expected location points stored in are used as indexes to classify the drones in N. For example, if the neighbor set obtained by drone i at time t is N t ={i,i1,i2,i3,i4,i5}, the tracking object set is T t {⑥, ⑥, ⑥, ⑤, ③, ③}, the classification result is: location ⑥ has 3 drones {i, i1, i2} that are tracking it, location ⑤ has 1 drone {i3} that is tracking it, location ③ has 2 drones {i4, i5} that are tracking it, and the other T t If no formation position is stored in the target position, it means that no drone is flying to the target position. Fig.11 shown.
[0116] (3) “Adaptive Adjustment” Operation
[0117] Step 1: UAV i flies towards the desired position point generated by the “virtual matching” operation, and at the same time uses local information N t and T t Determine whether there are other drones at the same expected location as you. If "no", continue to move towards the original expected location; if "yes", go to step 2.
[0118] Step 2: UAV i combines local information L t Determine whether the distance between the drone and the current desired location is closer than the distance between other drones with the same target and the desired location. If yes, continue to move toward the original desired location; if no, proceed to step 3.
[0119] Step 3: Drone i uses its local information N t and T t and L t Make judgments about the surrounding situation: Since Ω is the known information of each member of the formation, and T t The expected position point in is N t The drone in the occupancy, therefore, it will think that Ω-T t (belongs to the set Ω but not to the set T t The desired position in the set of all drones in Ω-T has not been occupied by other drones. t (Including 3 cases: the expected position point in Ω that is discovered by drone i and its neighboring drones and is not tracked by other drones; the expected position point in Ω that is not discovered by drone i and its neighboring drones and is not tracked by other drones; the expected position point in Ω that is not discovered by drone i and its neighboring drones and is tracked by other drones. The expected position point of drone i must belong to Ω-Tt Since N = n, when drone i enters step 3, it means Ω-T t Not empty, then drone i will t The positions in the traversal are visited in order from near to far to find their desired positions. Although the local information obtained by UAV i will change with the movement of the formation members, the above conclusion still holds. In order to prevent UAV i from repeatedly visiting the same position in a short period of time, the set M is used to store the position points just visited. The validity period of each position point is set to Δt, that is, it is prohibited to visit again during this time period and will not be released until the validity period expires. This will prompt UAV i to Ω-T t Perform liquidity access to the position in the algorithm, repeat steps 1 to 3, and you will find your desired position, and the algorithm ends.
[0120] According to some embodiments provided by the present invention, acquiring blocking boundary data according to blocking image information further includes: performing color enhancement according to the blocking image information to generate equalized image data.
[0121] In some more specific embodiments of the present invention, the step of performing color enhancement according to the blocking image information to generate equalized image data includes applying a color image enhancement algorithm based on histogram segmentation to the blocking image information collected by the drone to obtain an image with better visual effect after color enhancement. The purpose is to improve the visual effect of the image and convert the image taken by the drone into a form more suitable for machine processing and analysis. It should be noted that image enhancement is an important preprocessing process for image analysis and processing, and is also a means of using various mathematical methods and transformation means to improve the contrast and clarity of the image to highlight the main area. There are mainly the following ways to enhance color images using histogram equalization: (1) first divide the color image into three sub-images of RGB, and then process and merge the three sub-images separately; (2) directly process the image and calculate the joint probability density of the three RGB sub-images, and then perform histogram equalization based on this density; (3) convert the image from RGB color space to HSI color space and then enhance the brightness component.
[0122] As an example, the step of performing color enhancement according to the blocking image information to generate the equalized image data includes: generating histogram image data according to the blocking image information, and generating the equalized image data according to the histogram image data.
[0123] In some more specific embodiments provided by the present invention, some embodiments of the present invention propose a new color image histogram equalization method based on the above-mentioned enhancement algorithm. The method divides the grayscale histogram of each RGB component sub-image according to the image grayscale mean and the grayscale histogram equal area principle, first generates 4 sub-grayscale histograms, and then performs equalization processing on the generated 4 sub-grayscale histograms, and then calculates the proportion of the grayscale of each RGB component sub-image to the total grayscale of the original color image and merges the RGB component sub-images according to this proportion. The method not only has a low computational complexity for color image enhancement processing, but also has a good visual effect in color after the merged image. Histogram equalization is to transform the grayscale histogram distribution of a given image into an approximately uniform histogram distribution, so that the probability density of the pixel grayscale of the output image is uniformly distributed. Since the histogram equalization algorithm is an algorithm for processing grayscale images, each component is processed separately when it is used to process color images. The algorithm proposed in some embodiments of the present invention is to perform equalization processing according to the histograms in each interval after histogram segmentation.
[0124] According to some embodiments provided by the present invention, after performing color enhancement according to the blocking image information to generate equalized image data, the method further includes: generating grayscale image data according to the equalized image data.
[0125] In some embodiments of the present invention, the equalized image data is subjected to noise reduction processing to generate noise-reduced image data, and further, the noise-reduced image data is subjected to grayscale processing to generate the grayscale image data.
[0126] In some embodiments provided by the present invention, the weighted average method is used to grayscale the color enhanced image to obtain a grayscale image of the target area. It should be noted that the weighted average grayscale processing: the process of converting a color image into a grayscale image is called the grayscale processing of the image. Some embodiments of the present invention use the weighted average method to grayscale the image. Among them, the weighted average method refers to the weighted average of the three components according to different weights based on factors such as importance. Since the human eye is most sensitive to green and least sensitive to blue, the weighted average of the RGB three components is performed according to the following formula to obtain a more reasonable grayscale image, as shown in the following formula:
[0127] Gray(i,j)=0.299*R(i,j)+0.578*G(i,j)+0.114*B(i,j)
[0128] The weighted average grayscale processing method can better retain the details of the image, effectively avoid the problem of pixel overflow, and obtain a grayscale image with good effect, which can better complete the grayscale processing of the image.
[0129] According to some embodiments provided by the present invention, after generating grayscale image data according to the equalized image data, the method further includes: generating blocking boundary data of the target area according to the grayscale image data.
[0130] In some more specific embodiments provided by the present invention, a three-dimensional block matching algorithm (Block-Matching 3D, BM3D) denoising method is used to reduce the noise of the image. The purpose is to improve the efficiency of image processing, suppress and eliminate noise, improve the quality of the image, and facilitate the subsequent preprocessing of the image. In some more specific embodiments provided by the present invention, it can be the following 9 steps, steps 1-2 are the equipment preparation and image acquisition part, steps 3-5 are the image preprocessing part, step 6 is the obstacle position information calculation part, steps 7-8 are the formation reconstruction part, and step 9 is the obstacle crossing part. Fig.12 shown.
[0131] Step 1: Prepare five quadcopter drones, one of which is the lead drone, two are first-level follower drones, and the remaining two are second-level follower drones. Install a 4k high-definition camera on the lead drone and check to make sure that all drones are fault-free and have sufficient power to ensure normal flight.
[0132] Step 2: Start the five drones and form a formation to maintain flight in a V-shape. The camera on the lead drone starts shooting the environment ahead.
[0133] Step 3: Use the histogram segmentation color image enhancement algorithm on the photos taken by the drone to obtain images with more obvious color effects and easier to process by computer.
[0134] Step 4: Perform noise reduction and grayscale processing on the color enhanced image to obtain a decolorized image, thereby improving the image processing efficiency.
[0135] Step 5: Use the Canny operator to detect and extract the edge of the image processed in step 4 to obtain the edge contour of the obstacle in front.
[0136] Step 6: Input the edge-extracted image into the R-SLAM algorithm, calculate and update the current position and map feature point information, use the ResNet50 network to obtain the obstacle crossing position information and update the coordinates in real time, and send out a formation reconstruction signal to prepare for formation reconstruction.
[0137] Step 7: Use the expected position generation algorithm for all members in the UAV formation to generate the coordinates of the reconstructed position of each UAV for each UAV.
[0138] Step 8: Use the target position selection algorithm for all members in the drone swarm formation, assign a coordinate position generated by the expected position algorithm to each drone, and reconstruct the line-shaped formation.
[0139] Step 9: Maintain the reconstructed straight-line formation to pass through the target area. After passing, restore the original formation status to continue the flight mission.
[0140] In some more specific embodiments provided by the present invention, some embodiments of the present invention equip the head machine with a 4K high-definition camera to capture the obstacle environment ahead, grayscale the image to obtain the corresponding grayscale image, use the noise reduction algorithm to reduce the noise of the image, and improve the image quality. Then, the edge of the image is extracted by the Canny operator to prepare for subsequent work. After the image preprocessing is completed, the image is input into the SLAM algorithm and combined with the ResNet50 network to perform feature analysis, feature processing and other operations to obtain the ideal obstacle crossing position information of the drone group, and at the same time, the drone formation is reconstructed from the original "human" formation to the "one" formation that is easy to pass through the target area, so as to realize the drone group's crossing of obstacles.
[0141] Reference Fig.13 According to the second aspect of the present invention, the drone swarm control system 1300 includes:
[0142] The information module 1301 is used to collect the array information of the drone group and the blocked image information of the target area. The array information of the drone group includes the number of drones in the drone array and the position arrangement of each drone in the drone array;
[0143] The boundary judgment module 1302 is used to obtain blocking boundary data according to the blocking image information, and the blocking boundary data reflects the edge contour of the obstacle in the target area;
[0144] The position processing module 1303 is used to generate obstacle crossing position information based on the blocking boundary data, where the obstacle crossing position information reflects the obstacle crossing position in the target area that can cross the obstacle;
[0145] The array conversion module 1304 is used to adjust the drone array to a swarm obstacle crossing array according to the swarm array information and obstacle crossing position information.
[0146] Fig.14 The electronic device 1400 provided by an embodiment of the present invention is shown. The electronic device 1400 includes: a processor 1401, a memory 1402, and a computer program stored in the memory 1402 and executable on the processor 1401, and the computer program is used to execute the above-mentioned drone swarm obstacle surmounting method when it is executed.
[0147] The processor 1401 and the memory 1402 may be connected via a bus or other means.
[0148] The memory 1402 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs, such as the drone swarm obstacle surmounting method described in the embodiment of the present invention. The processor 1401 implements the above-mentioned drone swarm obstacle surmounting method by running the non-transitory software programs and instructions stored in the memory 1402.
[0149] The memory 1402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function. The data storage area may store and execute the above-mentioned drone swarm obstacle crossing method. In addition, the memory 1402 may include a high-speed random access memory 1402, and may also include a non-volatile memory 1402, such as at least one storage device storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1402 may optionally include a memory 1402 remotely disposed relative to the processor 1401, and these remote memories 1402 may be connected to the electronic device 1400 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0150] The non-transient software program and instructions required to implement the above-mentioned drone swarm obstacle surmounting method are stored in the memory 1402. When executed by one or more processors 1401, the above-mentioned drone swarm obstacle surmounting method is executed, for example, Figure 1 Steps S101 to S104 of the method, Figure 2 Steps S201 to S202 of the method, Figure 3 Steps S301 to S304 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5 Steps S501 to S502 of the method, Figure 6 Steps S601 to S602 of the method, Figure 7 Steps S701 to S702 of the method and Fig.12 The steps shown.
[0151] An embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned drone swarm obstacle surmounting method.
[0152] In one embodiment, the computer readable storage medium stores computer executable instructions, which are executed by one or more control processors, for example, Figure 1 Steps S101 to S104 of the method, Figure 2 Steps S201 to S202 of the method, Figure 3 Steps S301 to S304 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5 Steps S501 to S502 of the method, Figure 6 Steps S601 to S602 of the method, Figure 7 Steps S701 to S702 of the method and Fig.12 The steps shown.
[0153] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0154] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. It should also be understood that the various implementations provided in the embodiments of the present invention can be arbitrarily combined to achieve different technical effects.
[0155] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A method for drone swarm obstacle crossing, It is characterized in that include: Collecting fleet array information and blocked image information of the target area, wherein the fleet array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array; Acquire blocking boundary data according to the blocking image information, wherein the blocking boundary data reflects the edge contour of the obstacle in the target area; extracting target blocking features in the blocking image information according to the blocking boundary data; Performing pose estimation on the viewing angle camera corresponding to the blocking image information to obtain group pose data, wherein the group pose data reflects the spatial position of the UAV array; Obtaining a depth value of a feature point of each obstacle in the target area according to the target blocking feature and the group posture data; Based on the depth values of the feature points, determining the spatial positional relationship of the obstacles in the target area, and generating a target map, wherein the target map reflects the spatial positional relationship of the obstacles in the target area; Determining the relative position relationship between the drone array and each obstacle according to the group posture data and the target map; Based on the relative position relationship between the drone array and the obstacles, generating obstacle crossing position information, the obstacle crossing position information reflecting the obstacle crossing position in the target area that can cross the obstacles; According to the swarm array information and the obstacle crossing position information, the drone array is adjusted to a swarm obstacle crossing array.
2. The method according to claim 1, It is characterized in that The step of adjusting the drone array to a drone array according to the drone array information and the obstacle crossing position information includes: Generate an expected position of each of the drones according to the fleet array information and the obstacle crossing position information; According to the desired position, the drone array is adjusted to the swarm obstacle crossing array.
3. The method according to claim 2, It is characterized in that Generating the expected position of each of the drones according to the fleet array information and the obstacle crossing position information includes: According to the position arrangement of each drone in the drone array and the obstacle crossing position information, each expected position corresponding to each drone is obtained through an expected position generation algorithm.
4. The method according to claim 3, It is characterized in that The step of adjusting the drone array to the obstacle-crossing array according to the expected position includes: Allocating target positions corresponding to the desired positions to each of the drones in the drone array; Based on each of the target positions, the drone array is adjusted to the swarm obstacle crossing array.
5. A drone swarm control system, It is characterized in that include: An information module, used to collect information about the drone array and blocked image information of the target area, wherein the drone array information includes the number of drones in the drone array and the position arrangement of each drone in the drone array; A boundary judgment module, used for acquiring blocking boundary data according to the blocking image information, wherein the blocking boundary data reflects the edge contour of the obstacle in the target area; A position processing module, used for extracting target blocking features in the blocking image information according to the blocking boundary data; Performing pose estimation on the viewing angle camera corresponding to the blocking image information to obtain group pose data, wherein the group pose data reflects the spatial position of the drone array; obtaining the depth value of the feature point of each obstacle in the target area according to the target blocking feature and the group pose data; Based on the depth values of the feature points, determining the spatial positional relationship of the obstacles in the target area, and generating a target map, wherein the target map reflects the spatial positional relationship of the obstacles in the target area; Determining the relative position relationship between the drone array and each obstacle according to the group posture data and the target map; Based on the relative position relationship between the drone array and the obstacles, generating obstacle crossing position information, the obstacle crossing position information reflecting the obstacle crossing position in the target area that can cross the obstacles; An array conversion module is used to adjust the drone array into a swarm obstacle-crossing array according to the swarm array information and the obstacle-crossing position information.
6. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the drone swarm obstacle surmounting method as described in any one of claims 1 to 4 is implemented.
7. A computer-readable storage medium, It is characterized in that The storage medium stores a program, and the program is executed by a processor to implement the drone swarm obstacle surmounting method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Unmanned aerial vehicle (UAV) obstacle avoidance method and system based on binocular vision and optical flow fusion
CN106681353A
Obstacle-crossing identification system and method
CN107139666A
Cluster unmanned aerial vehicle formation method based on model prediction control framework
CN110162086A
Close formation flight and anti-collision control method for fixed-wing unmanned aerial vehicles
CN113238583A
Unmanned aerial vehicle cluster obstacle avoidance method
CN114428516A
