Unmanned aerial vehicle dynamic obstacle avoidance and path planning method and device based on multi-modal image recognition

Abstract

The invention discloses an unmanned aerial vehicle dynamic obstacle avoidance and path planning method and device based on multi-modal image recognition, and relates to the technical field of intelligent unmanned systems and path planning. The method comprises the steps that an unmanned aerial vehicle cluster collects sensing data and obtains an obstacle instance list, and according to the obstacle instance list, global optimal flight path parameters and local environment map data layers of a geometric layer and a semantic layer are calculated; calculating an initial flight path in combination with the candidate path and a local map; a subgroup is finely collected to obtain the volume, detailed category and dynamic attribute of an obstacle, newly added or changed is detected, a local optimal path is recalculated, an updating instruction only containing increment is generated, and dynamic avoiding and tracking are achieved; and obtaining a full-scene perception result. According to the method, the data acquisition efficiency of the unmanned aerial vehicle group in a complex environment can be remarkably improved by utilizing an incremental updating mechanism, the flight safety of the unmanned aerial vehicles can be ensured by the mechanism, and immediate avoidance and continuous tracking of static and dynamic obstacles are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent unmanned systems and path planning, in particular to a method and device for dynamic obstacle avoidance and path planning of unmanned aerial vehicles based on multi-modal image recognition. BACKGROUND

[0002] As a highly efficient and flexible aerial platform, unmanned aerial vehicles (UAVs) have been widely used in various fields, including infrastructure inspection under complex terrain (such as power lines, pipeline monitoring), emergency search and rescue tasks (such as locating personnel at disaster sites), high-precision mapping (such as terrain modeling), and logistics distribution (such as transporting goods in mountainous areas). In these tasks, UAVs often need to fly in dynamic and unstructured environments, such as forests, urban building complexes, or disaster sites, which contain a large number of static and dynamic obstacles.

[0003] Currently, the autonomous flight of UAVs in complex environments mainly relies on pre-set paths or simple sensors (such as GPS or ultrasonic waves) for obstacle avoidance, but these methods have limitations. For example, pre-set paths cannot adapt to real-time environmental changes, which may lead to collision risks; and traditional sensors lack accuracy in identifying complex obstacles (such as transparent objects or dynamic targets), making it difficult to achieve reliable obstacle avoidance. Therefore, how to achieve high-precision environmental perception, real-time obstacle identification, and dynamic path planning has become a key issue to improve the reliability and adaptability of UAVs in complex tasks. The present technical solution combines high-precision perception and high-reliability obstacle avoidance algorithms to improve the ability of UAVs to collect data in complex environments such as inspection, search and rescue, mapping, and logistics. SUMMARY

[0004] To solve the technical problem that existing technologies lack high-precision perception and high-reliability obstacle avoidance capabilities in implementing UAV tasks such as inspection, search and rescue, mapping, and logistics in complex terrain, the present application provides a method and device for dynamic obstacle avoidance and path planning of unmanned aerial vehicles based on multi-modal image recognition. The technical solution is as follows:

[0005] On the one hand, a method for dynamic obstacle avoidance and path planning of unmanned aerial vehicles based on multi-modal image recognition is provided, which is implemented by a device for dynamic obstacle avoidance and path planning of unmanned aerial vehicles based on multi-modal image recognition. The method includes: S1: Generate obstacle perception task instructions and issue to the UAV cluster, obtain obstacle perception data, the UAV cluster includes an obstacle perception data collection sub-cluster and a fine data collection sub-cluster, the obstacle perception data collection sub-cluster is used to collect RGB optical images and infrared thermal imaging, the fine data collection sub-cluster is used to collect RGB optical images, infrared thermal imaging, depth maps, and laser radar point cloud data, the obstacle perception data collection sub-cluster is also equipped with a calibration board module and a geometric registration module, the geometric registration module unifies the obstacle perception data to the same world coordinate system based on the calibration board, the fine data collection sub-cluster is also equipped with a synchronous collection scheduler, the synchronous collection scheduler is used to ensure time alignment of multi-modal data; S2: Use image segmentation algorithms on the obstacle perception data respectively or jointly to identify static obstacles in the flight path, and output an obstacle instance list, calculate N candidate paths and global optimal flight path parameters based on the obstacle instance list, the obstacle instance list includes position information and class labels; S3: Generate a local environment map data layer based on the obstacle instance list, the local environment map data layer includes a geometry layer and a semantic layer; S4: Based on the N candidate paths and the local environment map data layer, calculate the initial flight path from the starting point to the target point, and encapsulate the path information as pre-planning instructions; S5: The fine data collection sub-cluster executes the task according to the pre-planning instructions, and performs fine-grained data collection on the obstacle instance list, the fine-grained data includes obstacle volume size, detailed class labels and dynamic attributes, when detecting new or changed obstacles, an updated obstacle instance list is obtained, and a local optimal path is recalculated based on the updated obstacle instance list to achieve dynamic obstacle avoidance and tracking, and an update instruction is obtained, the update instruction only carries incremental information; S6: The fine data collection sub-cluster obtains the final fine-grained data according to the update instruction, merges the local environment map data layer and the final fine-grained data, and obtains the full-scene perception result, the final fine-grained data includes detailed motion information of dynamic obstacles.

[0006] Preferably, the S1 generates obstacle perception task instructions and issues them to the UAV cluster, obtaining obstacle perception data, the UAV cluster including an obstacle perception data acquisition sub-cluster and a fine data acquisition sub-cluster, the obstacle perception data acquisition sub-cluster being used to acquire RGB optical images and infrared thermal imaging, and the fine data acquisition sub-cluster being used to acquire RGB optical images, infrared thermal imaging, depth maps, and laser radar point cloud data, the obstacle perception data acquisition sub-cluster further being provided with a calibration board module and a geometric registration module, the geometric registration module being based on the calibration board to unify the obstacle perception data to the same world coordinate system, and the fine data acquisition sub-cluster further being provided with a synchronous acquisition scheduler, the synchronous acquisition scheduler being used to ensure time alignment of multi-modal data, including: S11: generating a 3D flight route grid covering the entire target operation area using a digital elevation model and target operation area boundary information; S12: generating obstacle perception task instructions based on the 3D flight route grid and flight parameters, the obstacle perception task instructions including a target grid coverage series required to be covered by each UAV flight path in the obstacle perception data acquisition sub-cluster, the target grid coverage series including a grid center point sequence; S13: spatially calibrating the acquired RGB optical images, infrared thermal imaging, and laser radar point cloud through a calibration board installation module to form geometric registration parameters including camera intrinsic parameters, extrinsic parameters, and laser radar attitude information; S14: based on the geometric registration parameters, mapping the obstacle perception data obtained by different UAVs to the same world coordinate system to obtain obstacle perception data.

[0007] Preferably, the S2 separately or jointly uses an image segmentation algorithm on the obstacle perception data to identify static obstacles in the flight path and output an obstacle instance list, calculates N candidate paths and global optimal flight path parameters based on the obstacle instance list, and the obstacle instance list includes position information and category labels, including: S21: setting the total number of paths, the total number of air route key point coordinates, the maximum number of iterations, the scaling factor, and the crossover probability, and based on the geometric position information of each static obstacle in the obstacle instance list and a preset safety distance, first eliminating all volumes intersecting with obstacles or violating the safety distance in the entire operation space to form a feasible subspace, and randomly generating Np groups of candidate path parameter vectors in the feasible subspace, each vector containing three-dimensional coordinates of all key waypoints and satisfying all geometric constraints; S22: further updating the Np groups of candidate path parameter vectors using a differential evolution mutation and adaptive crossover strategy to obtain updated candidate path parameter vectors; S23: Geometric constraint checking is performed, and if the corresponding waypoint coordinates conflict with any obstacle instance, the candidate is marked as infeasible in this dimension or resampled directly; S24: After updating all candidates, the next iteration is entered until the maximum number of iterations is reached or the convergence criterion is met, obtaining N candidate paths and the global optimal flight path parameters.

[0008] Preferably, S3 generates a local environment map data layer based on the obstacle instance list, which includes a geometric layer and a semantic layer, including: S31: Based on the obstacle instance list, a geometric layer for calculating optimal path search is generated, which contains obstacle shape, position, and waypoint coordinates; S32: Based on the global optimal flight path parameters and the geometric layer, a semantic layer for calculating optimal path search is constructed; S33: Merge the geometric layer and the semantic layer to generate a local environment map data layer, which is used to support optimal path search.

[0009] Preferably, S4 calculates the initial flight path from the starting point to the target point based on the N candidate paths and the local environment map data layer, and encapsulates the path information as pre-planning instructions, including: S41: Based on the maximum number of iterations, N candidate paths and local environment map data layer, an initial population X containing N candidate paths is generated; S42: Calculate the fitness function of each candidate path in the population; S43: In the mutual benefit stage, two randomly selected candidate paths are interacted to generate an average vector MV and a new path individual; S44: Enter the symbiotic stage, and randomly select another path for local update for a candidate path, the local update including simulating the local influence of one candidate path on another path in symbiotic behavior; S45: In the parasitic stage, a parasitic vector PV is generated to replace part of the dimensions of a candidate path and introduce perturbation; S46: If the maximum number of iterations or the convergence precision is reached, output the global optimal path as the initial flight path from the starting point to the target point, and encapsulate the path information as pre-planning instructions, otherwise return to the mutual benefit stage and re-symbiosis.

[0010] Preferably, the fine data collection subgroup S5 performs tasks according to pre-planning instructions to perform fine-grained data collection on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute, when detecting new or changed obstacles, obtaining an updated obstacle instance list, recalculating the local optimal path according to the updated obstacle instance list to realize dynamic avoidance and tracking of dynamic obstacles, obtaining an update instruction, the update instruction only carrying incremental information, including: S51: The fine data collection subgroup performs tasks according to pre-planning instructions to perform fine-grained data collection on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute; S52: When detecting new or changed obstacles, an updated obstacle instance list is obtained; S53: Recalculating the local optimal path according to the updated obstacle instance list to realize dynamic avoidance and tracking of dynamic obstacles, obtaining an update instruction, specifically including: inputting the current position vector of the UAV q Input the potential field function to obtain an attractive force vector:

[0011] Wherein, q is the current position vector of the UAV, q goal is the target point position vector, d is the Euclidean distance between the current position q of the UAV and the target point position q goal k att is the attractive potential field coefficient, is the segmented threshold distance; Convert the updated obstacle instance list into an obstacle position vector q obs， And Input the repulsive potential field function to obtain a repulsive force vector,

[0012] Wherein, q obs is the obstacle position vector, d ( q , q obs ) is the distance from the current position of the UAV to the obstacle, d ( q , q goal ) is the distance from the current position of the UAV to the target point, k rep is the repulsive potential field coefficient;​d 0 is the radius of influence of the obstacle, when d q q obs d 0, the influence of the obstacle on the UAV is almost zero, is an exponential decay function, if the obstacle is a dynamic obstacle and is required to track the obstacle, the repulsive potential field coefficient is negative; S54: vector superposition of the attractive force vector and the repulsive force vector to form a synthetic potential field vector, recalculate the local optimal path based on the synthetic potential field vector, and output an update instruction.

[0013] Preferably, the fine data acquisition subgroup of S6 obtains final fine-grained data according to the update instruction, merges the local environment map data layer and the final fine-grained data to obtain the full-scene perception result, and the final fine-grained data includes detailed motion information of dynamic obstacles, including: S61: The fine data acquisition subgroup acquires depth image and laser radar point cloud information according to the local optimal path according to the update instruction; S62: Extract detailed motion information of dynamic obstacles to obtain final fine-grained data; S63: Merge the local environment map data layer and the final fine-grained data to obtain the full-scene perception result.

[0014] On the other hand, a UAV dynamic obstacle avoidance and path planning device based on multi-modal image recognition is provided, which is applied to a UAV dynamic obstacle avoidance and path planning method based on multi-modal image recognition, and the device comprises: An obstacle perception module is configured to generate an obstacle perception task instruction and issue it to a UAV cluster to obtain obstacle perception data, wherein the UAV cluster comprises an obstacle perception data acquisition subgroup and a fine data acquisition subgroup, the obstacle perception data acquisition subgroup is configured to acquire RGB optical images and infrared thermal images, and the fine data acquisition subgroup is configured to acquire RGB optical images, infrared thermal images, depth maps, and laser radar point cloud data, the obstacle perception data acquisition subgroup is further provided with a calibration board module and a geometric registration module, the geometric registration module is configured to unify the obstacle perception data to the same world coordinate system based on the calibration board, and the fine data acquisition subgroup is further provided with a synchronous acquisition scheduler, which is configured to ensure time alignment of multi-modal data; An obstacle instance module is configured to respectively or jointly use an image segmentation algorithm on the obstacle perception data to identify static obstacles in the flight path, and output an obstacle instance list, calculate N candidate paths and global optimal flight path parameters based on the obstacle instance list, and the obstacle instance list includes position information and category labels; ​​​a local environment map module configured to generate a local environment map data layer based on the obstacle instance list, the local environment map data layer including a geometry layer and a semantic layer; a pre-planning instruction module configured to calculate an initial flight path from the start point to the target point based on the N candidate paths and the local environment map data layer, and encapsulate the path information as pre-planning instructions; a fine-grained data acquisition module configured to perform a task according to the pre-planning instructions, and to perform fine-grained data acquisition on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label, and dynamic attribute, to obtain an updated obstacle instance list when detecting new or changed obstacles, to recalculate a locally optimal path to achieve dynamic obstacle avoidance and tracking according to the updated obstacle instance list, and to obtain an update instruction, the update instruction carrying only incremental information; a full-scene perception module configured to obtain final fine-grained data according to the update instruction, to merge the local environment map data layer and the final fine-grained data, and to obtain full-scene perception results, the final fine-grained data including detailed motion information of dynamic obstacles.

[0015] In another aspect, an unmanned aerial vehicle dynamic obstacle avoidance and path planning device based on multi-modal image recognition is provided, the unmanned aerial vehicle dynamic obstacle avoidance and path planning device based on multi-modal image recognition comprising a processor, and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the methods described above.

[0016] In another aspect, a computer readable storage medium is provided, the computer readable storage medium having program code stored therein, the program code being executable by a processor to implement any one of the methods of claims 1 to 7.

[0017] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects: The flight environment images and obstacle information are obtained by the multi-modal sensors, the multi-level data acquisition scheme is used, the static data acquisition and real-time identification and positioning of obstacles are combined, the perception accuracy and obstacle avoidance reliability are improved, the task adaptability is enhanced in path pre-planning and dynamic updating, the unmanned aerial vehicle can safely and efficiently complete tasks such as inspection, search and rescue, surveying and mapping, and logistics in complex terrain, and the flight safety and efficiency are improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0019] Figure 1 is a flow chart of a multi-modal image recognition-based dynamic obstacle avoidance and path planning method for a UAV provided by an embodiment of the present application; Figure 2 is a flow chart of a pre-planning instruction generation method provided by an embodiment of the present application; Figure 3 is a block diagram of a multi-modal image recognition-based dynamic obstacle avoidance and path planning device for a UAV provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of a multi-modal image recognition-based dynamic obstacle avoidance and path planning device for a UAV provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the present application will be described below with reference to the drawings.

[0021] In the embodiments of the present application, the words such as “example”, “for example” and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as “example” in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word “example” is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by “and / or” can be both, or can be one of the two.

[0022] In the embodiments of the present application, “image” and “picture” can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent. “Of”, “corresponding” and “corresponding” can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meanings expressed are consistent.

[0023] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1. When the distinction is not emphasized, the meanings expressed are consistent.

[0024] In order to make the technical problems, technical solutions and advantages of the present application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0025] The embodiment of the application provides a multi-modal image recognition-based unmanned aerial vehicle dynamic obstacle avoidance and path planning method, which can be implemented by a multi-modal image recognition-based unmanned aerial vehicle dynamic obstacle avoidance and path planning device. Figure 1 As shown in the multi-modal image recognition-based unmanned aerial vehicle dynamic obstacle avoidance and path planning method flow chart, the processing flow of the method can include the following steps:

[0026] S1: generating an obstacle perception task instruction and delivering the same to a UAV cluster to obtain obstacle perception data, wherein the UAV cluster comprises an obstacle perception data acquisition subcluster and a fine data acquisition subcluster, the obstacle perception data acquisition subcluster is used for acquiring an RGB optical image and an infrared thermal image, the fine data acquisition subcluster is used for acquiring an RGB optical image, an infrared thermal image, a depth map and laser radar point cloud data, the obstacle perception data acquisition subcluster is further provided with a calibration plate module and a geometric registration module, the geometric registration module is used for unifying the obstacle perception data to a same world coordinate system based on the calibration plate, and the fine data acquisition subcluster is further provided with a synchronous acquisition scheduler, and the synchronous acquisition scheduler is used for ensuring time alignment of multi-modal data. Preferably, S1 comprises the following steps. S11: generating a 3D flight route grid by using a digital elevation model and target operation area boundary information, wherein the 3D flight route grid covers the entire target operation area; S12: generating an obstacle perception task instruction based on the 3D flight route grid and flight parameters, wherein the obstacle perception task instruction comprises a target grid coverage series required by each UAV flight path in the obstacle perception data acquisition subcluster to cover, and the target grid coverage series comprises a grid center point sequence; S13: performing spatial calibration on the acquired RGB optical image, infrared thermal image and laser radar point cloud by using a calibration plate installation module to form geometric registration parameters comprising camera intrinsic parameters, extrinsic parameters and laser radar attitude information; S14: unifying and mapping the obstacle perception data acquired by different UAVs to a same world coordinate system based on the geometric registration parameters to obtain the obstacle perception data.

[0027] In some embodiments, in the task planning module of the ground station, the operation area boundary is taken as an outer frame, and a plurality of isohypse planes are divided in the vertical direction according to set flight height levels (for example, 30m, 45m and 60m).

[0028] It should be noted that all the grid of the same height plane is merged to form a three-dimensional route grid (3D route grid) covering the entire target operation area, and a unique identification (GridID) and center coordinates (X, Y, Z) are assigned to each grid unit in the database.

[0029] It should be further pointed out that a calibration plate with a known size and a high-contrast checkerboard pattern can be fixed on the sensor platform of each selected unmanned aerial vehicle, or a plurality of groups of calibration plates with different attitudes can be arranged on a known plane outdoors before the experiment. Based on data acquisition, the camera is calibrated with the camera intrinsic parameters, and the extrinsic matrix is obtained by, for example, Zhang Zhengyou method.

[0030] Preferably, according to the attitude information of the corresponding unmanned aerial vehicle and the geometric registration parameters, the sensor data is first converted from the sensor coordinate system to the body coordinate system, and then mapped to the unified world coordinate system through the body attitude matrix.

[0031] S2: respectively or jointly using image segmentation algorithm on the obstacle perception data, identifying static obstacles in the flight path, and outputting an obstacle instance list, calculating N candidate paths and global optimal flight path parameters based on the obstacle instance list, the obstacle instance list including position information and category label; Preferably, S2 includes: S21: setting the total number of paths , the total number of key point coordinates of the route , the maximum number of iterations , the scaling factor , and the crossover probability ; according to the geometric position information of each static obstacle in the obstacle instance list and the preset safety distance, all volumes intersecting with the obstacles or violating the safety distance are first removed in the entire operation space to form a feasible subspace:

[0032] Np groups of candidate path parameter vectors are randomly generated in the feasible subspace, each vector containing the three-dimensional coordinates of all key waypoints and satisfying all geometric constraints; S22: further using differential evolution mutation and adaptive crossover strategy to update the Np groups of candidate path parameter vectors to obtain updated candidate path parameter vectors, specifically including: for any candidate of the current iteration t step, randomly selecting three different individuals , , , generating a mutation vector:

[0033]

[0034] wherein the dynamic scaling factor changes with the number of iterations, the mutation vector still lies in the entire job space, the cross-candidate path parameters are calculated:

[0035]

[0036] wherein, X DE is the final selected path, V j is the position information generated by mutation, X i,j is the original position of the current individual, CR is the adaptive crossover probability, rand j is a randomly generated parameter value, j rand is a randomly generated selection index; S23: Geometric constraint checking is performed, and if the corresponding waypoint coordinates conflict with any obstacle instance, the candidate is marked as infeasible in that dimension or resampled directly: ; S24: After updating all candidates, proceed to the next iteration until the maximum number of iterations is reached or the convergence criterion is met, obtaining N candidate paths and global optimal flight path parameters.

[0037] In some embodiments, the total number of paths N is used to output the top N candidate paths after the iteration ends. The total number of waypoints K includes the number of intermediate waypoints in addition to the start and end points. The maximum number of iterations Gmax is used to control the upper limit of differential evolution convergence. The scaling factor F is greater than 0 and less than 1, and the adaptive range is, for example, [0.4, 0.9]; the crossover probability CR is greater than 0 and less than 1, and the adaptive range is, for example, [0.5, 0.95].

[0038] It should be noted that the geometric boundary B of each obstacle Oj is buffered by dsafe to obtain a safe buffer boundary. Different obstacle buffer zones are not necessarily consistent. After removing the obstacles, a connectedness analysis is performed on the remaining regions to further remove disconnected regions to enhance the feasibility sampling efficiency.

[0039] It should be further noted that the waypoints are used as basic units for block crossover, i.e., some intermediate waypoints are replaced in blocks to preserve local geometric continuity and reduce U-turns and sharp turns.

[0040] Preferably, if two consecutive local samplings fail, the candidate path is marked as infeasible and replaced by the one with the highest cost function in the same generation to avoid affecting convergence.

[0041] S3: Generate a local environment map data layer based on the obstacle instance list, the local environment map data layer including a geometric layer and a semantic layer; Preferably, S3 includes: S31: Based on the list of obstacle instances, generate a geometry layer for calculating the optimal path search, wherein the geometry layer includes the shape, position, and coordinates of the obstacle points; S32: A semantic layer for calculating the optimal path search is constructed based on the globally optimal flight path parameters and the geometric layer. This semantic layer includes weight parameters for the fitness function and various constraint cost functions. The path fitness function includes:

[0042] in, C (Y) is the fitness function of path Y. C L (Y) is the path length function. C H (Y) is the path height function. C S (Y) is the path smoothness function. C C (Y) is the path-constrained cost function. These are all weight parameters of their respective cost functions; The path length function includes:

[0043] in, C L (Y) represents the cost of flight distance. s The number of waypoints on the flight path, ( x i , y i , z i ) is the first i The coordinates of each waypoint; The path height function includes:

[0044] in, C H (Y) represents highly variable costs. z i For the first i trackpoints z Axis coordinate values;

[0045] wherein (Y) is a turn cost, is a vector of flight segments x i +1- x i , y i +1- y i , z i +1- z i ); S33: merging the geometry layer and the semantic layer to generate a local environment map data layer, the local environment map data layer being used to support optimal path search.

[0046] In some embodiments, each obstacle instance contains at least obstacle ID, type label, bounding box / contour, pose information and confidence evaluation.

[0047] It should be noted that the track point coordinates in the geometry layer are sorted according to timestamps, and are accompanied by geometric properties such as curvature, climb rate, turn radius, etc. for calling by the optimal path search algorithm when evaluating the cost.

[0048] It should be further noted that the global optimal flight path parameters include but are not limited to: mission priority parameters (time optimal / energy optimal / safety priority), global no-fly zone and height limit zone constraints, waypoint passing time window, minimum route interval, formation keeping parameters, flight control dynamics constraints (maximum climb rate, maximum lateral / longitudinal overload, minimum turn radius), and redundancy safety factor.

[0049] Preferably, the local environment map data layer supports the optimal path search algorithm; in the search process, the geometry layer provides passability judgment and collision detection, and the semantic layer provides cost evaluation and constraint filtering.

[0050] S4: based on the N candidate paths and the local environment map data layer, calculating an initial flight path from the starting point to the target point, and encapsulating the path information as pre-planning instructions; As Figure 2 shown, preferably, S4 includes: S41: based on the maximum number of iterations, N candidate paths and the local environment map data layer, initializing to generate an initial population containing N candidate paths ; S42: calculating the fitness function of each candidate path in the population ; S43: Through the mutual benefit stage, the randomly selected two candidate paths are interacted to generate an average vector MV and a new path individual:

[0051]

[0052] wherein, is a random number, is an average vector, is a mutual benefit factor, a mutation term is introduced; DE S44: In the symbiosis stage, a local update is performed on a candidate path by randomly selecting another path, and the local update includes simulating the local influence of one candidate path on another path in symbiotic behavior:

[0053] wherein, is a dynamic control factor; S45: In the parasitism stage, a parasitic vector is generated, a partial dimension of a candidate path is replaced, and disturbance is introduced:

[0054] wherein γ ∈ (0, 1), if , then is replaced; S46: If the maximum number of iterations or the convergence precision is reached, a global optimal path is output as an initial flight path from a starting point to a target point, and the path information is encapsulated as a pre-planning instruction, otherwise the mutual benefit stage is returned to and the symbiosis is performed again.

[0055] In some embodiments, for each candidate path Pi in the population, a fitness function is calculated, and the smaller the fitness, the better the path. The fitness can be composed of the following multi-objective weighted sum: the path length cost is the sum of the Euclidean distances of each segment; the smoothness cost is the cumulative penalty based on the angle or curvature of adjacent segments; and the safety cost is the sampling and accumulation of the environmental cost map along the path. The optimal individual and its fitness of the current population are recorded synchronously.

[0056] It should be noted that only a subset of control points is modified during local updating, improving local exploration ability; constraints are applied to the updated control points to repair and smooth local curvature, avoiding large turning angles or crossing obstacles.

[0057] ​​It needs to be further explained that the path point sequence is extracted in a fixed step or key point thinning manner, and corresponding heading, climb rate and speed suggestion values are generated; the flight parameters of the path segment are checked (maximum inclination angle, maximum climb / descent rate, minimum turn radius), and if necessary, the end is fine-tuned to meet the flight control constraints; a pre-planning instruction package is generated, including: path point set, expected height and speed of each point, passing time window, obstacle avoidance priority, path ID and version number, time stamp and parameter verification information.

[0058] S5: The fine data acquisition sub-group executes the task according to the pre-planning instruction, and performs fine-grained data acquisition on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute, when detecting a new or changed obstacle, obtaining an updated obstacle instance list, recalculating a local optimal path according to the updated obstacle instance list to realize dynamic avoidance and tracking of dynamic obstacles, obtaining an update instruction, the update instruction carrying only incremental information; Preferably, S5 includes: S51: The fine data acquisition sub-group executes the task according to the pre-planning instruction, and performs fine-grained data acquisition on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute; S52: When detecting a new or changed obstacle, an updated obstacle instance list is obtained; S53: Recalculating a local optimal path according to the updated obstacle instance list to realize dynamic avoidance and tracking of dynamic obstacles, obtaining an update instruction, specifically including: taking the current position vector of the unmanned aerial vehicle q Input the potential field function to obtain an attractive force vector:

[0059] Wherein, q is the current position vector of the unmanned aerial vehicle, q goal is the target point position vector, d is the current position of the unmanned aerial vehicle q and the target point position q goal Euclidean distance, k att is the attractive potential field coefficient, is the segmented threshold distance; Convert the updated obstacle instance list into an obstacle position vector q obs， And Input the repulsive potential field function to obtain a repulsive force vector,

[0060] Wherein, qobs is the obstacle position vector, d q q obs is the distance from the current position of the UAV to the obstacle, d q q goal is the distance from the current position of the UAV to the target point, k rep is the repulsive potential field coefficient; d 0 is the obstacle influence radius, when d q q obs ) d 0, the influence of the obstacle on the UAV is almost zero, is an exponential decay function, if the obstacle is a dynamic obstacle and needs to track the obstacle, the repulsive potential field coefficient is negative; S54: vector superposition of the attractive force vector and the repulsive force vector to form a synthetic potential field vector, recalculate the local optimal path based on the synthetic potential field vector, and output an update instruction.

[0061] In some embodiments, gradient following is performed on the local map based on the synthetic potential field vector to generate a candidate local optimal path.

[0062] It should be noted that the ground station or formation control node issues instructions to the fine data acquisition sub-group and sets the effective time window and the re-planning trigger threshold; when the obstacle instance list changes significantly within the effective time window, the rapid re-planning cycle of S53 to S54 is automatically triggered to ensure dynamic avoidance and tracking of dynamic obstacles.

[0063] S6: The fine data acquisition sub-group obtains final fine-grained data according to the update instruction, merges the local environment map data layer and the final fine-grained data to obtain the full-scene perception result, and the final fine-grained data includes detailed motion information of dynamic obstacles.

[0064] Preferably, S6 includes: S61: The fine data acquisition sub-group acquires depth image and laser radar point cloud information according to the update instruction and the local optimal path; S62: Extract detailed motion information of dynamic obstacles to obtain final fine-grained data; S63: Merge the local environment map data layer and the final fine-grained data to obtain the full-scene perception result.

[0065] ​​​​​​In some embodiments, the local occupancy grid / semantic grid and the dynamic instance layer are hierarchically fused under a unified coordinate system; the static layer retains high-resolution geometric structures and semantic categories, and the dynamic layer superimposes states and uncertainties in units of instances; when the two layers conflict, the latest timestamp and higher confidence data are given priority.

[0066] It should be noted that the full-scene perception result forms a multi-resolution map, specifically including a global low-frequency updated geometric layer and a local high-frequency updated dynamic instance layer.

[0067] The above is an introduction to the method embodiment, and the following will further illustrate the scheme of the present application through the device embodiment.

[0068] Figure 3 is a kind of unmanned aerial vehicle dynamic obstacle avoidance and path planning device block diagram based on multi-modal image recognition provided by the embodiment of the application, and the device is used for the unmanned aerial vehicle dynamic obstacle avoidance and path planning method based on multi-modal image recognition. Refer to Figure 3 The device includes an obstacle perception module, an obstacle instance module, a local environment map module, a pre-planning instruction module, a fine-grained data acquisition module and a full-scene perception module.

[0069] The obstacle perception module is used to generate obstacle perception task instructions and issue them to a UAV cluster to obtain obstacle perception data, the UAV cluster includes an obstacle perception data acquisition subgroup and a fine data acquisition subgroup, the obstacle perception data acquisition subgroup is used to acquire RGB optical images and infrared thermal images, the fine data acquisition subgroup is used to acquire RGB optical images, infrared thermal images, depth maps and laser radar point cloud data, the obstacle perception data acquisition subgroup is also provided with a calibration board module and a geometric registration module, the geometric registration module unifies the obstacle perception data to the same world coordinate system based on the calibration board, and the fine data acquisition subgroup is also provided with a synchronous acquisition scheduler, which is used to ensure time alignment of multi-modal data. The obstacle instance module is used to respectively or jointly use an image segmentation algorithm on the obstacle perception data, identify static obstacles in the flight path, and output an obstacle instance list, calculate N candidate paths and global optimal flight path parameters based on the obstacle instance list, and the obstacle instance list includes position information and category labels. The local environment map module is used to generate a local environment map data layer based on the obstacle instance list, and the local environment map data layer includes a geometric layer and a semantic layer. The pre-planning instruction module is used to calculate an initial flight path from a starting point to a target point based on the N candidate paths and the local environment map data layer, and encapsulate the path information as a pre-planning instruction. Fine-grained data acquisition module: for fine data acquisition subgroup to execute tasks according to pre-planning instructions, fine-grained data acquisition is performed on the obstacle instance list, the fine-grained data includes obstacle volume size, detailed category label and dynamic attribute, when detecting new or changed obstacles, an updated obstacle instance list is obtained, and a local optimal path is recalculated according to the updated obstacle instance list to realize dynamic obstacle avoidance and tracking, and an update instruction is obtained, the update instruction only carries incremental information; Full-scene perception module: for fine data acquisition subgroup to obtain final fine-grained data according to the update instruction, combine the local environment map data layer and the final fine-grained data to obtain full-scene perception results, and the final fine-grained data includes detailed motion information of dynamic obstacles.

[0070] The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition.

[0071] The application provides a computer readable storage medium. Figure 4 The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition. Figure 4 The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition. Figure 3 The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition.

[0072] The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition.

[0073] The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition.

[0074] The application provides a dynamic obstacle avoidance and path planning device based on multi-modal image recognition. Figure 4The various constituent components of the unmanned aerial vehicle dynamic obstacle avoidance and path planning device 410 based on multi-modal image recognition are described in detail as follows: The first processor 2001 is the control center of the unmanned aerial vehicle dynamic obstacle avoidance and path planning device 410 based on multi-modal image recognition, and can be one processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0075] Alternatively, the first processor 2001 can execute various functions of the unmanned aerial vehicle dynamic obstacle avoidance and path planning device 410 based on multi-modal image recognition by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0076] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as the CPU0 and CPU1 shown in FIG. 1. Figure 4

[0077] In a specific implementation, as an embodiment, the unmanned aerial vehicle dynamic obstacle avoidance and path planning device 410 based on multi-modal image recognition can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 1. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Figure 4

[0078] The memory 2002 is used to store software programs for implementing the present application scheme, and is controlled by the first processor 2001 for execution. The specific implementation manner can refer to the above method embodiments, and will not be described here.

[0079] ​​Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected via the interface circuit of the UAV dynamic obstacle avoidance and path planning device 410 based on multimodal image recognition. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0080] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0081] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0082] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently, and it can be connected to the interface circuit of the UAV dynamic obstacle avoidance and path planning device 410 based on multimodal image recognition. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0083] It should be noted that, Figure 4 The structure of the UAV dynamic obstacle avoidance and path planning device 410 based on multimodal image recognition shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown, or combine some components, or have different component arrangements.

[0084] In addition, the technical effects of the unmanned aerial vehicle dynamic obstacle avoidance and path planning device 410 based on multi-modal image recognition can refer to the technical effects of the unmanned aerial vehicle dynamic obstacle avoidance and path planning method based on multi-modal image recognition described in the above method embodiments, which will not be repeated here.

[0085] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0086] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

[0087] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0088] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0089] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0090] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0091] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0093] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0094] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0095] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0096] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for dynamic obstacle avoidance and path planning of a UAV based on multi-modal image recognition, characterized in that, The method comprises: S1: generating obstacle perception task instructions and issuing them to a UAV cluster to obtain obstacle perception data, the UAV cluster comprising an obstacle perception data collection sub-cluster and a fine data collection sub-cluster, the obstacle perception data collection sub-cluster being configured to collect RGB optical images and infrared thermal images, and the fine data collection sub-cluster being configured to collect RGB optical images, infrared thermal images, depth maps, and laser radar point cloud data, the obstacle perception data collection sub-cluster further being provided with a calibration board module and a geometric registration module, the geometric registration module being configured to unify the obstacle perception data to the same world coordinate system based on the calibration board, and the fine data collection sub-cluster further being provided with a synchronous collection scheduler, the synchronous collection scheduler being configured to ensure time alignment of multi-modal data; S2: using image segmentation algorithms on the obstacle perception data respectively or jointly to identify static obstacles in a flight path and output an obstacle instance list, calculating N candidate paths and global optimal flight path parameters based on the obstacle instance list, the obstacle instance list comprising position information and category labels; S3: generating a local environment map data layer based on the obstacle instance list, the local environment map data layer comprising a geometric layer and a semantic layer; S4: calculating an initial flight path from a starting point to a target point based on the N candidate paths and the local environment map data layer, and encapsulating the path information as pre-planning instructions; S5: the fine data collection sub-cluster performing a task according to the pre-planning instructions, collecting fine-grained data on the obstacle instance list, the fine-grained data comprising obstacle volume size, detailed category labels, and dynamic attributes, obtaining an updated obstacle instance list when detecting new or changed obstacles, recalculating a locally optimal path based on the updated obstacle instance list to achieve dynamic obstacle avoidance and tracking, obtaining an update instruction, the update instruction carrying only incremental information; S6: the fine data collection sub-cluster obtaining final fine-grained data according to the update instruction, merging the local environment map data layer and the final fine-grained data to obtain full-scene perception results, the final fine-grained data comprising detailed motion information of dynamic obstacles.

2. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The S1 of generating obstacle perception task instructions and issuing them to a UAV cluster to obtain obstacle perception data, the UAV cluster comprising an obstacle perception data collection sub-cluster and a fine data collection sub-cluster, the obstacle perception data collection sub-cluster being configured to collect RGB optical images and infrared thermal images, and the fine data collection sub-cluster being configured to collect RGB optical images, infrared thermal images, depth maps, and laser radar point cloud data, the obstacle perception data collection sub-cluster further being provided with a calibration board module and a geometric registration module, the geometric registration module being configured to unify the obstacle perception data to the same world coordinate system based on the calibration board, and the fine data collection sub-cluster further being provided with a synchronous collection scheduler, the synchronous collection scheduler being configured to ensure time alignment of multi-modal data, comprises: S11: generating a 3D flight route grid using a digital elevation model and target operation area boundary information, the 3D flight route grid covering the entire target operation area; S12: generating obstacle perception task instructions based on the 3D route grid and flight parameters, the obstacle perception task instructions including a target grid coverage series required to cover each UAV flight path in an obstacle perception data collection subgroup, the target grid coverage series including a grid center point sequence; S13: spatially calibrating the collected RGB optical images, infrared thermal images, and laser radar point clouds through a calibration plate installation module to form geometric registration parameters including camera intrinsic parameters, extrinsic parameters, and laser radar attitude information; S14: based on the geometric registration parameters, mapping the obstacle perception data obtained by different UAVs to the same world coordinate system to obtain obstacle perception data.

3. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The S2 uses an image segmentation algorithm on the obstacle perception data respectively or jointly to identify static obstacles in the flight path, and outputs an obstacle instance list, calculates N candidate paths and global optimal flight path parameters based on the obstacle instance list, and the obstacle instance list includes position information and category labels, including: S21: setting the total number of paths, the total number of air route key point coordinates, the maximum number of iterations, the scaling factor, and the crossover probability, eliminating all volumes intersecting with obstacles or violating the safety distance in the entire operation space according to the geometric position information of each static obstacle in the obstacle instance list and the preset safety distance, forming a feasible subspace, and randomly generating Np groups of candidate path parameter vectors in the feasible subspace, each vector containing three-dimensional coordinates of all key waypoints and satisfying all geometric constraints; S22: further updating the Np groups of candidate path parameter vectors using a differential evolution mutation and adaptive crossover strategy to obtain updated candidate path parameter vectors; S23: performing geometric constraint checking, if the corresponding waypoint coordinates conflict with any obstacle instance, resampling in that dimension or directly marking the candidate as infeasible; S24: after completing the update of all candidates, entering the next iteration until the maximum number of iterations is reached or the convergence criterion is met, obtaining N candidate paths and global optimal flight path parameters.

4. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The S3 generates a local environment map data layer based on the obstacle instance list, the local environment map data layer including a geometric layer and a semantic layer, including: S31: generating a geometric layer for calculating optimal path search based on the obstacle instance list, the geometric layer containing obstacle shape, position, and waypoint coordinates; S32: constructing a semantic layer for calculating optimal path search based on the global optimal flight path parameters and the geometric layer; S33: merging the geometric layer and the semantic layer to generate a local environment map data layer, the local environment map data layer being used to support optimal path search.

5. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The S4 calculates an initial flight path from the starting point to the target point based on the N candidate paths and the local environment map data layer, and encapsulates the path information as pre-planning instructions, including: S41: initializing based on the maximum number of iterations, N candidate paths, and local environment map data layer to generate an initial population X containing N candidate paths; S42: Calculate the fitness function of each candidate path in the population; S43: In the mutual benefit stage, interact randomly selected two candidate paths to generate an average vector MV and a new path individual; S44: Enter the symbiotic stage, and randomly select another path to implement local update on a candidate path, the local update including simulating the local influence of one candidate path on another path in symbiotic behavior; S45: In the parasitic stage, generate a parasitic vector PV, and perform partial dimension replacement on a candidate path to introduce disturbance; S46: If the maximum number of iterations or convergence accuracy is reached, output the global optimal path as the initial flight path from the starting point to the target point, and encapsulate the path information as a pre-planning instruction, otherwise return to the mutual benefit stage and re-perform symbiosis.

6. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The fine data acquisition subgroup of S5 executes tasks according to the pre-planning instruction, and performs fine-grained data acquisition on the obstacle instance list, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute, when detecting new or changed obstacles, obtaining an updated obstacle instance list, recalculating the local optimal path according to the updated obstacle instance list to realize dynamic obstacle avoidance and tracking, obtaining an update instruction, the update instruction only carrying incremental information, including: S51: The fine data acquisition subgroup acquires fine-grained data of the obstacle instance list according to the pre-planning instruction, the fine-grained data including obstacle volume size, detailed category label and dynamic attribute; S52: When detecting new or changed obstacles, obtain an updated obstacle instance list; S53: Recalculating the local optimal path according to the updated obstacle instance list to realize dynamic avoidance and tracking of dynamic obstacles, and obtaining an updated instruction, specifically including: obtaining a current position vector of the UAV q inputting a potential field function to obtain an attractive force vector, wherein, q is a current position vector of the UAV, q goal is a target point position vector, d is a current position of the UAV q and the target point position q goal Euclidean distance, k att is an attractive potential field coefficient, d ∗ is a piecewise threshold distance; convert the updated obstacle instance list into an obstacle position vector q obs， And input the repulsive potential field function to obtain a repulsive force vector, wherein, q obs is the obstacle position vector, d q , q obs is the distance from the current position of the UAV to the obstacle, d q , q goal is the distance from the current position of the UAV to the target point, k rep is the repulsive potential field coefficient; d 0 is the obstacle influence radius, when d q , q obs ) d 0, the influence of the obstacle on the UAV is almost zero, is an exponential decay function, if the obstacle is a dynamic obstacle and is required to track the obstacle, the repulsive potential field coefficient is negative;​​​ S54: Superimpose the attractive force vector and the repulsive force vector to form a synthetic potential field vector, recalculate the local optimal path based on the synthetic potential field vector, and output the update instruction.

7. The multi-modal image recognition based dynamic obstacle avoidance and path planning method for UAVs according to claim 1, characterized in that, The fine data acquisition subgroup of S6 obtains final fine-grained data according to the update instruction, merges the local environment map data layer and the final fine-grained data to obtain the full-scene perception result, the final fine-grained data including detailed motion information of dynamic obstacles, including: S61: The fine data acquisition subgroup acquires depth image and laser radar point cloud information according to the update instruction and the local optimal path; S62: Extract detailed motion information of dynamic obstacles to obtain final fine-grained data; S63: Merge the local environment map data layer and the final fine-grained data to obtain the full-scene perception result.

8. A multi-modal image recognition based dynamic obstacle avoidance and path planning device for unmanned aerial vehicles, the multi-modal image recognition based dynamic obstacle avoidance and path planning device being configured to implement the multi-modal image recognition based dynamic obstacle avoidance and path planning method for unmanned aerial vehicles according to any one of claims 1-7. The device comprises: An obstacle perception module is configured to generate obstacle perception task instructions and send them to a UAV cluster, so as to obtain obstacle perception data, wherein the UAV cluster comprises an obstacle perception data acquisition sub-cluster and a fine data acquisition sub-cluster, the obstacle perception data acquisition sub-cluster is configured to acquire RGB optical images and infrared thermal images, the fine data acquisition sub-cluster is configured to acquire RGB optical images, infrared thermal images, depth maps and laser radar point cloud data, the obstacle perception data acquisition sub-cluster is further provided with a calibration board module and a geometric registration module, the geometric registration module is configured to unify the obstacle perception data to the same world coordinate system based on the calibration board, and the fine data acquisition sub-cluster is further provided with a synchronous acquisition scheduler, the synchronous acquisition scheduler is configured to ensure time alignment of multi-modal data; An obstacle instance module is configured to respectively or jointly use an image segmentation algorithm on the obstacle perception data, identify static obstacles in a flight path, and output an obstacle instance list, calculate N candidate paths and global optimal flight path parameters based on the obstacle instance list, and the obstacle instance list comprises position information and category labels; A local environment map module is configured to generate a local environment map data layer based on the obstacle instance list, wherein the local environment map data layer comprises a geometry layer and a semantic layer; A pre-planning instruction module is configured to calculate an initial flight path from a starting point to a target point based on the N candidate paths and the local environment map data layer, and encapsulate the path information as pre-planning instructions; A fine-grained data acquisition module is configured to perform a task according to the pre-planning instructions by the fine data acquisition sub-cluster, and perform fine-grained data acquisition on the obstacle instance list, wherein the fine-grained data comprises obstacle volume size, detailed category labels and dynamic attributes, an updated obstacle instance list is obtained when a new or changed obstacle is detected, a local optimal path is recalculated based on the updated obstacle instance list to realize dynamic obstacle avoidance and tracking, and an update instruction is obtained, wherein the update instruction only carries incremental information; A full-scene perception module is configured to obtain final fine-grained data according to the update instruction by the fine data acquisition sub-cluster, combine the local environment map data layer and the final fine-grained data, and obtain full-scene perception results, wherein the final fine-grained data comprises detailed motion information of dynamic obstacles.

9. An unmanned aerial vehicle dynamic obstacle avoidance and path planning device based on multi-modal image recognition, characterized in that, The UAV dynamic obstacle avoidance and path planning processor based on multi-modal image recognition; a memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to implement the method of any one of claims 1 to 7.

Images