A UAV autonomous flight technology method integrating panoramic SLAM and target recognition

Abstract

The present invention discloses a method for autonomous flight of unmanned aerial vehicles that integrates panoramic SLAM and target recognition. A four-rotor unmanned aerial vehicle equipped with a panoramic camera is used as a carrier, and the panoramic SLAM algorithm is integrated with the target recognition and obstacle avoidance algorithm to realize autonomous flight of the unmanned aerial vehicle in a location environment. This set of unmanned aerial vehicle autonomous flight technology first relies on panoramic SLAM to complete the construction of the environmental map in an unknown environment while performing self-positioning, and uses the YOLOv5 algorithm to identify the object target in the field of view, and after determining the obstacle position information, it combines the positioning information and uses the D*Lite algorithm to perform autonomous obstacle avoidance and path planning in real time. This system has the following advantages: small size, low cost, wide application scenarios, accurate positioning, the recognition and obstacle avoidance process can be completed automatically and intelligently, reducing manual intervention and improving work efficiency.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing mapping, and in particular relates to an autonomous flight technology of a UAV integrating panoramic SLAM and target recognition. Background Art

[0002] With the development of drone platforms and sensor technology, drone systems are constantly developing in the direction of identification-autonomy-perception-behavior, thus gradually moving towards the development of autonomous flying drones. In the field of autonomous flying drones, after receiving the take-off command, the drone can use onboard sensors to help the drone perceive the surrounding environment, locate its own movement and status without further manipulation, so as to achieve the purpose of autonomous control and flight of the drone. In traditional industrial inspections, manual visual inspections are required, which are labor-intensive and the harsh environment will bring greater challenges to the inspections. Autonomous flying drone technology can realize the refined and automated inspection of equipment such as wind turbine towers and transmission lines in high altitudes and forests; in the field of emergency rescue, autonomous flying drone technology can quickly build environmental maps, search for designated targets, and assist in rescue.

[0003] The onboard sensors of drones are the key to autonomous control and flight. Onboard sensors include visual cameras, lidar, IMU, GNSS, etc. In order to achieve accurate and reliable autonomous flight, multi-source sensors need to be integrated, which places higher demands on the payload and endurance of drones. At the same time, the hardware cost, integration difficulty and safety considerations of multi-source sensors also restrict the application of autonomous flight of drones.

[0004] The characteristics of different sensors play a decisive role in the degree of autonomy of drone flight inspection. GNSS and IMU are carrier state sensors and cannot perceive the surrounding environment. Driven by the development of SLAM technology, visual cameras and lidar can not only estimate the carrier state, but also perceive and map the surrounding environment. Compared with visual cameras and lidar, although lidar can complete positioning and mapping at an absolute scale, the reliability and accuracy of target recognition are still insufficient, and laser SLAM with a high frame rate is easily affected by high-speed motion and large rotation angle motion. Visual technology can not only complete SLAM positioning and mapping, but also has the advantage of real-time target recognition, which is very suitable for target obstacle avoidance and target inspection. In addition, the camera is small in size, light in weight, small in power load, and low in cost, which is very suitable for small wheelbase quadcopter drones. Visual SLAM includes monocular, binocular, and RGB-D branches. The mainstream sensors currently carried by drones are monocular, binocular and depth cameras, and the corresponding visual SLAM algorithms have certain limitations. The monocular SLAM has a small field of view, which limits the amount of environmental information obtained, and cannot achieve 360° perception of the surrounding environment. The accuracy of the scale estimation of the monocular camera will drop sharply as the cumulative error increases. The field of view of the binocular SLAM still does not increase on the basis of solving the scale drift, which will cause the feature points to be lost in the case of intense movement. The field of view can be increased by combining multiple cameras, but at the same time it will increase the load of the drone, which is not conducive to endurance work, and the calibration between cameras will further introduce errors. Therefore, visual SLAM is not widely used in large-scale mobile mapping. In response to the above problems, panoramic vision has the advantages of 360° environmental perception and fast and complete information acquisition, and the omnidirectional view can improve the reliability and orientation accuracy of image frame matching, and is not affected by the motion state. However, the current algorithms based on panoramic vision SLAM are rarely used, and most of them are used for data collection. There is a lack of a complete SLAM solution and its application in industrial applications.

[0005] In summary, in autonomous UAVs, panoramic vision sensors can provide UAVs with maximum flight inspection autonomy with minimal hardware and integration costs, while realizing environmental perception, target recognition and obstacle avoidance functions. How to propose a UAV autonomous flight technology that integrates panoramic SLAM and target recognition is a key issue that needs to be solved in this field. Summary of the invention

[0006] The present invention proposes an autonomous flight technology for UAVs that integrates panoramic SLAM and target recognition. The technology uses a panoramic vision camera as the only sensor for the UAV platform. By integrating panoramic SLAM technology and target recognition technology, the technology performs real-time path planning and obstacle avoidance on the basis of solving the scale uncertainty and scale drift problems of visual SLAM, thereby realizing autonomous flight of the UAV in a location environment.

[0007] The autonomous flying drone system constructed by the present invention is suitable for map construction and self-positioning in unknown environments, and can also identify objects in the environment, perform real-time autonomous obstacle avoidance and path planning. To achieve the above functions, the technical problems to be solved by the present invention are: construction of a lightweight panoramic SLAM drone platform; panoramic SLAM technology; panoramic visual target recognition and obstacle avoidance.

[0008] The present invention proposes a UAV autonomous flight technology method integrating panoramic SLAM and target recognition, comprising the following steps:

[0009] Step 1: Build a small drone platform; select a small-wheelbase quadcopter drone as the flight platform, and install a panoramic camera on the front of the drone platform to effectively obtain 360° viewing angle information; transmit the video information to the onboard computer in real time for real-time calculation, so as to obtain the drone's flight attitude and the target of interest in the scene; then transmit the flight attitude data to the flight control system to autonomously control the drone's flight;

[0010] Step 2: Use the panoramic camera to collect environmental images, run real-time panoramic SLAM, and complete the positioning and autonomous flight of the drone;

[0011] Step 3: Based on target recognition technology, detect objects in real-time images, combine the positioning information output by SLAM, plan the optimal path with the target object as the end point, and complete autonomous obstacle avoidance flight.

[0012] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0013] Step 2.1, constructing an imaging model of a panoramic image, including a single-lens imaging model and a multi-lens imaging model, to achieve efficient stitching of panoramic images and complete spherical mapping of image data;

[0014] Step 2.2, using the SPHORB algorithm to perform feature extraction and matching stitching on the spherical image mapped in step 2.1;

[0015] Step 2.3, based on the feature point pairs extracted in step 2.2, solve the pose transformation relationship between adjacent frames through epipolar constraints, and use nonlinear optimization to optimize the three-dimensional coordinates of the feature points and the camera pose to minimize the reprojection error, output the optimal camera pose to complete positioning, and use this pose to transform and splice the feature points to complete the map.

[0016] Furthermore, the specific implementation of step 2.1 is as follows;

[0017] First, a panoramic camera coordinate system is established. The panoramic camera on the UAV platform is abstracted as a spherical camera model. Considering the single-lens case, the center of the sphere is made to coincide with the optical center of the camera. Let O-xyz be the camera coordinate system. For the object point P, it is mapped to point p on the image plane, and the corresponding coordinates in the image coordinate system are (u, v); at the same time, the light OP intersects the sphere with O as the center at point P. s ;

[0018] Define spherical mapping as a function mapping relationship Map any point p(u,v) on the image to a sphere with a certain radius and express it in spherical coordinates as To use the right-hand rule, the thumb points toward the y-axis, and the x-axis and z-axis rotate by an angle θ toward the direction of the other fingers; For the right-hand rule, the thumb points toward the z-axis, and the x-axis and y-axis rotate in the direction of the other fingers. Angle value;

[0019] Let α be the vector The angle between the projection and the axis on the plane O-yz, β is the vector The angle between the plane O-yz and the image; for the actual image, the coordinates of the u-axis and v-axis are finite values, so the value range of α and β is (-π / 2,π / 2). According to the spatial geometric relationship, the formula (2.1) is as follows:

[0020]

[0021] Where f is the focal length of the camera, (u 0 ,v 0 ) is the main point bias; in fact, the above formula provides a mapping relationship from the image coordinate system to the local angle in the sphere That is, the one-to-one correspondence between points on the image plane and some points on the spherical surface;

[0022] In a multi-lens panoramic camera, each sub-camera has an independent camera coordinate system, and the coordinate system of the multi-lens panoramic camera is based on the single-lens spherical coordinate system. However, due to the process and the volume of the lens itself, the optical centers of the lenses do not coincide. Therefore, when performing spherical mapping on the collected image data, it is necessary to unify the sub-cameras of the multi-lens panoramic camera into an overall coordinate system, and then convert the pixel coordinates of the images collected by each lens into the overall coordinate system through formula 2.1, and perform spherical mapping uniformly.

[0023] Furthermore, the specific implementation of step 3 is as follows;

[0024] Step 3.1, using the YOLOv5 algorithm as the target recognition algorithm, inputting the target data set of the panoramic image to train the neural network model, using the trained model to perform real-time target recognition on the panoramic image transmitted in step 2, and outputting the recognition result;

[0025] Step 3.2, using the time difference between adjacent frames and the posture transformation obtained in step 2.3, the aircraft speed, acceleration and angular velocity are calculated, and the aircraft posture is dynamically tracked through the Kalman filter navigation algorithm; during the flight, according to the obstacle position information identified in step 3.1 and the real-time positioning information of the aircraft, the D*Lite search algorithm is used to perform three-dimensional track planning to achieve real-time autonomous obstacle avoidance.

[0026] Furthermore, the YOLOv5 network model in step 3.1 includes an input end, a baseline network, a Neck network, and a Head output end; the input end represents the input image, and the input image size is 608*608. At this stage, the input image can be scaled to the input size of the network and normalized; in the network training stage, YOLOv5 uses the Mosaic data enhancement operation to improve the training speed of the model and the accuracy of the network; the baseline network represents a classifier network with excellent performance, which is used to extract general feature representations, and uses CSPDarknet53 and Focus structures as baseline networks; the CSPDarknet53 structure draws on the design ideas of CSPNet and is designed to improve the feature representation dimension in the backbone network, and the Focus structure is through the slice operation The input image is cropped by the operation. The original input image size is 608*608*3. After the Slice and Concat operations, a feature map of 304*304*12 is output; then it passes through a Conv layer with 32 channels to output a feature map of 304*304*32; the Neck network is located in the middle of the benchmark network and the Head output end. It draws on the CSP2 structure designed by CSPnet, improves the SPP module and FPN+PAN module, thereby enhancing the network feature fusion capability; the Head output end is used to complete the output of the target recognition result; Yolov5 adopts the GIOU_Loss function, which solves the problem when the bounding boxes do not overlap based on IOU, thereby further improving the detection accuracy of the algorithm.

[0027] Furthermore, the specific implementation of step 3.2 is as follows;

[0028] First, the time difference and posture change between two adjacent frames obtained in step 2.3 are used to infer the speed, acceleration and angular velocity of the aircraft, and the Kalman filter navigation algorithm is used to dynamically track the aircraft posture to achieve autonomous navigation; during the flight, the Yolov5 algorithm in step 3.1 is used to identify obstacles in the panoramic image, and the D*Lite algorithm is used according to the obstacle position information and the real-time positioning information of the aircraft. For the three-dimensional trajectory planning problem of the unmanned aerial vehicle when the target moves in an uncertain environment, the D*Lite search algorithm is used to perform rapid three-dimensional trajectory planning to achieve autonomous obstacle avoidance during flight.

[0029] The present invention makes the following improvements on traditional UAV flight technology: it is equipped with a panoramic vision sensor to instantly obtain rich large-scale environmental information, and combines it with SLAM technology to improve positioning accuracy; it applies target recognition technology to panoramic images to achieve real-time target recognition in a large-scale environment, and according to the self-positioning information output by SLAM and the identified obstacle and target information, it completes highly robust autonomous obstacle avoidance flight through path planning technology.

[0030] The present invention has the following advantages: 1. The UAV platform adopts a small-wheelbase four-rotor UAV, and the core sensor is a panoramic camera, which is small in size, low in cost, highly maneuverable and flexible, and applicable to a wide range of scenarios. It is a set of lightweight, portable, iteratively updateable autonomous UAV platform system. 2. The combination of panoramic camera and SLAM technology can obtain complete information about the environment in all directions and in real time, and achieve high-precision self-positioning. 3. The use of panoramic images for target recognition effectively improves the real-time and accuracy of recognition and increases the information dimension. 4. According to the positioning information and target recognition information, real-time autonomous obstacle avoidance is performed to achieve rapid obstacle avoidance and plan the optimal path. 5. The entire environmental perception and autonomous obstacle avoidance process is automated and intelligent, without the need for human intervention, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 : Flow chart of the overall method of the present invention.

[0032] Figure 2 : Schematic diagram of the UAV platform.

[0033] Figure 3 : Panoramic SLAM technology flow chart.

[0034] Figure 4 : Schematic diagram of spherical imaging.

[0035] Figure 5 : Overall flow chart of recognition and obstacle avoidance.

[0036] Figure 6 : Target recognition technology flow chart.

[0037] Figure 7 : Autonomous obstacle avoidance technology flow chart. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clear, the present application is further described in detail below in conjunction with specific examples. It should be understood that the specific examples described here are only used to explain the present application and are not used to limit the present application.

[0039] Combine the following Figures 1 to 7 The specific implementation of the present invention is a drone autonomous flight technology that integrates panoramic SLAM and target recognition. The overall method flow chart of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0040] Step 1: Design a small drone platform. Design the frame structure and complete the hardware construction according to the weight and size of the panoramic camera and other payloads. Figure 2 As shown in the figure, the specific design of the hardware platform in this example is divided into the following three parts:

[0041] (1) The frame is made of 3K carbon fiber material with an integrated design. Common external parts (such as blade protectors, landing and take-off tripods, etc.) are integrated with the horizontal structure of the fuselage. The thickness of the integrated plate of the arm frame is 3mm, the thickness of the blade protector is 2mm, and the landing and take-off tripod is perpendicular to the horizontal surface at an angle of 30 degrees. It is made of aluminum alloy and has a shock-absorbing rubber pad installed at the end that contacts the ground.

[0042] (2) The vertical structure of the frame is divided into three layers, and the layers are tightly connected by light-weight and high-strength copper single-pass columns and aluminum hollow threaded columns. The spacing between the first and second layers is 25mm, and the spacing between the second and third layers is 30mm. From bottom to top, the first layer is only extended on the integrated board layer of the arm keel, and the PixRacer flight controller is installed in a convex manner to facilitate the wiring of peripheral devices. The ESP-8266 wireless network module is connected to the top layer of the flight controller for wireless flight controller parameter detection and debugging; the second layer is installed with motors, UBEC power modules, electronic speed regulators, propellers and other power devices as well as panoramic cameras and GNSS receivers. The lower layer of the blade protector is on the same plane as the second layer. The telemetry signal receiver and BeneWake TFmini-S laser TOF radar are installed on the panel close to the ground on the second layer. Among them, TOF radars are connected to the flight controller through the UART interface, the electronic speed regulator is connected to the corresponding AUX interface of the flight controller after channel separation, the telemetry signal receiver is connected to the SBUS interface of the flight controller, and the camera is connected to the airborne computer through the USB interface; the third layer is similar to the first layer, only expanded on the arm keel integrated board, mainly installing the airborne microcomputer, USB expansion module and serial communication module.

[0043] (3) In terms of the ratio of chord length to load, compared with the common UAV flight platform with a wheelbase of 450mm or more, I reduced the wheelbase to 290mm while still maintaining the load-bearing performance. After arranging all the necessary equipment, the flight platform weighs about 1.1kg and can still carry an additional load of about 0.5kg. In terms of numerical value, it can carry an additional load of 45.4%.

[0044] (4) In terms of power layout, considering the high load and low wheelbase characteristics of this model, it is necessary to select a motor that can provide sufficient torque. A combination of a 26.8 mm 1750KV brushless motor and a 6-inch three-blade propeller is used. In terms of system power supply, a 4S 3300mAh 25CLi-PO battery is used as the power source and a PM02V3 splitter ammeter and a dual-channel UBEC separate power supply are used to effectively avoid the impact of changes in electrical power consumption on the flight controller or onboard computer.

[0045] A small wheelbase quad-rotor drone is selected as the flight platform, which is light, convenient, and stable in flight. A panoramic camera is installed on the front of the drone platform to effectively obtain 360° viewing angle information; the video information is transmitted to the onboard computer in real time for real-time calculation, so as to obtain the flight attitude of the drone and the information of the target of interest in the scene; the flight attitude data is then transmitted to the flight control system to autonomously control the flight of the drone; the whole system is provided with stable power by 14.8V / 16000mAh to ensure the normal flight of the drone. After the construction of this system is completed, environmental images can be collected, real-time panoramic SLAM can be run, and the positioning and autonomous flight of the drone can be completed.

[0046] Step 2: Figure 3 The panoramic SLAM technology shown first constructs an imaging model of the panoramic image to achieve efficient stitching of the panoramic image, then extracts and matches the image features, and finally completes the positioning through pose estimation and optimization. The specific process is as follows:

[0047] Step 2.1: First, establish the panoramic camera coordinate system, abstract the panoramic camera on the UAV platform (the panoramic camera is used to obtain a 360° panoramic image of the surrounding environment, and the subsequent algorithm processes the image to complete the positioning) into a spherical camera model, consider the single-lens case, and make the center of the sphere coincide with the optical center of the camera. Figure 4 As shown, O-xyz is the camera coordinate system. For the object point P, it is mapped to point p on the image plane, and the corresponding coordinates in the image coordinate system are (u, v). At the same time, the light OP intersects the sphere with O as the center at point P. s .

[0048] Define spherical mapping as a function mapping relationship Map any point p(u,v) on the image to a sphere with a certain radius and express it in spherical coordinates as To use the right-hand rule, the thumb points toward the y-axis, and the x-axis and z-axis rotate by an angle θ toward the direction of the other fingers; For the right-hand rule, the thumb points toward the z-axis, and the x-axis and y-axis rotate in the direction of the other fingers. Angle value.

[0049] Let α be the vector The angle between the projection and the axis on the plane O-yz, β is the vector The angle with the plane O-yz. For the actual image, the coordinates of the u-axis and v-axis are finite values, so the range of α and β is (-π / 2, π / 2). According to the spatial geometric relationship, there is formula (2.1) as follows:

[0050]

[0051] Where f is the focal length of the camera, (u 0 ,v 0 ) is the principal point offset. In fact, the above formula provides a mapping relationship from the image coordinate system to the local angle in the sphere That is, there is a one-to-one correspondence between points on the image plane and some points on the spherical surface.

[0052] In a multi-lens panoramic camera, each sub-camera has an independent camera coordinate system. The coordinate system of a multi-lens panoramic camera is based on the single-lens spherical coordinate system. However, due to the process and the volume of the lens itself, the optical centers of the lenses do not coincide. Therefore, when performing spherical mapping on the collected image data, it is necessary to unify the sub-cameras of the multi-lens panoramic camera into an overall coordinate system, and then convert the pixel coordinates of the images collected by each lens into the overall coordinate system through formula 2.1, and perform spherical mapping uniformly.

[0053] Step 2.2: Based on mapping the panoramic image to the spherical model, the spherical image is feature extracted and matched based on the SPHORB algorithm. After the feature points are extracted from the spherical image, there will be pairs of feature points with the same name in the two images (i.e., a feature point appears in both images). At this time, the correspondence between the points with the same name is determined by iteratively calculating the distance between the points. At this time, the matching is completed. The relative transformation matrix between the two is solved according to the coordinate relationship between the points with the same name, and it is transformed into a unified coordinate system to complete the stitching. The idea of ​​this algorithm is to first approximate the spherical image to obtain a hexagonal spherical grid similar to a football, and then directly construct fine-grained pyramids and robust features on the hexagonal spherical grid, thereby avoiding the time-consuming calculation of spherical harmonics and their related bandwidth limitations, and it has the scale and rotation invariance of spherical features.

[0054] Step 2.3: After feature extraction and matching, a large number of feature points are obtained. The position and posture transformation relationship between adjacent frames is solved by the epipolar constraint method (the position and posture transformation relationship refers to t k At (x k ,y k ,z k ) k ,pitch k ,yaw k ) to obtain a frame of image, at t k+1 At (x k+1 ,y k+1 ,z k+1 ) k+1 ,pitch k+1 ,yaw k+1 )’s attitude angle acquires a frame of image. The position and attitude change between two moments is called attitude transformation). A nonlinear optimization method is used to optimize the three-dimensional coordinates of the feature points and the camera’s attitude at the same time to minimize the reprojection error, thereby outputting the optimal camera attitude and determining the camera position.

[0055] Step 3: In step 2, the drone completes positioning and obtains real-time panoramic images. Through target recognition, it detects obstacles and targets in the surrounding environment in real time, plans the best feasible path, and completes autonomous obstacle avoidance flight. Figure 5 As shown, the specific steps are as follows:

[0056] Step 3.1: The technical process of target identification is as follows Figure 6As shown in the figure, it mainly consists of making target sample data sets, building neural network models, training models and real-time recognition. Considering the limited computing unit capacity and operation efficiency on light and small drones, the YOLOv5 algorithm was selected as the target recognition algorithm. The network framework is divided into 4 common modules, including: input end, benchmark network, Neck network and Head output end. The input end represents the input image. The input image size represented by this network is 608*608. At this stage, the input image can be scaled to the input size of the network and normalized. In the network training stage, YOLOv5 uses Mosaic data enhancement operations to improve the training speed of the model and the accuracy of the network; and proposes an adaptive anchor box calculation and adaptive image scaling method. The benchmark network represents a classifier network with excellent performance. This module is used to extract general feature representations. CSPDarknet53 and Focus structures are used as benchmark networks. Among them, the CSPDarknet53 structure draws on the design ideas of CSPNet and is designed to increase the feature representation dimension in the backbone network. The Focus structure mainly crops the input image through slice operations. The original input image size is 608*608*3. After Slice and Concat operations, a 304*304*12 feature map is output; then after a Conv layer with 32 channels, a 304*304*32 feature map is output. The Neck network is located in the middle of the baseline network and the head network. It draws on the CSP2 structure designed by CSPnet, improves the SPP module and FPN+PAN module, and thus enhances the network feature fusion capability. The Head output segment is used to complete the output of the target recognition result. Yolov5 uses the GIOU_Loss function, which solves the problem of non-overlapping bounding boxes based on IOU, thereby further improving the detection accuracy of the algorithm.

[0057] Step 3.2: Based on the real-time detection of surrounding obstacles in step 3.1, the drone performs path planning, navigates and flies along the optimal obstacle avoidance path, and completes autonomous obstacle avoidance. The overall obstacle avoidance plan is as follows: Figure 7 As shown, first, the time difference and posture change between two adjacent frames obtained in step 2.3 are used to infer the speed, acceleration and angular velocity of the aircraft, and the Kalman filter navigation algorithm is used to dynamically track the posture of the aircraft to achieve autonomous navigation. During the flight, the Yolov5 algorithm in step 3.1 is used to identify obstacles in the panoramic image. According to the obstacle position information and the real-time positioning information of the aircraft, the D*Lite algorithm is used to solve the three-dimensional trajectory planning problem of the unmanned aerial vehicle when the target moves in an uncertain environment. The D*Lite search algorithm is used to perform rapid three-dimensional trajectory planning to achieve autonomous obstacle avoidance during flight.

[0058] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0059] It should be understood that the above description of the preferred embodiment is relatively specific and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also independently build a variety of drone platforms for different usage scenarios without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A UAV autonomous flight technology method integrating panoramic SLAM and target recognition, It is characterized in that The following steps are involved: Step 1: Build a small drone platform; select a small-wheelbase quadcopter drone as the flight platform, and install a panoramic camera on the front of the drone platform to effectively obtain 360° viewing angle information; transmit the video information to the onboard computer in real time for real-time calculation, so as to obtain the drone's flight attitude and the target of interest in the scene; then transmit the flight attitude data to the flight control system to autonomously control the drone's flight; Step 2: Use the panoramic camera to collect environmental images, run real-time panoramic SLAM, and complete the positioning and autonomous flight of the drone; The specific implementation of step 2 includes the following sub-steps: Step 2.1, constructing an imaging model of a panoramic image, including a single-lens imaging model and a multi-lens imaging model, to achieve efficient stitching of panoramic images and complete spherical mapping of image data; Step 2.2, using the SPHORB algorithm to perform feature extraction and matching stitching on the spherical image mapped in step 2.1; Step 2.3, based on the feature point pairs extracted in step 2.2, solve the pose transformation relationship between adjacent frames through epipolar constraints, and use nonlinear optimization to optimize the three-dimensional coordinates of the feature points and the camera pose to minimize the reprojection error, output the optimal camera pose to complete positioning, and transform and splice the feature points with this pose to complete the map construction; Step 3: Based on target recognition technology, detect objects in real-time images, combine the positioning information output by SLAM, plan the optimal path with the target as the end point, and complete autonomous obstacle avoidance flight; The specific implementation of step 3 is as follows; Step 3.1, using the YOLOv5 algorithm as the target recognition algorithm, inputting the target data set of the panoramic image to train the neural network model, using the trained model to perform real-time target recognition on the panoramic image transmitted in step 2, and outputting the recognition result; Step 3.2, using the time difference between adjacent frames and the posture transformation obtained in step 2.3, the aircraft speed, acceleration and angular velocity are calculated, and the aircraft posture is dynamically tracked through the Kalman filter navigation algorithm; during the flight, according to the obstacle position information identified in step 3.1 and the real-time positioning information of the aircraft, the D*Lite search algorithm is used to perform three-dimensional track planning to achieve real-time autonomous obstacle avoidance.

2. According to claim 1, a method for autonomous flight of unmanned aerial vehicles integrating panoramic SLAM and target recognition, Features: The specific implementation of step 2.1 is as follows; First, a panoramic camera coordinate system is established. The panoramic camera on the UAV platform is abstracted as a spherical camera model. Considering the single-lens case, the center of the sphere is made to coincide with the optical center of the camera. Let O-xyz be the camera coordinate system. For the object point P, it is mapped to point p on the image plane, and the corresponding coordinates in the image coordinate system are (u, v); at the same time, the light OP intersects the sphere with O as the center at point P. s ; Define the spherical map as a function mapping relationship f s : Map any point p(u,v) on the image to a sphere with a certain radius and express it in spherical coordinates as To use the right-hand rule, the thumb points toward the y-axis, and the x-axis and z-axis rotate by an angle θ toward the direction of the other fingers; For the right-hand rule, the thumb points toward the z-axis, and the x-axis and y-axis rotate in the direction of the other fingers. Angle value; Let α be the vector The angle between the projection and the axis on the plane O-yz, β is the vector The angle between the plane O-yz and the image; for the actual image, the coordinates of the u-axis and v-axis are finite values, so the value range of α and β is (-π / 2,π / 2). According to the spatial geometric relationship, the formula (2.1) is as follows: Where f is the focal length of the camera, (u 0 ,v 0 ) is the main point bias; in fact, the above formula provides a mapping relationship from the image coordinate system to the local angle in the sphere {[uv]′}→{[αβ]′}, which is the one-to-one correspondence between points on the image plane and some points on the sphere; In a multi-lens panoramic camera, each sub-camera has an independent camera coordinate system, and the coordinate system of the multi-lens panoramic camera is based on the single-lens spherical coordinate system. However, due to the process and the volume of the lens itself, the optical centers of the lenses do not coincide. Therefore, when performing spherical mapping on the collected image data, it is necessary to unify the sub-cameras of the multi-lens panoramic camera into an overall coordinate system, and then convert the pixel coordinates of the images collected by each lens into the overall coordinate system through formula 2.1, and perform spherical mapping uniformly.

3. The method for autonomous flight of unmanned aerial vehicle integrating panoramic SLAM and target recognition according to claim 1, Features: In step 3.1, the YOLOv5 network model includes the input end, the baseline network, the Neck network and the Head output end; the input end represents the input image, and the input image size is 608*608. At this stage, the input image can be scaled to the input size of the network and normalized; In the network training phase, YOLOv5 uses Mosaic data augmentation operations to improve the model training speed and network accuracy; the benchmark network represents a classifier network with excellent performance, which is used to extract general feature representations. CSPDarknet53 and Focus structures are used as benchmark networks; The CSPDarknet53 structure draws on the design ideas of CSPNet and is designed to improve the feature representation dimension in the backbone network. The Focus structure crops the input image through the slice operation. The original input image size is 608*608*3. After the Slice and Concat operations, a 304*304*12 feature map is output; then it passes through a Conv layer with 32 channels and outputs a 304*304*32 feature map; the Neck network is located in the middle of the benchmark network and the Head output end. Drawing on the CSP2 structure designed by CSPnet, the SPP module and FPN+PAN module are improved to enhance the network feature fusion capability; The Head output is used to complete the output of the target recognition results; Yolov5 uses the GIOU_Loss function, which solves the problem that occurs when the bounding boxes do not overlap based on IOU, thereby further improving the detection accuracy of the algorithm.

4. The method for autonomous flight of unmanned aerial vehicle integrating panoramic SLAM and target recognition according to claim 3, Features: The specific implementation of step 3.2 is as follows; First, the time difference and posture change between two adjacent frames obtained in step 2.3 are used to infer the speed, acceleration and angular velocity of the aircraft, and the Kalman filter navigation algorithm is used to dynamically track the aircraft posture to achieve autonomous navigation; during the flight, the Yolov5 algorithm in step 3.1 is used to identify obstacles in the panoramic image, and the D*Lite algorithm is used according to the obstacle position information and the real-time positioning information of the aircraft. For the three-dimensional trajectory planning problem of the unmanned aerial vehicle when the target moves in an uncertain environment, the D*Lite search algorithm is used to perform rapid three-dimensional trajectory planning to achieve autonomous obstacle avoidance during flight.

Images