Method and system for realizing double-screen display of unmanned aerial vehicle based on embedded map

Abstract

The invention relates to the technical field of unmanned aerial vehicles, and discloses a method and system for realizing double-screen display of an unmanned aerial vehicle based on an embedded map, and the method comprises the steps: calibrating flight data and environment perception data, and obtaining calibrated flight data and calibrated environment perception data; marking an unmanned aerial vehicle position and an unmanned aerial vehicle flight path of the unmanned aerial vehicle, and marking obstacle elements of a flight area corresponding to the unmanned aerial vehicle; calculating obstacle coefficients of the obstacle elements, and performing three-dimensional reconstruction on the obstacle elements in the embedded map to obtain an embedded obstacle map; establishing an obstacle avoidance path of the unmanned aerial vehicle, and constructing a first display picture of the unmanned aerial vehicle; and constructing an identification frame and attribute information of the video main body, constructing a second display picture of the unmanned aerial vehicle according to the video main body, the identification frame and the attribute information, establishing a double-screen cooperation rule of the unmanned aerial vehicle, and executing double-screen display of the first display picture and the second display picture according to the double-screen cooperation rule. The operation efficiency of the unmanned aerial vehicle in complex tasks can be improved.

Description

Technical Field

[0001] The invention relates to a method and system for realizing dual-screen display of an unmanned aerial vehicle based on an embedded map, and belongs to the technical field of unmanned aerial vehicles. Background Art

[0002] Drone dual-screen display refers to the use of two screens in a drone system to simultaneously display different information or different perspectives of the same information. The drone dual-screen display system improves operational efficiency and safety, enabling drone operators to better perform complex tasks and make quick decisions.

[0003] The traditional dual-screen display method of drones usually has one screen displaying flight data and map information according to a pre-set flight path, and the other screen displaying real-time video from the camera. This cannot integrate environmental perception data and real-time analysis, cannot provide more comprehensive situational awareness, lacks real-time data fusion, and cannot effectively display obstacle avoidance paths and obstacle information, resulting in poor operating efficiency and safety in complex tasks. Summary of the invention

[0004] The present invention provides a method and system for realizing dual-screen display of unmanned aerial vehicles based on embedded maps, the main purpose of which is to improve the operating efficiency of unmanned aerial vehicles in complex tasks.

[0005] To achieve the above object, the present invention provides a method for realizing dual-screen display of a drone based on an embedded map, comprising: Acquire flight data and environmental perception data of the UAV in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; Based on the calibration flight data, marking the drone position and the drone flight trajectory of the drone, and based on the calibration environment perception data, marking obstacle elements in the corresponding flight area of ​​the drone; Calculating an obstacle coefficient of the obstacle element, and performing three-dimensional reconstruction of the obstacle element in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map; Establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map, and constructing a first display screen for the drone according to the drone position, the drone flight trajectory, the embedded obstacle map, and the obstacle avoidance path; Acquire real-time flight video of the drone, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the drone based on the video subject, the identification frame and the attribute information, establish a dual-screen collaboration rule for the drone, and execute dual-screen display of the first display screen and the second display screen based on the dual-screen collaboration rule.

[0006] Optionally, calibrating the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes: Identify sensor types for the flight data and environmental perception data; analyzing flight data characteristics and environmental perception data characteristics of the flight data and environmental perception data; Based on the sensor type, the flight data characteristics, and the environmental perception data characteristics, defining a joint calibration algorithm for the flight data and the environmental perception data; Determining a feasible domain of parameters of the joint calibration algorithm; The flight data and the environmental perception data are calibrated through the parameter feasible domain and using the joint calibration algorithm to obtain the calibrated flight data and the calibrated environmental perception data.

[0007] Optionally, marking a drone position and a drone flight trajectory of the drone based on the calibration flight data comprises: extracting position information from the calibration flight data, and determining a drone position of the drone based on the position information; Marking the starting point of the trajectory of the UAV; According to the trajectory starting point, defining the trajectory data structure of the UAV; Determine the trajectory point of the drone through the location information; Adding the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the UAV; The initial flight trajectory is smoothed to obtain the UAV flight trajectory of the UAV.

[0008] Optionally, smoothing the initial flight trajectory to obtain the drone flight trajectory of the drone includes: defining a sliding window of the initial flight trajectory; Determining trajectory point weights of trajectory points corresponding to the initial flight trajectory; According to the sliding window and the weight of the trajectory point, the smoothing coefficient of the trajectory point is calculated using the following formula: ; in, Indicates at a point in time The smoothing coefficient corresponding to the trajectory point, Indicates time point , represents the size of the sliding window, Indicates at a point in time The trajectory point, Indicates at a point in time The trajectory point weight corresponding to the trajectory point; The initial flight trajectory is smoothed based on the smoothing coefficient to obtain a UAV flight trajectory of the UAV.

[0009] Optionally, marking obstacle elements in a flight area corresponding to the drone based on the calibration environment perception data includes: Determining a regional RGB image, 3D point cloud data, and detecting heat sources of the flight area based on the calibrated environmental perception data; Performing subject separation on the regional RGB image to obtain the image subject; Supplementing depth information of the image subject by using the 3D point cloud data to obtain a supplemented image subject; Extracting image subject features and heat source features of the supplementary image subject and the detected heat source; Based on the main features and heat source features of the image, obstacle elements in the flight area are identified using a trained obstacle recognition model.

[0010] Optionally, the calculating the obstacle coefficient of the obstacle element includes: Identifying the obstacle type of the obstacle element, wherein the obstacle type includes static obstacles, dynamic obstacles, and living obstacles; Extracting obstacle features of the obstacle type, wherein the obstacle features include obstacle distance, obstacle volume, obstacle dynamics, and heat source value; an obstacle characteristic score defining the obstacle characteristic; Normalizing the obstacle features to obtain normalized obstacle features, wherein the normalized obstacle features include normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, and normalized heat source value; The obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score.

[0011] Optionally, calculating the obstacle coefficient of the obstacle element according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score includes: Defining environmental impact factors of the obstacle elements, wherein the environmental impact factors include wind speed and light conditions; Based on the wind speed, the lighting condition, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score, the obstacle coefficient of the obstacle element is calculated using the following formula: ; in, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, The obstacle feature score representing the normalized obstacle distance, represents the normalized obstacle volume, The obstacle feature score representing the normalized obstacle volume, represents the normalized obstacle dynamics, The obstacle feature score representing the normalized obstacle dynamics, represents the normalized heat source value, The obstacle feature score representing the normalized heat source value, Indicates wind speed, The influence weight of wind speed is shown. Indicates the lighting conditions, Represents the influence weight of lighting conditions.

[0012] Optionally, the three-dimensionally reconstructing the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map includes: Determining the element to be modeled of the obstacle element according to the obstacle coefficient; Collecting element point cloud data of the element to be modeled; Denoising the element point cloud data to obtain denoised element point cloud data; determining a reference point cloud of the embedded map; Defining a rotation matrix of the denoised element point cloud data in the embedded map; Based on the reference point cloud, the rotation matrix and the element point cloud data, the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map is calculated using the following formula: ; in, Indicates The map pose of the sensor in the embedded map is Indicates the first The corresponding point position of the denoised element point cloud data in the reference point cloud, Indicates the first Noise-reduced element point cloud data, represents the rotation matrix, represents the translation vector; According to the map posture and the noise reduction element point cloud data, the obstacle elements are three-dimensionally reconstructed in the embedded map to obtain an embedded obstacle map.

[0013] Optionally, establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map includes: Marking potential waypoints of the embedded obstacle map according to the drone destination and the drone position; Determining a current waypoint of the potential waypoint based on the drone position; Calculating a waypoint response value of the potential waypoint based on the embedded obstacle map and the UAV flight trajectory; The obstacle avoidance path of the UAV is determined by the response value.

[0014] In order to solve the above problems, the present invention also provides a dual-screen display system for drones based on embedded maps, the system comprising: A drone data acquisition module is used to acquire the flight data and environmental perception data of the drone in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; An obstacle element marking module, used to mark the drone position and the drone flight trajectory of the drone based on the calibration flight data, and mark the obstacle elements of the drone corresponding to the flight area based on the calibration environment perception data; An obstacle map construction module is used to calculate the obstacle coefficient of the obstacle element, and according to the obstacle coefficient, perform three-dimensional reconstruction of the obstacle element in the embedded map to obtain an embedded obstacle map; A first display screen establishing module, used to establish an obstacle avoidance path of the drone according to a preset drone target location, the drone position, the drone flight trajectory and the embedded obstacle map, and to construct a first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map and the obstacle avoidance path; The second display screen establishment module is used to obtain the real-time flight video of the UAV, identify the video body of the real-time flight video, construct an identification frame and attribute information of the video body, construct the second display screen of the UAV according to the video body, the identification frame and the attribute information, establish the dual-screen coordination rules of the UAV, and execute the dual-screen display of the first display screen and the second display screen according to the dual-screen coordination rules.

[0015] Compared with the problems described in the background technology, firstly, by acquiring and calibrating the flight data and environmental perception data of the UAV, the system ensures the accuracy and reliability of the data, providing a solid foundation for subsequent flight decisions. The precise marking of position, altitude, speed and attitude information enables the operator to grasp the flight status of the UAV in real time, and the calibration of lidar point cloud data, infrared image data and ultrasonic sensor data improves the accuracy of environmental perception and effectively avoids potential collision risks. Secondly, the three-dimensional reconstruction of the embedded obstacle map provides a detailed environmental model for the UAV, enabling the UAV to perform more intelligent path planning and obstacle avoidance in complex environments. The calculation of the obstacle coefficient further enhances the pertinence of the obstacle avoidance strategy and ensures It ensures the safe flight of the drone when facing obstacles of different difficulty levels. Furthermore, the construction of the first display screen makes the drone's flight trajectory, position, obstacle elements and obstacle avoidance path clear at a glance, greatly improving the operator's situational awareness and facilitating effective flight command and decision-making. In addition, the real-time flight video analysis and video subject recognition of the second display screen provide powerful visual assistance for the drone to perform specific tasks (such as monitoring, search, rescue, etc.). The real-time display of the recognition frame and attribute information enables the operator to quickly identify and respond to key targets in the video. Finally, the establishment of dual-screen collaborative rules realizes the seamless docking and collaborative work of the first display screen and the second display screen, improves operational efficiency and reduces operational complexity. Therefore, the present invention improves the operational efficiency of drones in complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of a method for realizing dual-screen display of a drone based on an embedded map provided in one embodiment of the present invention; Figure 2 A schematic diagram of a module for implementing the method for realizing dual-screen display of a drone based on an embedded map provided in one embodiment of the present invention.

[0017] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0019] The embodiment of the present application provides a method for realizing dual-screen display of drones based on embedded maps. The execution subject of the method for realizing dual-screen display of drones based on embedded maps includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for realizing dual-screen display of drones based on embedded maps can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0020] Embodiment 1: Reference Figure 1 FIG. 1 is a flow chart of a method for realizing dual-screen display of a drone based on an embedded map according to an embodiment of the present invention. In this embodiment, the method for realizing dual-screen display of a drone based on an embedded map includes: S1. Obtain flight data and environmental perception data of the UAV in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes lidar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data.

[0021] It should be explained that the position refers to the specific coordinate position of the drone in the embedded map, the height refers to the vertical distance of the drone relative to the ground, the speed refers to the flight speed of the drone, and the attitude information refers to the direction and tilt state of the drone in three-dimensional space. The laser radar point cloud data refers to the data obtained by the laser radar (LiDAR) sensor, the infrared image data refers to the data captured by the infrared camera, and the ultrasonic sensor data refers to the data obtained by the ultrasonic sensor.

[0022] The present invention calibrates the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data, which can make the data more accurate and reliable and suitable for precise navigation and control of unmanned aerial vehicles.

[0023] In detail, the calibrating the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes: Identify sensor types for the flight data and environmental perception data; analyzing flight data characteristics and environmental perception data characteristics of the flight data and environmental perception data; Based on the sensor type, the flight data characteristics, and the environmental perception data characteristics, defining a joint calibration algorithm for the flight data and the environmental perception data; Determining a feasible domain of parameters of the joint calibration algorithm; The flight data and the environmental perception data are calibrated through the parameter feasible domain and using the joint calibration algorithm to obtain the calibrated flight data and the calibrated environmental perception data.

[0024] The sensor type refers to the type of sensor used to collect flight data and environmental perception data. This may include but is not limited to inertial measurement unit (IMU), global positioning system (GPS), radar, laser range finder (LIDAR), camera, ultrasonic sensor, etc. The flight data characteristics refer to the description of the quality and attributes of flight data, including but not limited to data accuracy, resolution, sampling rate, noise level, dynamic range, nonlinear error, system deviation, time drift, etc. The environmental perception data characteristics refer to the quality and attributes of data collected by environmental perception sensors, such as resolution, accuracy, range, viewing angle, occlusion, reflectivity, environmental interference, etc. The joint calibration algorithm refers to an algorithm used to simultaneously calibrate flight data and environmental perception data to reduce systematic errors and deviations between different sensors. The parameter feasible domain refers to the range of acceptable parameter values ​​in the joint calibration algorithm. The calibrated flight data refers to flight data processed by the joint calibration algorithm, and the calibrated environmental perception data refers to environmental perception data processed by the joint calibration algorithm.

[0025] Optionally, the flight data and environmental perception data are calibrated through the parameter feasible domain using the joint calibration algorithm, and the calibrated flight data and calibrated environmental perception data can be jointly processed by statistical methods (such as least squares method) and filtering techniques (such as Kalman filtering).

[0026] S2. Based on the calibration flight data, mark the drone position and drone flight trajectory of the drone, and based on the calibration environment perception data, mark obstacle elements in the corresponding flight area of ​​the drone.

[0027] The present invention marks the drone position and the drone flight trajectory of the drone based on the calibration flight data, which can provide a basis for subsequent drone flight optimization.

[0028] In detail, the marking the drone position and the drone flight trajectory of the drone based on the calibration flight data includes: extracting position information from the calibration flight data, and determining a drone position of the drone based on the position information; Marking the starting point of the trajectory of the UAV; According to the trajectory starting point, defining the trajectory data structure of the UAV; Determine the trajectory point of the drone through the location information; Adding the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the UAV; The initial flight trajectory is smoothed to obtain the UAV flight trajectory of the UAV.

[0029] Among them, the position information refers to the specific coordinate information of the UAV extracted from the calibration flight data, usually including the position in two-dimensional or three-dimensional space, such as longitude and latitude, altitude; the UAV position refers to a specific position information point, which represents the exact position of the UAV at a certain moment; the trajectory starting point refers to the starting point of the UAV flight trajectory, usually the position when the UAV takes off; the trajectory data structure refers to a data structure used to store and represent the UAV flight trajectory, which can be an array, a list, or a database; the trajectory point refers to a single data point in the flight trajectory, each point contains the UAV position information and the corresponding timestamp; the initial flight trajectory refers to the flight trajectory preliminarily constructed based on the extracted position information; the UAV flight trajectory refers to the final flight trajectory after smoothing and optimization.

[0030] Furthermore, the smoothing of the initial flight trajectory to obtain the UAV flight trajectory of the UAV includes: defining a sliding window of the initial flight trajectory; Determining trajectory point weights of trajectory points corresponding to the initial flight trajectory; According to the sliding window and the weight of the trajectory point, the smoothing coefficient of the trajectory point is calculated using the following formula: ; in, Indicates at a point in time The smoothing coefficient corresponding to the trajectory point, Indicates time point , represents the size of the sliding window, Indicates at a point in time The trajectory point, Indicates at a point in time The trajectory point weight corresponding to the trajectory point; The initial flight trajectory is smoothed based on the smoothing coefficient to obtain a UAV flight trajectory of the UAV.

[0031] Among them, the sliding window refers to a data window of a fixed size, which slides on the initial flight trajectory, the trajectory point weight refers to the numerical value assigned to each trajectory point, which indicates the importance of the trajectory point relative to other trajectory points in the smoothing process, and the smoothing coefficient refers to the numerical value used to adjust the contribution of each trajectory point to the final smooth trajectory.

[0032] The present invention marks obstacle elements in the flight area corresponding to the drone based on the calibration environment perception data, which can effectively mark obstacle elements in the flight area of ​​the drone and ensure flight safety.

[0033] In detail, marking obstacle elements in the corresponding flight area of ​​the drone based on the calibration environment perception data includes: Determining a regional RGB image, 3D point cloud data, and detecting heat sources of the flight area based on the calibrated environmental perception data; Performing subject separation on the regional RGB image to obtain the image subject; Supplementing depth information of the image subject by using the 3D point cloud data to obtain a supplemented image subject; Extracting image subject features and heat source features of the supplementary image subject and the detected heat source; Based on the main features and heat source features of the image, obstacle elements in the flight area are identified using a trained obstacle recognition model.

[0034] Among them, the regional RGB image refers to a color image of the flight area taken by an RGB camera carried by the drone, the 3D point cloud data refers to the three-dimensional point cloud data of the flight area obtained by LiDAR or other depth sensors, the detected heat source refers to the heat source information in the flight area detected by an infrared sensor, the main target or area separated from the regional RGB image by the image subject is usually a potential obstacle (such as a building, a tree, a vehicle, etc.), the supplementary image subject refers to combining the depth information of the 3D point cloud data with the image subject in the RGB image to generate a subject image with depth information, the image subject feature refers to the feature extracted from the supplementary image subject, including shape, texture, color, depth and other information, the heat source feature refers to the feature extracted from the detected heat source, including temperature distribution, shape, size and other information of the heat source, the obstacle element refers to an object or area in the flight area that may hinder the flight of the drone, such as a building, a tree, a wire, a vehicle, a pedestrian, etc., and the obstacle recognition model refers to a model used to identify obstacle information, which can be trained using a neural network.

[0035] Optionally, the using of the 3D point cloud data to supplement the depth information of the image body to obtain the supplemented image body may be implemented by using OpenCV and Open3D software.

[0036] S3. Calculate the obstacle coefficient of the obstacle element, and perform three-dimensional reconstruction of the obstacle element in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map.

[0037] The obstacle coefficient of the obstacle element calculated by the present invention can provide a basis for optimizing the aircraft path of the unmanned aerial vehicle, thereby improving the safety of the flight of the unmanned aerial vehicle.

[0038] In detail, the calculating the obstacle coefficient of the obstacle element includes: Identifying the obstacle type of the obstacle element, wherein the obstacle type includes static obstacles, dynamic obstacles, and living obstacles; Extracting obstacle features of the obstacle type, wherein the obstacle features include obstacle distance, obstacle volume, obstacle dynamics, and heat source value; an obstacle characteristic score defining the obstacle characteristic; Normalizing the obstacle features to obtain normalized obstacle features, wherein the normalized obstacle features include normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, and normalized heat source value; The obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score.

[0039] The static obstacle refers to an obstacle with a fixed position, such as a building, a tree, a telephone pole, etc. The dynamic obstacle refers to an obstacle with a position that changes over time, such as a vehicle, a drone, a mobile device, etc. The living obstacle refers to an obstacle with life characteristics, such as a bird, an animal, a pedestrian, etc. The obstacle distance refers to the straight-line distance between the obstacle and the drone. The obstacle volume refers to the volume of the obstacle in three-dimensional space. The obstacle dynamics refers to the motion state of the obstacle (such as speed, direction). The heat source value refers to the heat radiation intensity of the obstacle (such as temperature value). The normalized obstacle distance refers to the distance value normalized to the range of [0,1][0,1]. The normalized obstacle volume refers to the value normalized to the range of [0,1][0,1]. The normalized obstacle dynamics refers to the value normalized to the range of [0,1][0,1]. The normalized heat source value refers to the value normalized to the range of [0,1][0,1]. The obstacle feature score refers to the score assigned to each obstacle feature, reflecting the influence of the feature on the obstacle coefficient. The obstacle coefficient refers to the indicator calculated by combining the obstacle feature score and the normalized obstacle feature, which is used to quantify the threat degree of obstacles to UAV flight.

[0040] Furthermore, the obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, including: Defining environmental impact factors of the obstacle elements, wherein the environmental impact factors include wind speed and light conditions; Based on the wind speed, the lighting condition, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score, the obstacle coefficient of the obstacle element is calculated using the following formula: ; in, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, The obstacle feature score representing the normalized obstacle distance, represents the normalized obstacle volume, The obstacle feature score representing the normalized obstacle volume, represents the normalized obstacle dynamics, The obstacle feature score representing the normalized obstacle dynamics, represents the normalized heat source value, The obstacle feature score representing the normalized heat source value, Indicates wind speed, The influence weight of wind speed is shown. Indicates the lighting conditions, Represents the influence weight of lighting conditions.

[0041] Among them, the wind speed refers to the air flow speed in the drone flight area, the lighting condition refers to the light intensity and direction in the drone flight area, and the impact weight refers to the influence of environmental impact factors on the drone flight process.

[0042] The present invention performs three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient, and obtains an embedded obstacle map that can provide a basis for obstacle avoidance of the unmanned aerial vehicle.

[0043] In detail, the three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient to obtain the embedded obstacle map includes: Determining the element to be modeled of the obstacle element according to the obstacle coefficient; Collecting element point cloud data of the element to be modeled; Denoising the element point cloud data to obtain denoised element point cloud data; determining a reference point cloud of the embedded map; Defining a rotation matrix of the denoised element point cloud data in the embedded map; Based on the reference point cloud, the rotation matrix and the element point cloud data, the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map is calculated using the following formula: ; in, Indicates The map pose of the sensor in the embedded map is Indicates the first The corresponding point position of the denoised element point cloud data in the reference point cloud, Indicates the first Noise-reduced element point cloud data, represents the rotation matrix, represents the translation vector; According to the map posture and the noise reduction element point cloud data, the obstacle elements are three-dimensionally reconstructed in the embedded map to obtain an embedded obstacle map.

[0044] Among them, the element to be modeled refers to an object identified as an obstacle in the flight area of ​​the drone, the element point cloud data refers to the point cloud data of the obstacle element to be modeled collected by a sensor (such as a lidar or an RGB-D camera), and these data contain the geometric information of the obstacle element, the denoised element point cloud data refers to the data obtained by denoising the original element point cloud data (such as removing outliers, smoothing, etc.), the purpose of which is to improve the accuracy of the data and the efficiency of subsequent processing, the reference point cloud refers to a known point cloud in the embedded map, which is usually part of the global map constructed by the SLAM algorithm and used as a reference for positioning and mapping, the rotation matrix refers to a 3x3 matrix used to describe the rotation relationship of the denoised element point cloud data relative to the reference point cloud, the map pose refers to the position and direction of the sensor in the embedded map, the embedded map refers to the original map used for drone flight navigation, and the embedded obstacle map refers to a three-dimensional map containing environmental obstacles and features, which is used for drone navigation and obstacle avoidance.

[0045] S4. Establish an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map; and construct a first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map, and the obstacle avoidance path.

[0046] The present invention establishes an obstacle avoidance path for the drone according to a preset drone target, the drone position, the drone flight trajectory and the embedded obstacle map to ensure the safety of the drone and the smooth completion of the mission.

[0047] In detail, the step of establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map includes: Marking potential waypoints of the embedded obstacle map according to the drone destination and the drone position; Determining a current waypoint of the potential waypoint based on the drone position; Calculating a waypoint response value of the potential waypoint based on the embedded obstacle map and the UAV flight trajectory; The obstacle avoidance path of the UAV is determined by the response value.

[0048] Among them, the potential path point refers to a series of pre-selected position points that the drone may pass through in the process of flying from the current position to the target destination, the current path point refers to the current path point of the drone, the path point response value refers to the evaluation index of the potential path point, which reflects the flight difficulty, safety, energy consumption or other costs from the current drone position to the path point, and the obstacle avoidance path refers to the safe flight path planned by the drone to avoid collision during flight.

[0049] Optionally, the calculation of the path point response value of the potential path point based on the embedded obstacle map and the UAV flight trajectory can be implemented using a cost map.

[0050] The present invention constructs the first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map and the obstacle avoidance path to obtain an intuitive display screen, so as to better monitor the flight status of the drone and perform obstacle avoidance operations. The first display screen refers to the display screen interface seen by the operator on the first screen in the drone control system or monitoring system, which includes a map view, a drone icon, a drone flight trajectory, an embedded obstacle map, an obstacle avoidance path and status information (drone speed, altitude, heading, battery power, signal strength and other key flight parameters).

[0051] S5. Acquire the real-time flight video of the UAV, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the UAV according to the video subject, the identification frame and the attribute information, establish a dual-screen collaboration rule for the UAV, and execute dual-screen display of the first display screen and the second display screen according to the dual-screen collaboration rule.

[0052] It should be explained that the real-time flight video refers to the image stream captured and transmitted in real time by the camera onboard the drone during the flight; the identification frame refers to a rectangular frame used to mark and locate specific objects (such as people, vehicles, animals, etc.) in the real-time flight video; the attribute information refers to a series of data descriptions related to the video subject in the identification frame, such as category, ID, behavior, distance and other descriptions; the second display screen refers to a display screen interface in the drone control system or monitoring system specifically used to display real-time flight video, identification frame and attribute information.

[0053] The invention establishes the dual-screen collaboration rules of the drone, which can be more efficient and safe, thereby improving the operator's work efficiency and the overall performance of the drone. The dual-screen collaboration rules include information synchronization rules, interaction design rules, visual coordination rules, warning and notification rules, user customization rules, and system integration rules.

[0054] Compared with the problems described in the background technology, firstly, by acquiring and calibrating the flight data and environmental perception data of the UAV, the system ensures the accuracy and reliability of the data, providing a solid foundation for subsequent flight decisions. The precise marking of position, altitude, speed and attitude information enables the operator to grasp the flight status of the UAV in real time, and the calibration of lidar point cloud data, infrared image data and ultrasonic sensor data improves the accuracy of environmental perception and effectively avoids potential collision risks. Secondly, the three-dimensional reconstruction of the embedded obstacle map provides a detailed environmental model for the UAV, enabling the UAV to perform more intelligent path planning and obstacle avoidance in complex environments. The calculation of the obstacle coefficient further enhances the pertinence of the obstacle avoidance strategy and ensures It ensures the safe flight of the drone when facing obstacles of different difficulty levels. Furthermore, the construction of the first display screen makes the drone's flight trajectory, position, obstacle elements and obstacle avoidance path clear at a glance, greatly improving the operator's situational awareness and facilitating effective flight command and decision-making. In addition, the real-time flight video analysis and video subject recognition of the second display screen provide powerful visual assistance for the drone to perform specific tasks (such as monitoring, search, rescue, etc.). The real-time display of the recognition frame and attribute information enables the operator to quickly identify and respond to key targets in the video. Finally, the establishment of dual-screen collaborative rules realizes the seamless docking and collaborative work of the first display screen and the second display screen, improves operational efficiency and reduces operational complexity. Therefore, the present invention improves the operational efficiency of drones in complex tasks.

[0055] Embodiment 2: like Figure 2 As shown, it is a functional module diagram of a dual-screen display system for unmanned aerial vehicles based on an embedded map according to the present invention.

[0056] The dual-screen display system 200 for drones based on embedded maps described in the present invention can be installed in an electronic device. According to the functions implemented, the dual-screen display system for drones based on embedded maps can include a drone data acquisition module 201, an obstacle element marking module 202, an obstacle map construction module 203, a first display screen establishment module 204, and a second display screen establishment module 205. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0057] In the embodiment of the present invention, the functions of each module / unit are as follows: The drone data acquisition module 201 is used to acquire the flight data and environmental perception data of the drone in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; The obstacle element marking module 202 is used to mark the drone position and the drone flight trajectory of the drone based on the calibration flight data, and mark the obstacle elements of the drone corresponding to the flight area based on the calibration environment perception data; The obstacle map construction module 203 is used to calculate the obstacle coefficient of the obstacle element, and perform three-dimensional reconstruction of the obstacle element in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map; The first display screen establishing module 204 is used to establish an obstacle avoidance path of the drone according to a preset drone target location, the drone position, the drone flight trajectory and the embedded obstacle map, and to construct a first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map and the obstacle avoidance path; The second display screen establishment module 205 is used to obtain the real-time flight video of the UAV, identify the video body of the real-time flight video, construct an identification frame and attribute information of the video body, construct the second display screen of the UAV according to the video body, the identification frame and the attribute information, establish the dual-screen coordination rules of the UAV, and perform dual-screen display of the first display screen and the second display screen according to the dual-screen coordination rules.

[0058] In detail, the modules in the drone dual-screen display system 200 based on embedded maps in the embodiment of the present invention are used in the same manner as described above. Figure 1The same technical means as described in the method for realizing dual-screen display of drones based on embedded maps can produce the same technical effects, so they will not be repeated here.

[0059] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for realizing dual-screen display of drone based on embedded map, characterized in that: The method comprises: Acquire flight data and environmental perception data of the UAV in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; Based on the calibration flight data, marking the drone position and the drone flight trajectory of the drone, and based on the calibration environment perception data, marking obstacle elements in the corresponding flight area of ​​the drone; Calculating an obstacle coefficient of the obstacle element, and performing three-dimensional reconstruction of the obstacle element in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map; Establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map, and constructing a first display screen for the drone according to the drone position, the drone flight trajectory, the embedded obstacle map, and the obstacle avoidance path; Acquire real-time flight video of the drone, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the drone based on the video subject, the identification frame and the attribute information, establish a dual-screen collaboration rule for the drone, and execute dual-screen display of the first display screen and the second display screen based on the dual-screen collaboration rule.

2. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 1, characterized in that: The step of calibrating the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes: Identify sensor types for the flight data and environmental perception data; analyzing flight data characteristics and environmental perception data characteristics of the flight data and environmental perception data; Based on the sensor type, the flight data characteristics, and the environmental perception data characteristics, defining a joint calibration algorithm for the flight data and the environmental perception data; Determining a feasible domain of parameters of the joint calibration algorithm; The flight data and the environmental perception data are calibrated through the parameter feasible domain and using the joint calibration algorithm to obtain the calibrated flight data and the calibrated environmental perception data.

3. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 2, characterized in that: The step of marking the drone position and the drone flight trajectory of the drone based on the calibration flight data comprises: extracting position information from the calibration flight data, and determining a drone position of the drone based on the position information; Marking the starting point of the trajectory of the UAV; According to the trajectory starting point, defining the trajectory data structure of the UAV; Determine the trajectory point of the drone through the location information; Adding the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the UAV; The initial flight trajectory is smoothed to obtain the UAV flight trajectory of the UAV.

4. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 3, characterized in that: The step of smoothing the initial flight trajectory to obtain the UAV flight trajectory of the UAV includes: defining a sliding window of the initial flight trajectory; Determining trajectory point weights of trajectory points corresponding to the initial flight trajectory; According to the sliding window and the weight of the trajectory point, the smoothing coefficient of the trajectory point is calculated using the following formula: ； in, Indicates at a point in time The smoothing coefficient corresponding to the trajectory point, Indicates time point , represents the size of the sliding window, Indicates at a point in time The trajectory point, Indicates at a point in time The trajectory point weight corresponding to the trajectory point; The initial flight trajectory is smoothed based on the smoothing coefficient to obtain a UAV flight trajectory of the UAV.

5. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 4, characterized in that: The step of marking obstacle elements in the corresponding flight area of ​​the UAV based on the calibration environment perception data includes: Determining a regional RGB image, 3D point cloud data, and detecting heat sources of the flight area based on the calibrated environmental perception data; Performing subject separation on the regional RGB image to obtain the image subject; Supplementing depth information of the image subject by using the 3D point cloud data to obtain a supplemented image subject; Extracting image subject features and heat source features of the supplementary image subject and the detected heat source; Based on the main features and heat source features of the image, obstacle elements in the flight area are identified using a trained obstacle recognition model.

6. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 5, characterized in that: The calculating the obstacle coefficient of the obstacle element comprises: Identifying the obstacle type of the obstacle element, wherein the obstacle type includes static obstacles, dynamic obstacles, and living obstacles; Extracting obstacle features of the obstacle type, wherein the obstacle features include obstacle distance, obstacle volume, obstacle dynamics, and heat source value; an obstacle characteristic score defining the obstacle characteristic; Normalizing the obstacle features to obtain normalized obstacle features, wherein the normalized obstacle features include normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, and normalized heat source value; The obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score.

7. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 6, characterized in that: According to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, Calculating the obstacle coefficient of the obstacle element, including: Defining environmental impact factors of the obstacle elements, wherein the environmental impact factors include wind speed and light conditions; Based on the wind speed, the lighting condition, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score, the obstacle coefficient of the obstacle element is calculated using the following formula: ； in, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, The obstacle feature score representing the normalized obstacle distance, represents the normalized obstacle volume, The obstacle feature score representing the normalized obstacle volume, represents the normalized obstacle dynamics, The obstacle feature score representing the normalized obstacle dynamics, represents the normalized heat source value, The obstacle feature score representing the normalized heat source value, Indicates wind speed, represents the influence weight of wind speed, Indicates the lighting conditions, Represents the influence weight of lighting conditions.

8. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 7, characterized in that: The step of performing three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map includes: Determining the element to be modeled of the obstacle element according to the obstacle coefficient; Collecting element point cloud data of the element to be modeled; Denoising the element point cloud data to obtain denoised element point cloud data; determining a reference point cloud of the embedded map; Defining a rotation matrix of the denoised element point cloud data in the embedded map; Based on the reference point cloud, the rotation matrix and the element point cloud data, the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map is calculated using the following formula: ； in, Indicates The map pose of the sensor in the embedded map is Indicates the first The corresponding point position of the denoised element point cloud data in the reference point cloud, Indicates the first Noise-reduced element point cloud data, represents the rotation matrix, represents the translation vector; According to the map posture and the noise reduction element point cloud data, the obstacle elements are three-dimensionally reconstructed in the embedded map to obtain an embedded obstacle map.

9. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 8, characterized in that: The step of establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map includes: Marking potential waypoints of the embedded obstacle map according to the drone destination and the drone position; Determining a current waypoint of the potential waypoint based on the drone position; Calculating a waypoint response value of the potential waypoint based on the embedded obstacle map and the UAV flight trajectory; The obstacle avoidance path of the UAV is determined by the response value.

10. A dual-screen display system for drones based on embedded maps, characterized in that: The system comprises: A drone data acquisition module is used to acquire the flight data and environmental perception data of the drone in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; An obstacle element marking module, used to mark the drone position and the drone flight trajectory of the drone based on the calibration flight data, and mark the obstacle elements of the drone corresponding to the flight area based on the calibration environment perception data; An obstacle map construction module is used to calculate the obstacle coefficient of the obstacle element, and according to the obstacle coefficient, perform three-dimensional reconstruction of the obstacle element in the embedded map to obtain an embedded obstacle map; A first display screen establishing module, used to establish an obstacle avoidance path of the drone according to a preset drone target location, the drone position, the drone flight trajectory and the embedded obstacle map, and to construct a first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map and the obstacle avoidance path; The second display screen establishment module is used to obtain the real-time flight video of the UAV, identify the video body of the real-time flight video, construct an identification frame and attribute information of the video body, construct the second display screen of the UAV according to the video body, the identification frame and the attribute information, establish the dual-screen coordination rules of the UAV, and execute the dual-screen display of the first display screen and the second display screen according to the dual-screen coordination rules.

Images