Method and System for Implementing Dual-Screen Display of Unmanned Aerial Vehicle Based on Embedded Map

Abstract

The present invention relates to the technical field of unmanned aerial vehicles, and discloses a method and system for realizing dual-screen display of unmanned aerial vehicles based on an embedded map, including: calibrating flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; marking the position and flight trajectory of the unmanned aerial vehicle, and marking obstacle elements in the corresponding flight area of the unmanned aerial vehicle; calculating the obstacle coefficients of the obstacle elements, and performing three-dimensional reconstruction on the obstacle elements in the embedded map to obtain an embedded obstacle map; establishing an obstacle avoidance path for the unmanned aerial vehicle, and constructing a first display screen of the unmanned aerial vehicle; constructing an identification frame and attribute information of the video main body, and constructing a second display screen of the unmanned aerial vehicle according to the video main body, the identification frame and the attribute information, establishing a dual-screen cooperation rule for the unmanned aerial vehicle, and performing dual-screen display of the first display screen and the second display screen according to the dual-screen cooperation rule. The present invention can improve the operation efficiency of unmanned aerial vehicles in complex tasks.

Description

Technical Field

[0001] The present invention relates to a method and system for realizing dual-screen display of an unmanned aerial vehicle (UAV) based on an embedded map, and belongs to the technical field of UAVs. Background Art

[0002] Dual-screen display of a UAV refers to the use of two screens in a UAV system to simultaneously display different information or different perspectives of the same information. The dual-screen display system of a UAV improves the operation efficiency and safety, enabling UAV operators to better perform complex tasks and make quick decisions.

[0003] The traditional dual-screen display method of a UAV usually displays flight data and map information on one screen according to a pre-set flight path, and the other screen shows the real-time video of the camera. This cannot integrate environmental perception data and real-time analysis, cannot provide a more comprehensive situation awareness, lacks real-time data fusion, and cannot effectively display the obstacle avoidance path and obstacle information, resulting in poor operation efficiency and safety in complex tasks. Summary of the Invention

[0004] The present invention provides a method and system for realizing dual-screen display of a UAV based on an embedded map, and its main purpose is to improve the operation efficiency of the UAV in complex tasks.

[0005] To achieve the above object, a method for realizing dual-screen display of a UAV based on an embedded map provided by the present invention includes:

[0006] Obtain the flight data and environmental perception data of the UAV in the embedded map, where the flight data includes position, altitude, speed, and attitude information, and the environmental perception data includes lidar point cloud data, infrared image data, and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data;

[0007] Based on the calibrated flight data, mark the UAV position and UAV flight trajectory of the UAV, and based on the calibrated environmental perception data, mark the obstacle elements in the corresponding flight area of the UAV;

[0008] Calculate the obstacle coefficient of the obstacle elements, and perform three-dimensional reconstruction on the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map;

[0009] Establish an obstacle avoidance path for the UAV according to a preset UAV target location, the UAV position, the UAV flight trajectory, and the embedded obstacle map, and construct a first display screen of the UAV according to the UAV position, the UAV flight trajectory, the embedded obstacle map, and the obstacle avoidance path;

[0010] Obtain the real-time flight video of the drone, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the drone according to the video subject, the identification frame and the attribute information, establish a dual-screen collaboration rule for the drone, and perform dual-screen display of the first display screen and the second display screen according to the dual-screen collaboration rule.

[0011] Optionally, the calibrating the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes:

[0012] Identify the sensor types of the flight data and the environmental perception data;

[0013] Analyze the flight data characteristics of the flight data and the environmental perception data characteristics of the environmental perception data;

[0014] Define a joint calibration algorithm for the flight data and the environmental perception data based on the sensor type, the flight data characteristics, and the environmental perception data characteristics;

[0015] Determine the parameter feasible region of the joint calibration algorithm;

[0016] Calibrate the flight data and the environmental perception data through the parameter feasible region by using the joint calibration algorithm to obtain the calibrated flight data and the calibrated environmental perception data.

[0017] Optionally, the marking the drone position and the drone flight trajectory based on the calibrated flight data includes:

[0018] Extract the position information from the calibrated flight data, and determine the drone position of the drone according to the position information;

[0019] Mark the trajectory starting point of the drone;

[0020] Define the trajectory data structure of the drone according to the trajectory starting point;

[0021] Determine the trajectory points of the drone through the position information;

[0022] Add the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the drone;

[0023] Smooth the initial flight trajectory to obtain the drone flight trajectory of the drone.

[0024] Optionally, the smoothing the initial flight trajectory to obtain the drone flight trajectory of the drone includes:

[0025] Define a sliding window for the initial flight trajectory;

[0026] Determine the trajectory point weights of the trajectory points corresponding to the initial flight trajectory;

[0027] Calculate the smoothing coefficient of the trajectory point according to the sliding window and the trajectory point weights using the following formula:

[0028] ;

[0029] where, represents the smoothing coefficient of the corresponding trajectory point at time point , represents time point , represents the size of the sliding window, represents the trajectory point at time point , represents the trajectory point weight of the corresponding trajectory point at time point ;

[0030] Smooth the initial flight trajectory based on the smoothing coefficient to obtain the UAV flight trajectory of the UAV.

[0031] Optionally, the marking of the obstacle elements in the flight area corresponding to the UAV based on the calibrated environment perception data includes:

[0032] Determine the regional RGB image, 3D point cloud data, and detected heat source of the flight area according to the calibrated environment perception data;

[0033] Perform foreground separation on the regional RGB image to obtain the image foreground;

[0034] Use the 3D point cloud data to supplement the depth information of the image foreground to obtain a supplemented image foreground;

[0035] Extract the image foreground features and heat source features of the supplemented image foreground and the detected heat source;

[0036] Identify the obstacle elements in the flight area using the trained obstacle recognition model based on the image foreground features and heat source features.

[0037] Optionally, the calculation of the obstacle coefficient of the obstacle element includes:

[0038] Identify the obstacle type of the obstacle element, where the obstacle type includes static obstacles, dynamic obstacles, and living obstacles;

[0039] Extract the obstacle features of the obstacle type, where the obstacle features include obstacle distance, obstacle volume, obstacle dynamics, and heat source value;

[0040] Define the obstacle feature score of the obstacle features;

[0041] Normalize the obstacle features to obtain normalized obstacle features, where the normalized obstacle features include normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, and normalized heat source value;

[0042] Calculate the obstacle coefficient of the obstacle element according to the normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, normalized heat source value, and obstacle feature score.

[0043] Optionally, the calculating the obstacle coefficient of the obstacle element according to the normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, normalized heat source value, and obstacle feature score includes:

[0044] Define the environmental impact factor of the obstacle element, where the environmental impact factor includes wind speed and light condition;

[0045] Based on the wind speed, the light condition, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score, use the following formula to calculate the obstacle coefficient of the obstacle element:

[0046] ;

[0047] where, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, represents the obstacle feature score of the normalized obstacle distance, represents the normalized obstacle volume, represents the obstacle feature score of the normalized obstacle volume, represents the normalized obstacle dynamics, represents the obstacle feature score of the normalized obstacle dynamics, represents the normalized heat source value, represents the obstacle feature score of the normalized heat source value, represents the wind speed, represents the influence weight of the wind speed, represents the light condition, represents the influence weight of the light condition.

[0048] Optionally, performing three-dimensional reconstruction on the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map, including:

[0049] Determining the elements to be modeled of the obstacle elements according to the obstacle coefficient;

[0050] Collecting the element point cloud data of the elements to be modeled;

[0051] Denosing the element point cloud data to obtain denoised element point cloud data;

[0052] Determining the reference point cloud of the embedded map;

[0053] Defining the rotation matrix of the denoised element point cloud data in the embedded map;

[0054] Based on the reference point cloud, the rotation matrix, and the element point cloud data, calculating the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map by using the following formula:

[0055] ;

[0056] wherein, represents the map pose of the th sensor of the denoised element point cloud data in the embedded map, represents the point position corresponding to the th denoised element point cloud data captured by the sensor in the reference point cloud, represents the th denoised element point cloud data captured by the sensor, represents the rotation matrix, represents the translation vector;

[0057] Performing three-dimensional reconstruction on the obstacle elements in the embedded map according to the map pose and the denoised element point cloud data to obtain an embedded obstacle map.

[0058] Optionally, establishing an obstacle avoidance path for the drone according to the preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map, including:

[0059] Marking potential path points on the embedded obstacle map according to the drone target location and the drone position;

[0060] Determining the current path points of the potential path points according to the drone position;

[0061] Calculating the path point response values of the potential path points based on the embedded obstacle map and the drone flight trajectory;

[0062] Determine the obstacle avoidance path of the UAV based on the response value.

[0063] To solve the above problems, the present invention also provides a UAV dual-screen display system based on an embedded map. The system includes:

[0064] A UAV data acquisition module, configured to acquire the flight data and environmental perception data of the UAV in the embedded map. Among them, the flight data includes position, altitude, speed, and attitude information, and the environmental perception data includes lidar point cloud data, infrared image data, and ultrasonic sensor data. Calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data;

[0065] An obstacle element marking module, configured to mark the UAV position and UAV flight trajectory of the UAV based on the calibrated flight data, and mark the obstacle elements in the corresponding flight area of the UAV based on the calibrated environmental perception data;

[0066] An obstacle map construction module, configured to calculate the obstacle coefficient of the obstacle elements, and perform three-dimensional reconstruction on the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map;

[0067] A first display screen establishment module, configured to establish the obstacle avoidance path of the UAV according to the preset UAV destination, the UAV position, the UAV flight trajectory, and the embedded obstacle map, and construct the first display screen of the UAV according to the UAV position, the UAV flight trajectory, the embedded obstacle map, and the obstacle avoidance path;

[0068] A second display screen establishment module, configured to acquire the real-time flight video of the UAV, identify the video main body of the real-time flight video, construct the recognition frame and attribute information of the video main body, construct the second display screen of the UAV according to the video main body, the recognition frame, and the attribute information, establish the dual-screen collaboration rule of the UAV, and perform dual-screen display of the first display screen and the second display screen according to the dual-screen collaboration rule.

[0069] Compared with the problems described in the background art, first of all, by acquiring and calibrating the flight data and environmental perception data of the drone, the system ensures the accuracy and reliability of the data, providing a solid foundation for subsequent flight decisions. The precise marking of position, altitude, speed, and attitude information enables the operator to grasp the flight state of the drone in real time. The calibration of lidar point cloud data, infrared image data, and ultrasonic sensor data improves the accuracy of environmental perception and effectively avoids potential collision risks. Secondly, the three-dimensional reconstruction of the embedded obstacle map provides a detailed environmental model for the drone, enabling the drone to perform more intelligent path planning and obstacle avoidance in complex environments. The calculation of the obstacle coefficient further enhances the pertinence of the obstacle avoidance strategy, ensuring the safe flight of the drone when facing obstacles of different difficulties. Moreover, the construction of the first display screen makes the flight trajectory, position, obstacle elements, and obstacle avoidance path of the drone clear at a glance, greatly improving the operator's situation awareness ability and facilitating effective flight command and decision-making. In addition, the real-time flight video analysis and video subject recognition of the second display screen provide powerful visual assistance for the drone to perform specific tasks (such as monitoring, searching, rescue, etc.). The real-time display of the recognition frame and attribute information enables the operator to quickly identify and respond to key targets in the video. Finally, the establishment of the dual-screen collaboration rule realizes the seamless docking and collaborative work of the first display screen and the second display screen, improving the operation efficiency and reducing the operation complexity. Therefore, the present invention improves the operation efficiency of the drone in complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 FIG. is a schematic flow chart of a method for realizing dual-screen display of a drone based on an embedded map according to an embodiment of the present invention;

[0071] Figure 2 FIG. is a schematic block diagram of a module for realizing the method for realizing dual-screen display of a drone based on an embedded map according to an embodiment of the present invention.

[0072] The implementation, functional features, and advantages of the present invention will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] An embodiment of the present application provides a method for realizing dual-screen display of an unmanned aerial vehicle (UAV) based on an embedded map. The execution subject of the method for realizing dual-screen display of the UAV based on the embedded map includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the method for realizing dual-screen display of the UAV based on the embedded map can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc.

[0075] Embodiment 1:

[0076] Refer to Figure 1 As shown, it is a schematic flowchart of a method for realizing dual-screen display of a UAV based on an embedded map provided by an embodiment of the present invention. In this embodiment, the method for realizing dual-screen display of the UAV based on the embedded map includes:

[0077] S1. Obtain the flight data and environmental perception data of the UAV in the embedded map. Among them, the flight data includes position, altitude, speed, and attitude information, and the environmental perception data includes lidar point cloud data, infrared image data, and ultrasonic sensor data. Calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data.

[0078] It should be explained that the position refers to the specific coordinate position of the UAV in the embedded map, the altitude refers to the vertical distance of the UAV relative to the ground, the speed refers to the flight speed of the UAV, and the attitude information refers to the direction and tilt state of the UAV in three-dimensional space. The lidar point cloud data refers to the data obtained by a lidar (LiDAR) sensor, the infrared image data refers to the data captured by an infrared camera, and the ultrasonic sensor data refers to the data obtained by an ultrasonic sensor.

[0079] The present invention calibrates the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data, which can make the data more accurate and reliable and is suitable for precise navigation and control of the UAV.

[0080] Specifically, the calibration of the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes:

[0081] Identify the sensor types of the flight data and environmental perception data;

[0082] Analyze the flight data characteristics and environmental perception data characteristics of the flight data and environmental perception data;

[0083] Define a joint calibration algorithm for the flight data and environmental perception data based on the sensor type, the characteristics of the flight data, and the characteristics of the environmental perception data;

[0084] Determine the parameter feasible region of the joint calibration algorithm;

[0085] Calibrate the flight data and environmental perception data through the parameter feasible region using the joint calibration algorithm to obtain the calibrated flight data and calibrated environmental perception data.

[0086] Among them, the sensor type refers to the types of sensors used to collect flight data and environmental perception data. This may include, but is not limited to, inertial measurement units (IMUs), global positioning systems (GPS), radars, lidar, cameras, ultrasonic sensors, etc. The characteristics of the flight data refer to the quality and attributes that describe the flight data, including but not limited to data accuracy, resolution, sampling rate, noise level, dynamic range, nonlinear error, system bias, time drift, etc. The characteristics of the environmental perception data refer to the quality and attributes of the data collected by environmental perception sensors, such as resolution, accuracy, range, viewing angle, occlusion, reflectivity, environmental interference, etc. The joint calibration algorithm refers to an algorithm used to simultaneously calibrate flight data and environmental perception data to reduce systematic errors and biases between different sensors. The parameter feasible region refers to the range of acceptable parameter values in the joint calibration algorithm. The calibrated flight data refers to the flight data processed by the joint calibration algorithm, and the calibrated environmental perception data refers to the environmental perception data processed by the joint calibration algorithm.

[0087] Optionally, the step of calibrating the flight data and environmental perception data through the parameter feasible region using the joint calibration algorithm to obtain the calibrated flight data and calibrated environmental perception data can be jointly processed by statistical methods (such as the least squares method) and filtering techniques (such as Kalman filtering).

[0088] S2. Based on the calibrated flight data, mark the UAV position and UAV flight trajectory of the UAV, and based on the calibrated environmental perception data, mark the obstacle elements in the corresponding flight area of the UAV.

[0089] The marking of the UAV position and UAV flight trajectory of the UAV based on the calibrated flight data in the present invention can provide a basis for the later optimization of UAV flight.

[0090] Specifically, the marking of the UAV position and UAV flight trajectory of the UAV based on the calibrated flight data includes:

[0091] Extract the position information from the calibrated flight data, and determine the UAV position of the UAV according to the position information;

[0092] Mark the starting point of the UAV's trajectory;

[0093] Define the trajectory data structure of the UAV according to the starting point of the trajectory;

[0094] Determine the trajectory points of the UAV through the position information;

[0095] Add the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the UAV;

[0096] Smooth the initial flight trajectory to obtain the UAV flight trajectory of the UAV.

[0097] Among them, the position information refers to the specific coordinate information of the UAV extracted from the calibrated flight data, usually including positions in two-dimensional or three-dimensional space, such as longitude and latitude, altitude. The UAV position refers to a specific position information point, representing the exact position of the UAV at a certain moment. The starting point of the trajectory refers to the starting point of the UAV flight trajectory, usually the position when the UAV takes off. The trajectory data structure refers to the data structure used to store and represent the UAV flight trajectory, which can be an array, a list, a database. The trajectory point refers to a single data point in the flight trajectory, and each point contains the position information of the UAV and the corresponding timestamp. The initial flight trajectory refers to the flight trajectory initially constructed according to the extracted position information. The UAV flight trajectory refers to the final flight trajectory after smoothing and optimization processing.

[0098] Furthermore, the smoothing the initial flight trajectory to obtain the UAV flight trajectory of the UAV includes:

[0099] Define the sliding window of the initial flight trajectory;

[0100] Determine the trajectory point weights of the corresponding trajectory points of the initial flight trajectory;

[0101] According to the sliding window and the trajectory point weights, calculate the smoothing coefficient of the trajectory points using the following formula:

[0102] ;

[0103] Among them, represents the smoothing coefficient of the corresponding trajectory point at time point , represents time point , represents the size of the sliding window, represents at time point The trajectory points, indicating at the time point the trajectory point weight corresponding to the trajectory point;

[0104] Smooth the initial flight trajectory based on the smoothing coefficient to obtain the drone flight trajectory of the drone.

[0105] Among them, the sliding window refers to a data window with a fixed size that slides on the initial flight trajectory, the trajectory point weight refers to the numerical value assigned to each trajectory point, which represents the importance of the trajectory point relative to other trajectory points during the smoothing process, and the smoothing coefficient refers to the numerical value used to adjust the contribution degree of each trajectory point to the final smoothed trajectory.

[0106] Based on the calibrated environment perception data, marking the obstacle elements in the corresponding flight area of the drone can effectively mark the obstacle elements in the drone flight area and ensure flight safety.

[0107] Specifically, marking the obstacle elements in the corresponding flight area of the drone based on the calibrated environment perception data includes:

[0108] According to the calibrated environment perception data, determine the regional RGB image, 3D point cloud data and detect heat sources of the flight area;

[0109] Perform main body separation on the regional RGB image to obtain the image main body;

[0110] Use the 3D point cloud data to supplement the depth information of the image main body to obtain the supplemented image main body;

[0111] Extract the image main body features and heat source features of the supplemented image main body and the detected heat source;

[0112] Based on the image main body features and heat source features, use the trained obstacle recognition model to identify the obstacle elements in the flight area.

[0113] Among them, the regional RGB image refers to the color image of the flight area captured by the RGB camera carried by the drone. The 3D point cloud data refers to the three-dimensional point cloud data of the flight area obtained by LiDAR or other depth sensors. The detected heat source refers to the heat source information in the flight area detected by the infrared sensor. The image main body is the main target or area separated from the regional RGB image, usually potential obstacles (such as buildings, trees, vehicles, etc.). The supplementary image main body refers to combining the depth information of the 3D point cloud data with the image main body in the RGB image to generate a main body image with depth information. The image main body features refer to the features extracted from the supplementary image main body, including information such as shape, texture, color, and depth. The heat source features refer to the features extracted from the detected heat source, including information such as the temperature distribution, shape, and size of the heat source. The obstacle elements refer to the objects or areas in the flight area that may impede the flight of the drone, such as buildings, trees, wires, vehicles, pedestrians, etc. The obstacle recognition model refers to the model used to identify obstacle information and can be trained using neural networks.

[0114] Optionally, the use of the 3D point cloud data to supplement the depth information of the image main body to obtain the supplementary image main body can be implemented using OpenCV and Open3D software.

[0115] S3. Calculate the obstacle coefficient of the obstacle element, and based on the obstacle coefficient, perform three-dimensional reconstruction of the obstacle element in the embedded map to obtain the embedded obstacle map.

[0116] The calculation of the obstacle coefficient of the obstacle element in the present invention can provide a basis for the aircraft path optimization of the drone, thereby improving the flight safety of the drone.

[0117] Specifically, the calculation of the obstacle coefficient of the obstacle element includes:

[0118] Identify the obstacle type of the obstacle element, where the obstacle type includes static obstacles, dynamic obstacles, and living obstacles;

[0119] Extract the obstacle features of the obstacle type, where the obstacle features include obstacle distance, obstacle volume, obstacle dynamics, and heat source value;

[0120] Define the obstacle feature score of the obstacle features;

[0121] Normalize the obstacle features to obtain the normalized obstacle features, where the normalized obstacle features include normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics, and normalized heat source value;

[0122] Calculate the obstacle coefficient of the obstacle element according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score.

[0123] Among them, the static obstacle refers to an obstacle with a fixed position, such as a building, a tree, a utility pole, etc. The dynamic obstacle refers to an obstacle whose position changes over time, such as a vehicle, a drone, a mobile device, etc. The living obstacle refers to an obstacle with life characteristics, such as a bird, an animal, a pedestrian, etc. The obstacle distance refers to the straight-line distance between the obstacle and the drone. The obstacle volume refers to the volume of the obstacle in three-dimensional space. The obstacle dynamics refers to the motion state of the obstacle (such as speed, direction). The heat source value refers to the heat radiation intensity of the obstacle (such as a temperature value). The normalized obstacle distance refers to the distance value normalized to the range [0,1]. The normalized obstacle volume refers to the value normalized to the range [0,1]. The normalized obstacle dynamics refers to the value normalized to the range [0,1]. The normalized heat source value refers to the value normalized to the range [0,1]. The obstacle feature score refers to the score assigned to each obstacle feature, reflecting the influence degree of the feature on the obstacle coefficient. The obstacle coefficient refers to an index calculated by comprehensively considering the obstacle feature score and the normalized obstacle features, and is used to quantify the threat degree of the obstacle to the flight of the drone.

[0124] Further, the calculating the obstacle coefficient of the obstacle element according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score includes:

[0125] Define the environmental impact factor of the obstacle element, where the environmental impact factor includes wind speed and light conditions;

[0126] Based on the wind speed, the light conditions, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value, and the obstacle feature score, use the following formula to calculate the obstacle coefficient of the obstacle element:

[0127] ;

[0128] Among them, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, represents the obstacle feature score of the normalized obstacle distance, represents the normalized obstacle volume, An obstacle feature score representing the normalized obstacle volume Represents the normalized obstacle dynamics An obstacle feature score representing the normalized obstacle dynamics Represents the normalized heat source value An obstacle feature score representing the normalized heat source value Represents the wind speed Indicates the influence weight of the wind speed Represents the lighting condition Indicates the influence weight of the lighting condition

[0129] Wherein, the wind speed refers to the air flow speed in the UAV flight area, the lighting condition refers to the lighting intensity and direction in the UAV flight area, and the influence weight refers to the influence degree of the environmental influence factor during the UAV flight process.

[0130] Based on the obstacle coefficient, the present invention performs three-dimensional reconstruction of the obstacle elements in the embedded map, and the obtained embedded obstacle map can provide a basis for obstacle avoidance of the UAV.

[0131] Specifically, the three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map includes:

[0132] Determine the elements to be modeled of the obstacle elements according to the obstacle coefficient;

[0133] Collect the element point cloud data of the elements to be modeled;

[0134] Denoise the element point cloud data to obtain denoised element point cloud data;

[0135] Determine the reference point cloud of the embedded map;

[0136] Define the rotation matrix of the denoised element point cloud data in the embedded map;

[0137] Based on the reference point cloud, the rotation matrix, and the element point cloud data, use the following formula to calculate the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map:

[0138] ;

[0139] Wherein, Represents the map pose of the th sensor of the denoised element point cloud data in the embedded map, Represents the point position corresponding to the th denoised element point cloud data captured by the sensor in the reference point cloud Represents the nth noise reduction element point cloud data captured by the sensor, represents the rotation matrix, represents the translation vector;

[0140] According to the map pose and the noise reduction element point cloud data, perform three-dimensional reconstruction of the obstacle element in the embedded map to obtain an embedded obstacle map.

[0141] Among them, the element to be modeled refers to an object identified as an obstacle in the UAV flight area. The element point cloud data refers to the point cloud data of the obstacle element to be modeled collected by a sensor (such as a lidar or an RGB-D camera). These data contain the geometric information of the obstacle element. The noise reduction element point cloud data refers to the data obtained by performing noise reduction processing (such as removing outliers, smoothing, etc.) on the original element point cloud data, aiming to improve the accuracy of the data and the efficiency of subsequent processing. The reference point cloud refers to a known point cloud in the embedded map, which is usually a part of the global map constructed by the SLAM algorithm and is used as a benchmark for positioning and mapping. The rotation matrix refers to a 3x3 matrix used to describe the rotation relationship of the noise reduction element point cloud data relative to the reference point cloud. The map pose refers to the position and orientation of the sensor in the embedded map. The embedded map refers to the original map used for UAV flight navigation. The embedded obstacle map refers to a three-dimensional map containing environmental obstacles and features, which is used for UAV navigation and obstacle avoidance.

[0142] S4. According to the preset UAV target location, the UAV position, the UAV flight trajectory, and the embedded obstacle map, establish an obstacle avoidance path for the UAV, and construct a first display screen for the UAV according to the UAV position, the UAV flight trajectory, the embedded obstacle map, and the obstacle avoidance path.

[0143] The present invention establishes an obstacle avoidance path for the UAV according to the preset UAV target location, the UAV position, the UAV flight trajectory, and the embedded obstacle map to ensure the safety of the UAV and the smooth completion of the task.

[0144] Specifically, the step of establishing an obstacle avoidance path for the UAV according to the preset UAV target location, the UAV position, the UAV flight trajectory, and the embedded obstacle map includes:

[0145] Mark potential path points on the embedded obstacle map according to the UAV target location and the UAV position;

[0146] Determine the current path points of the potential path points according to the UAV position;

[0147] Calculate the path point response value of the potential path points based on the embedded obstacle map and the UAV flight trajectory;

[0148] Determine the obstacle avoidance path of the UAV based on the response value.

[0149] Among them, the potential path points refer to a series of preselected position points that the UAV may pass through during the process of flying from the current position to the target location. The current path point refers to the current path point of the UAV. The path point response value refers to an evaluation index for potential path points, which reflects the flight difficulty, safety, energy consumption, or other costs from the current UAV position to this path point. The obstacle avoidance path refers to the safe flight path planned by the UAV during flight to avoid collisions.

[0150] Optionally, calculating the path point response value of the potential path points based on the embedded obstacle map and the UAV flight trajectory can be achieved by using a cost map.

[0151] According to the UAV position, the UAV flight trajectory, the embedded obstacle map, and the obstacle avoidance path, the present invention constructs the first display screen of the UAV to obtain an intuitive display screen, so as to better monitor the flight state of the UAV and perform obstacle avoidance operations. Among them, the first display screen refers to the display screen interface that the operator sees on the first screen in the UAV control system or monitoring system, including a map view, a UAV icon, a UAV flight trajectory, an embedded obstacle map, an obstacle avoidance path, and status information (key flight parameters such as the speed, altitude, heading, battery power, signal strength, etc. of the UAV).

[0152] S5. Obtain the real-time flight video of the UAV, identify the video subject of the real-time flight video, construct the recognition frame and attribute information of the video subject, construct the second display screen of the UAV according to the video subject, the recognition frame, and the attribute information, establish the dual-screen collaboration rule of the UAV, and perform the dual-screen display of the first display screen and the second display screen according to the dual-screen collaboration rule.

[0153] It should be explained that the real-time flight video refers to the image stream captured and transmitted in real time by the camera carried by the UAV during flight. The recognition frame refers to a rectangular frame used to mark and locate specific objects (such as people, vehicles, animals, etc.) in the real-time flight video. The attribute information refers to a series of data descriptions related to the video subject within the recognition frame, such as category, ID, behavior, distance, etc. The second display screen refers to the display screen interface in the UAV control system or monitoring system dedicated to displaying the real-time flight video, the recognition frame, and the attribute information.

[0154] The establishment of the dual-screen collaboration rules for the UAV in the present invention can be more efficient and secure, thereby improving the work efficiency of the operator and the overall performance of the UAV. Among them, the dual-screen collaboration rules include information synchronization rules, interaction design rules, visual coordination rules, warning and notification rules, user customization rules, and system integration rules.

[0155] Compared with the problems described in the background art, first of all, by acquiring and calibrating the flight data and environmental perception data of the UAV, the system ensures the accuracy and reliability of the data, providing a solid foundation for subsequent flight decisions. The accurate marking of position, altitude, speed, and attitude information enables the operator to grasp the flight state of the UAV in real time, while the calibration of lidar point cloud data, infrared image data, and ultrasonic sensor data improves the accuracy of environmental perception and effectively avoids potential collision risks. Secondly, the three-dimensional reconstruction of the embedded obstacle map provides a detailed environmental model for the UAV, enabling the UAV to perform more intelligent path planning and obstacle avoidance in complex environments. The calculation of the obstacle coefficient further enhances the pertinence of the obstacle avoidance strategy, ensuring the safe flight of the UAV when facing obstacles of different difficulties. Furthermore, the construction of the first display screen makes the flight trajectory, position, obstacle elements, and obstacle avoidance path of the UAV clear at a glance, greatly improving the operator's situational awareness ability and facilitating effective flight command and decision-making. In addition, the real-time flight video analysis of the second display screen and the recognition of video subjects provide powerful visual assistance for the UAV to perform specific tasks (such as monitoring, searching, rescue, etc.). The real-time display of the recognition frame and attribute information enables the operator to quickly identify and respond to key targets in the video. Finally, the establishment of the dual-screen collaboration rules realizes the seamless docking and collaborative work of the first display screen and the second display screen, improving the operation efficiency and reducing the operation complexity. Therefore, the present invention improves the operation efficiency of the UAV in complex tasks.

[0156] Embodiment 2:

[0157] As Figure 2 shown, it is a functional module diagram of a UAV dual-screen display system based on an embedded map according to the present invention.

[0158] The UAV dual-screen display system 200 based on an embedded map according to the present invention can be installed in an electronic device. According to the functions achieved, the UAV dual-screen display system based on an embedded map can include a UAV data acquisition module 201, an obstacle element marking module 202, an obstacle map construction module 203, a first display screen establishment module 204, and a second display screen establishment module 205. The modules in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by the processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0159] In the embodiments of the present invention, the functions of each module / unit are as follows:

[0160] The UAV data acquisition module 201 is configured to acquire the flight data and environmental perception data of the UAV in the embedded map. Among them, the flight data includes position, altitude, speed, and attitude information, and the environmental perception data includes lidar point cloud data, infrared image data, and ultrasonic sensor data. Calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data;

[0161] The obstacle element marking module 202 is configured to mark the UAV position and UAV flight trajectory of the UAV based on the calibrated flight data, and mark the obstacle elements in the corresponding flight area of the UAV based on the calibrated environmental perception data;

[0162] The obstacle map construction module 203 is configured to calculate the obstacle coefficient of the obstacle elements, and perform three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map;

[0163] The first display screen establishment module 204 is configured to establish an obstacle avoidance path for the UAV according to the preset UAV target location, the UAV position, the UAV flight trajectory, and the embedded obstacle map, and construct a first display screen of the UAV according to the UAV position, the UAV flight trajectory, the embedded obstacle map, and the obstacle avoidance path;

[0164] The second display screen establishment module 205 is configured to acquire the real-time flight video of the UAV, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the UAV according to the video subject, the identification frame, and the attribute information, establish a dual-screen collaboration rule for the UAV, and perform dual-screen display of the first display screen and the second display screen according to the dual-screen collaboration rule.

[0165] Specifically, each module in the UAV dual-screen display system 200 based on the embedded map in the embodiments of the present invention uses the same technical means as those in the Figure 1 The UAV dual-screen display method based on the embedded map described above, and can produce the same technical effects, which will not be elaborated here.

[0166] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for realizing dual-screen display of drone based on embedded map, characterized in that: The method comprises: Acquire flight data and environmental perception data of the UAV in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; Based on the calibration flight data, marking the drone position and the drone flight trajectory of the drone, and based on the calibration environment perception data, marking obstacle elements in the corresponding flight area of ​​the drone; The obstacle coefficient of the obstacle element is calculated, and according to the obstacle coefficient, the obstacle element is three-dimensionally reconstructed in the embedded map to obtain an embedded obstacle map, wherein the calculation of the obstacle coefficient of the obstacle element includes: identifying the obstacle type of the obstacle element, wherein the obstacle type includes static obstacles, dynamic obstacles and living obstacles, extracting obstacle features of the obstacle type, wherein the obstacle features include obstacle distance, obstacle volume, obstacle dynamics and heat source value, defining an obstacle feature score of the obstacle feature, normalizing the obstacle feature to obtain a normalized obstacle feature, wherein the normalized obstacle feature includes normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics and normalized The normalized heat source value is used to calculate the obstacle coefficient of the obstacle element according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, wherein the obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, including: defining an environmental impact factor of the obstacle element, wherein the environmental impact factor includes wind speed and lighting conditions, and based on the wind speed, the lighting conditions, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, the obstacle coefficient of the obstacle element is calculated using the following formula: ； in, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, The obstacle feature score representing the normalized obstacle distance, represents the normalized obstacle volume, The obstacle feature score representing the normalized obstacle volume, represents the normalized obstacle dynamics, The obstacle feature score representing the normalized obstacle dynamics, represents the normalized heat source value, The obstacle feature score representing the normalized heat source value, Indicates wind speed, represents the influence weight of wind speed, Indicates the lighting conditions, Indicates the influence weight of lighting conditions; Establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map, and constructing a first display screen for the drone according to the drone position, the drone flight trajectory, the embedded obstacle map, and the obstacle avoidance path; Acquire real-time flight video of the drone, identify the video subject of the real-time flight video, construct an identification frame and attribute information of the video subject, construct a second display screen of the drone based on the video subject, the identification frame and the attribute information, establish a dual-screen collaboration rule for the drone, and execute dual-screen display of the first display screen and the second display screen based on the dual-screen collaboration rule.

2. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 1, characterized in that: The step of calibrating the flight data and the environmental perception data to obtain calibrated flight data and calibrated environmental perception data includes: Identify sensor types for the flight data and environmental perception data; analyzing flight data characteristics and environmental perception data characteristics of the flight data and environmental perception data; Based on the sensor type, the flight data characteristics, and the environmental perception data characteristics, defining a joint calibration algorithm for the flight data and the environmental perception data; Determining a feasible domain of parameters of the joint calibration algorithm; The flight data and the environmental perception data are calibrated through the parameter feasible domain and using the joint calibration algorithm to obtain the calibrated flight data and the calibrated environmental perception data.

3. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 2, characterized in that: The step of marking the drone position and the drone flight trajectory of the drone based on the calibration flight data comprises: extracting position information from the calibration flight data, and determining a drone position of the drone based on the position information; Marking the starting point of the trajectory of the drone; According to the trajectory starting point, defining the trajectory data structure of the UAV; Determine the trajectory point of the drone through the location information; Adding the trajectory points to the trajectory data structure to obtain the initial flight trajectory of the UAV; The initial flight trajectory is smoothed to obtain the UAV flight trajectory of the UAV.

4. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 3, characterized in that: The step of smoothing the initial flight trajectory to obtain the UAV flight trajectory of the UAV includes: defining a sliding window of the initial flight trajectory; Determining trajectory point weights of trajectory points corresponding to the initial flight trajectory; According to the sliding window and the weight of the trajectory point, the smoothing coefficient of the trajectory point is calculated using the following formula: ； in, Indicates at a point in time The smoothing coefficient corresponding to the trajectory point, Indicates time point , represents the size of the sliding window, Indicates at a point in time The trajectory point, Indicates at a point in time The trajectory point weight corresponding to the trajectory point; The initial flight trajectory is smoothed based on the smoothing coefficient to obtain a UAV flight trajectory of the UAV.

5. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 4, characterized in that: The step of marking obstacle elements in the corresponding flight area of ​​the UAV based on the calibration environment perception data includes: Determining a regional RGB image, 3D point cloud data, and detecting heat sources of the flight area based on the calibrated environmental perception data; Performing subject separation on the regional RGB image to obtain the image subject; Supplementing depth information of the image subject by using the 3D point cloud data to obtain a supplemented image subject; Extracting image subject features and heat source features of the supplementary image subject and the detected heat source; Based on the main features and heat source features of the image, obstacle elements in the flight area are identified using a trained obstacle recognition model.

6. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 5, characterized in that: The step of performing three-dimensional reconstruction of the obstacle elements in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map includes: Determining the element to be modeled of the obstacle element according to the obstacle coefficient; Collecting element point cloud data of the element to be modeled; Denoising the element point cloud data to obtain denoised element point cloud data; determining a reference point cloud of the embedded map; Defining a rotation matrix of the denoised element point cloud data in the embedded map; Based on the reference point cloud, the rotation matrix and the element point cloud data, the map pose of the sensor corresponding to the denoised element point cloud data in the embedded map is calculated using the following formula: ； in, Indicates The map pose of the sensor in the embedded map is Indicates the first The corresponding point position of the denoised element point cloud data in the reference point cloud, Indicates the first Noise-reduced element point cloud data, represents the rotation matrix, represents the translation vector; According to the map posture and the noise reduction element point cloud data, the obstacle elements are three-dimensionally reconstructed in the embedded map to obtain an embedded obstacle map.

7. The method for realizing dual-screen display of unmanned aerial vehicle based on embedded map according to claim 6, characterized in that: The step of establishing an obstacle avoidance path for the drone according to a preset drone target location, the drone position, the drone flight trajectory, and the embedded obstacle map includes: Marking potential waypoints of the embedded obstacle map according to the drone destination and the drone position; Determining a current waypoint of the potential waypoint based on the drone position; Calculating a waypoint response value of the potential waypoint based on the embedded obstacle map and the UAV flight trajectory; The obstacle avoidance path of the UAV is determined by the response value.

8. A dual-screen display system for drones based on embedded maps, characterized in that: The system comprises: A drone data acquisition module is used to acquire the flight data and environmental perception data of the drone in the embedded map, wherein the flight data includes position, altitude, speed and attitude information, and the environmental perception data includes laser radar point cloud data, infrared image data and ultrasonic sensor data, and calibrate the flight data and environmental perception data to obtain calibrated flight data and calibrated environmental perception data; An obstacle element marking module, used to mark the drone position and the drone flight trajectory of the drone based on the calibration flight data, and mark the obstacle elements of the drone corresponding to the flight area based on the calibration environment perception data; an obstacle map construction module, for calculating the obstacle coefficient of the obstacle element, and performing three-dimensional reconstruction of the obstacle element in the embedded map according to the obstacle coefficient to obtain an embedded obstacle map, wherein the calculation of the obstacle coefficient of the obstacle element includes: identifying the obstacle type of the obstacle element, wherein the obstacle type includes static obstacles, dynamic obstacles and living obstacles, extracting obstacle features of the obstacle type, wherein the obstacle features include obstacle distance, obstacle volume, obstacle dynamics and heat source value, defining an obstacle feature score of the obstacle feature, and normalizing the obstacle feature to obtain a normalized obstacle feature, wherein the normalized obstacle feature includes normalized obstacle distance, normalized obstacle volume, normalized obstacle dynamics and heat source value. The obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, wherein the obstacle coefficient of the obstacle element is calculated according to the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, including: defining an environmental influencing factor of the obstacle element, wherein the environmental influencing factor includes wind speed and lighting conditions, and based on the wind speed, the lighting conditions, the normalized obstacle distance, the normalized obstacle volume, the normalized obstacle dynamics, the normalized heat source value and the obstacle feature score, the obstacle coefficient of the obstacle element is calculated using the following formula: ； in, represents the obstacle coefficient of the obstacle element, represents the normalized obstacle distance, The obstacle feature score representing the normalized obstacle distance, represents the normalized obstacle volume, The obstacle feature score representing the normalized obstacle volume, represents the normalized obstacle dynamics, The obstacle feature score representing the normalized obstacle dynamics, represents the normalized heat source value, The obstacle feature score representing the normalized heat source value, Indicates wind speed, represents the influence weight of wind speed, Indicates the lighting conditions, Indicates the influence weight of lighting conditions; A first display screen establishing module, used to establish an obstacle avoidance path of the drone according to a preset drone target location, the drone position, the drone flight trajectory and the embedded obstacle map, and to construct a first display screen of the drone according to the drone position, the drone flight trajectory, the embedded obstacle map and the obstacle avoidance path; The second display screen establishment module is used to obtain the real-time flight video of the UAV, identify the video body of the real-time flight video, construct an identification frame and attribute information of the video body, construct the second display screen of the UAV according to the video body, the identification frame and the attribute information, establish the dual-screen coordination rules of the UAV, and execute the dual-screen display of the first display screen and the second display screen according to the dual-screen coordination rules.

Images