A virtual simulation method for drone autopilot and the drone

Abstract

This invention discloses a virtual simulation method for unmanned aerial vehicle (UAV) autopilot and the UAV itself. The method includes acquiring input driving commands and performing semantic analysis on the commands to obtain a virtual destination; acquiring a visual image and recognizing the visual image to obtain a recognition result; if the recognition result indicates that the visual image does not contain the virtual destination, adjusting the perspective of the simulated UAV so that the captured visual image includes the virtual destination, generating visual image data; generating UAV control information based on the visual image data and the virtual destination, and controlling the UAV flight according to the UAV control information. This technical solution, by simulating and generating UAV control information, improves the accuracy, environmental adaptability, and intelligence level of UAV flight control, giving it greater robustness and generalization.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) simulation and control technology, and in particular to a virtual simulation method for UAV autopilot and the UAV itself. Background Technology

[0002] With the deep integration of aerospace technology and intelligent control technology, drones have been widely used in various fields such as aerial surveying and mapping, logistics transportation, emergency rescue, and power line inspection. In the core technology system of drones, the autonomous driving capability is the key to determining their operational efficiency, safety, and the breadth of applicable scenarios. Path planning, as the core module of the autonomous driving system, is directly responsible for generating the optimal trajectory of the drone from the starting point to the target point, and must simultaneously meet multiple constraints such as obstacle avoidance, energy consumption optimization, and mission timeliness.

[0003] Currently, the path planning function of drone autopilot systems generally relies on path planning algorithms based on traditional control theory and path planning algorithms based on deep reinforcement learning.

[0004] However, in practical applications, path planning schemes based on traditional algorithms or deep reinforcement learning face the critical challenges of insufficient robustness and generalization. Summary of the Invention

[0005] This invention provides a virtual simulation method for drone autopilot and a drone. By generating drone control information through simulation, the accuracy, environmental adaptability and intelligence of drone flight control are improved, giving it more robustness and generalizability.

[0006] According to one aspect of the present invention, a virtual simulation method for autonomous driving of an unmanned aerial vehicle is provided, the method comprising:

[0007] The system acquires input driving commands and performs semantic analysis on the commands to obtain a virtual destination.

[0008] Acquire visual images and recognize the visual images to obtain recognition results; wherein, the visual images refer to image information collected by a simulated camera on a drone in a drone flight scenario, and the drone flight scenario refers to a virtual space constructed by image data and point cloud data in the real environment;

[0009] If the recognition result indicates that the visual image does not contain the virtual destination, the perspective of the simulated drone is adjusted so that the captured visual image contains the virtual destination, and visual image data is generated.

[0010] Based on the visual image data and the virtual destination, drone control information is generated, and the drone is controlled to fly according to the drone control information.

[0011] According to another aspect of the present invention, a drone is provided, the drone comprising:

[0012] The drone program, when executed by a processor, implements a drone autopilot virtual simulation method as described in any embodiment of the present invention.

[0013] The technical solution of this invention obtains a virtual destination by acquiring input driving commands and performing semantic analysis on the commands; it then acquires and recognizes visual images to obtain recognition results; if the recognition results indicate that the visual image does not contain the virtual destination, the perspective of the simulated drone is adjusted so that the captured visual image includes the virtual destination, generating visual image data; based on the visual image data and the virtual destination, drone control information is generated, and the drone is controlled to fly according to the drone control information. This technical solution improves the accuracy, environmental adaptability, and intelligence level of drone flight control by simulating the generation of drone control information, giving it greater robustness and generalization.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a virtual simulation method for unmanned aerial vehicle (UAV) autopilot provided according to Embodiment 1 of the present invention;

[0017] Figure 2 This is a schematic diagram of a virtual simulation process for drone autopilot provided in Embodiment 2 of the present invention;

[0018] Figure 3 This is a schematic diagram of a virtual simulation method for unmanned aerial vehicle (UAV) autopilot provided in Embodiment 3 of the present invention;

[0019] Figure 4 This is a flowchart of a virtual simulation process for drone autopilot provided in Embodiment 4 of the present invention;

[0020] Figure 5 This is a schematic diagram of the structure of an electronic device that implements a virtual simulation method for autonomous driving of a drone according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] Figure 1 This is a flowchart of a virtual simulation method for drone autopilot provided according to Embodiment 1 of the present invention. This embodiment is applicable to simulation generation scenarios of drone control information. Figure 1 As shown, the method includes:

[0025] S110. Obtain the input driving command and perform semantic analysis on the driving command to obtain the virtual destination.

[0026] In this embodiment, the driving command refers to the information command used to specify the virtual destination of the drone's flight. For example, the driving command could be to locate the crane in the drone's flight scene and go above it.

[0027] In this solution, the system uses Unreal Engine (UE) as the core to construct drone flight scenarios in the drone autopilot virtual simulation scenario, providing various simulation scenarios for the drone. Simultaneously, the Airsim drone simulator is integrated based on this rendering engine to achieve accurate drone simulation. A forward-facing perspective camera is mounted on the simulated drone to simulate environmental perception capabilities during real flight.

[0028] Furthermore, in the virtual simulation scenario of drone autopilot, the system relies on the UE to develop an interactive input dialog box. Users can intuitively perceive the current drone flight scenario by combining the realistic terrain elements (such as terrain, buildings, and landmarks) presented in the simulation environment, and then perform input operations through the dialog box. The system then responds to the operation in real time to obtain the input drone piloting commands.

[0029] Furthermore, a semantic analysis model is used to perform semantic analysis on the driving instructions to obtain the virtual destination. This semantic analysis model is trained on a large amount of driving instruction data annotated with key destination information.

[0030] S120. Acquire a visual image and recognize the visual image to obtain a recognition result; wherein, the visual image refers to the image information collected by a simulated camera mounted on a drone in a drone flight scenario, and the drone flight scenario refers to a virtual space constructed by image data and point cloud data in the real environment.

[0031] Visual images refer to image information captured by a simulated camera mounted on a drone during drone flight.

[0032] In this solution, visual images are acquired using the SceneCapture2D component built into the UE in the virtual simulation scenario of drone autopilot. This component can capture visual images collected by the drone during its flight in the virtual scene in real time based on preset acquisition parameters and viewpoints, efficiently completing the entire process of visual image perception and output.

[0033] In this embodiment, the recognition result refers to the judgment and description of the key information contained in the visual image.

[0034] Specifically, image recognition technology is used to identify visual images and obtain recognition results; or, a large model is used to identify visual images and obtain recognition results. The large model is trained on a large number of visual images.

[0035] S130. If the recognition result indicates that the visual image does not contain the virtual destination, then adjust the perspective of the simulated drone so that the captured visual image contains the virtual destination, and generate visual image data.

[0036] In this scheme, the identification result indicates whether the visual image contains a virtual destination. If the identification result indicates that the visual image does not contain a virtual destination, the perspective of the simulated drone is adjusted so that the visual image captured after adjustment contains the virtual destination.

[0037] Specifically, based on the location information of the virtual destination and the current location information of the drone, the viewing angle adjustment parameters are determined; based on the viewing angle adjustment parameters, the drone's shooting angle and / or shooting direction are adjusted. For example, the drone's shooting angle is adjusted to 45°.

[0038] If the recognition result indicates that the visual image does not contain the virtual destination, the result is fed back to the backend server and then to the frontend program. At this time, the frontend will adjust the perspective of the simulated drone so that the visual image captured after adjustment contains the virtual destination.

[0039] In this embodiment, if the recognition result indicates that the visual image contains a virtual destination, then visual image data is generated.

[0040] S140. Generate UAV control information based on the visual image data and the virtual destination, and control the UAV to fly according to the UAV control information.

[0041] The drone control information includes the target flight direction and target distance.

[0042] In this scheme, visual image data and virtual destination are input into a pre-determined UAV control model. Based on the UAV control model, the visual image data and virtual destination are analyzed and predicted to generate UAV control information.

[0043] In this embodiment, in the development of the Airsim drone simulation component, the target flight time of the drone is set according to the front-end interactive interface; then the target flight direction and target flight distance of the drone are obtained through system logic, and the target distance is used as the core parameter. Combined with the preset flight time, the speed required for the drone to fly in a straight line is calculated in reverse; finally, the drone flight control interface provided by the Airsim platform is called to input the calculated speed and target flight direction parameters, thereby realizing real-time and precise control of the drone and driving it to fly stably to the virtual destination.

[0044] The technical solution of this invention obtains a virtual destination by acquiring input driving commands and performing semantic analysis on the commands; it then acquires and recognizes visual images to obtain recognition results; if the recognition results indicate that the visual image does not contain the virtual destination, the perspective of the simulated drone is adjusted so that the captured visual image includes the virtual destination, generating visual image data; based on the visual image data and the virtual destination, drone control information is generated, and the drone is controlled to fly according to the drone control information. By implementing this technical solution and generating drone control information through simulation, the accuracy, environmental adaptability, and intelligence level of drone flight control are improved, giving it greater robustness and generalization.

[0045] Example 2

[0046] Figure 2 This is a schematic diagram of a virtual simulation process for drone autopilot provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the drone flight scene construction process. Figure 2 As shown, the method includes:

[0047] S210. Acquire image data and first point cloud data of the target area; wherein, the image data is acquired from multiple angles by a camera mounted on a drone; the target area is an area existing in the real environment.

[0048] In this embodiment, image data of the target area is collected from multiple angles using a camera mounted on a drone; and point cloud technology is combined with laser scanning and other methods to obtain the first point cloud data of the target area.

[0049] The target area is a region that exists in the real environment. For example, the target area could be a region consisting of several power towers.

[0050] S220. Generate a first model based on the image data;

[0051] The first model refers to a virtual 3D model that can cover a wide range of scenes.

[0052] Specifically, the location between image data is identified by feature extraction algorithm, and aerial triangulation is performed by combining position and attitude information to calculate the exterior orientation elements of each image and construct a unified geographic coordinate system. Then, based on the aerial triangulation results, dense matching technology is used to generate a 3D point cloud to restore the spatial geometry of the scene. Based on the point cloud, a textured 3D mesh model is generated by meshing, and the texture information of the image data is mapped onto the mesh surface to generate the first model.

[0053] Optionally, generating a first model based on the image data includes:

[0054] The image data is preprocessed to generate second point cloud data with point cloud elevation values;

[0055] A triangular network is constructed based on the second point cloud data; wherein, the triangular network is composed of multiple triangular facets, and the vertices of each triangular facet are points in the second point cloud data;

[0056] Determine the image region of each triangular patch in the image data;

[0057] The texture information of the image region is mapped onto the triangular facet to generate the first model.

[0058] Among them, image processing technology is used to preprocess the image data to generate second point cloud data with point cloud elevation values.

[0059] In this embodiment, the triangular mesh is composed of multiple triangular facets, where each triangle's vertex is a point in the second point cloud data. A 3D modeling algorithm is used to construct the triangular mesh based on the second point cloud data. This 3D modeling algorithm includes Poisson reconstruction and Alpha Shapes algorithms, among others.

[0060] Furthermore, the pixel coordinates of the three vertices of each triangular facet in the image data are obtained; based on the pixel coordinates of the three vertices, the polygon region corresponding to the triangular facet in the image data is determined by a polygon fitting algorithm, and the polygon region is used as the image region.

[0061] In this scheme, pixel color information and texture detail information within the image region are extracted as texture information; a coordinate mapping relationship is established between the vertices of the triangular facet and the pixels within the image region; based on the coordinate mapping relationship, the extracted texture information is mapped pixel by pixel onto the surface of the corresponding triangular facet to complete the texture mapping and generate the first model.

[0062] Specifically, the image data is preprocessed to generate second point cloud data with attached point cloud elevation values; a triangular mesh is constructed based on the second point cloud data; then, the image region corresponding to each triangular patch in the image data is further determined; finally, the texture information contained in each image region is mapped onto the corresponding triangular patch to generate the first model.

[0063] By preprocessing the image data to generate second point cloud data with point cloud elevation values, a precise 3D spatial foundation is provided for subsequent modeling. Based on the second point cloud data, a triangular mesh composed of triangular facets is constructed, which can efficiently and stably restore the geometric shape of the target object or scene, ensuring the rationality and integrity of the model structure. Further, the image region corresponding to each triangular facet is determined and texture information is mapped, which can give the first model realistic visual texture details while retaining accurate geometric features, and finally achieve the generation of a 3D model with both high geometric accuracy and high visual realism.

[0064] S230. Generate a second model based on the point cloud elevation values ​​in the image data; wherein, the point cloud elevation values ​​represent the vertical height of each point in the point cloud relative to a preset reference plane;

[0065] In this embodiment, the second model is a virtual 3D model constructed for a single independent object (such as a building, a piece of equipment, or a utility pole).

[0066] Specifically, point clouds are extracted from image data; region segmentation is performed based on the continuity and abrupt changes in point cloud elevation values, and point clouds with similar elevation change patterns are clustered into independent target units; a geometric model is constructed for each segmented target unit; and then, texture mapping technology is used to attach the texture information of the corresponding region in the image data to the surface of the geometric model, thereby generating a second model.

[0067] Optionally, a second model is generated based on the point cloud elevation values ​​in the image data, including:

[0068] The point cloud in the first model is classified to obtain third point cloud data of different categories;

[0069] The third point cloud data of different categories are segmented respectively to identify the point cloud data corresponding to different objects in each category;

[0070] For each of the objects, extract the point cloud elevation value of the object;

[0071] A second model is constructed based on the point cloud elevation values ​​of the object.

[0072] In this embodiment, the point cloud in the first model is classified based on a preset classification model to obtain third point cloud data of different categories. Furthermore, the third point cloud data of different categories is segmented based on a preset segmentation model to identify the point cloud data corresponding to different objects within each category. The classification model and segmentation model include support vector machine models, random forest models, convolutional neural network models, etc.

[0073] The point cloud elevation values ​​for each object are acquired using photogrammetry or 3D modeling equipment. Specifically, the 3D coordinates of each point within each object are first determined, and then the elevation coordinates are extracted from these 3D coordinates as the point cloud elevation values.

[0074] In this scheme, a second model is constructed based on the point cloud elevation values ​​of each object using 3D modeling algorithms. These 3D modeling algorithms include Poisson reconstruction and Alpha Shapes algorithms.

[0075] Specifically, the point cloud data in the first model is first classified to obtain third point cloud data belonging to different categories; then, segmentation operations are performed on these different categories of third point cloud data to identify the point cloud data of different objects under each category; then, for each identified object, its corresponding point cloud elevation value is extracted; finally, based on the extracted point cloud elevation values ​​of each object, the second model is constructed.

[0076] By constructing a second model, the model's accuracy in reproducing the spatial morphology of objects and its ability to represent details are greatly improved. It can also provide a more accurate and targeted data foundation for subsequent analysis based on elevation information, significantly optimizing the efficiency and quality of the entire process from point cloud data to model construction.

[0077] S240. The second model and the first point cloud data are superimposed on the first model to construct a drone flight scene corresponding to the real environment.

[0078] In this embodiment, using the first model as a reference, the second model is loaded into the corresponding layer of the first model according to a preset spatial relationship through the layer overlay function of the 3D rendering engine. At the same time, the transparency, display priority and other parameters of the second model are adjusted to ensure that its spatial logical relationship with the first model is clearly visible. Finally, the first point cloud data is imported and overlaid on the above composite model in the form of discrete point sets. The spatial correspondence between the first point cloud data and the first and second models can be further strengthened by setting the point size and color mapping rules of the point cloud, so as to achieve accurate overlay and visualization of the three in the same view.

[0079] Optionally, the second model and the first point cloud data are overlaid onto the first model to construct a drone flight scene corresponding to the real environment, including:

[0080] The second model and the first point cloud data are superimposed on the first model to generate a base model;

[0081] The base model is defined as a drone flight scenario corresponding to the real environment; or, meteorological data is added to the base scenario, and the environment with the base scenario and meteorological conditions is defined as a drone flight scenario corresponding to the real environment.

[0082] The meteorological data includes temperature-related data, humidity-related data, air pressure-related data, wind-related data, precipitation-related data, and cloud-related data.

[0083] In this embodiment, the second model and the first point cloud data can be superimposed on the first model to generate a model with a basic scene framework; then, the basic model is directly determined as a drone flight scene that matches the real physical environment in terms of spatial structure, object distribution and other dimensions.

[0084] In this scheme, the second model can be overlaid with the first point cloud data onto the first model to generate a base model; then meteorological data is incorporated into this base scene, and finally the environment that integrates the base scene and meteorological conditions is determined as a drone flight scene that can match the real environment.

[0085] Specifically, in the construction of drone simulation scenarios, the powerful rendering and interaction capabilities of the UE engine can be relied upon to build the main framework of the scene with the first model as the core foundation. At the same time, the second model is integrated to realize the refined presentation of key objects such as buildings and facilities. The scene details and spatial dimension information are supplemented by the first point cloud data. The three work together to construct a complex simulation environment with ultra-high-definition visual effects and high fidelity.

[0086] Furthermore, it can not only directly call the weather system built into UE5, but also be further developed according to needs. With the platform's interactive interface, it supports users to flexibly adjust lighting parameters such as light intensity and light and shadow angle, as well as switch between sunny, cloudy, rainy, and snowy weather states with one click. At the same time, the system supports the rapid retrieval and seamless switching of different simulation scenarios, which greatly improves the efficiency of scenario configuration and provides high-fidelity and high-flexibility virtual environment support for various flight tests and mission simulations of UAVs.

[0087] The technical solution of this invention involves acquiring image data and first point cloud data of a target area; generating a first model based on the image data; generating a second model based on the point cloud elevation values ​​in the image data; and superimposing the second model and the first point cloud data onto the first model to construct a drone flight scenario corresponding to the real environment. By implementing this technical solution, the realism and accuracy of the drone flight scenario are effectively improved, providing a reliable scenario foundation that closely matches the actual environment for subsequent drone path planning and simulated flight. This significantly reduces the risks of drones flying in real-world environments while improving the efficiency and accuracy of related operations.

[0088] Example 3

[0089] Figure 3 This is a schematic diagram of a virtual simulation method for drone autopilot provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the virtual destination acquisition process. Figure 3 As shown, the method includes:

[0090] S310. Perform word segmentation on the virtual destination to obtain multiple word units.

[0091] In this solution, after obtaining the virtual destination, it is transmitted to the backend server of the drone autopilot virtual simulation system via the HTTP protocol, where the virtual destination is processed.

[0092] The term unit includes terms used to characterize virtual destination name elements, terms used to characterize virtual destination attribute elements, and terms used to characterize virtual destination associated scene elements.

[0093] Specifically, word segmentation algorithms are used to segment the virtual destination, resulting in multiple word units. These word segmentation algorithms include at least one of dictionary-based, statistical, and deep learning-based algorithms.

[0094] S320. Perform part-of-speech analysis on each word unit and determine the part-of-speech analysis results for each word unit.

[0095] Among them, the part-of-speech analysis results reflect the correspondence between each word unit and its corresponding part of speech.

[0096] In this embodiment, part-of-speech (POS) analysis can be performed on each word unit using natural language processing techniques to determine the POS analysis result for each word unit; alternatively, POS analysis can be performed on each word unit based on a POS analysis model to output its corresponding POS analysis result. The POS analysis model encompasses deep learning models, large-scale models, and reinforcement learning models, among others.

[0097] S330. Based on the part-of-speech analysis results, determine whether there is a first word and a second word in the virtual destination; wherein, the first word is a word indicating the execution of an action, and the second word is a word indicating the virtual destination.

[0098] In this embodiment, the first word is a word indicating the execution of an action. For example, the first word is "go to," "go to," "arrive," etc. The second word is a word indicating a virtual destination. For example, the second word is "on the horizontal bar of the tower," "above the crane," etc.

[0099] In this scheme, the part-of-speech analysis results are matched with the first word and the second word respectively to determine whether the first word and the second word exist in the virtual destination.

[0100] S340. If the virtual destination contains a first word and a second word, then determine whether the attributes of the second word meet the preset reachable attribute conditions.

[0101] The reachability attribute condition is a standard used to determine whether a specific area of ​​a target object is reachable. For example, the area above the crane has the objective conditions to be actually reachable, which meets the preset reachability attribute condition; the internal area of ​​the crane, due to limitations such as spatial structure and functional design, does not have the objective conditions to be actually reachable, which does not meet the preset reachability attribute condition.

[0102] In this scheme, when a first word and a second word exist in the virtual destination, it is determined whether the attributes of the second word meet the preset reachability attribute conditions.

[0103] S350. If the conditions are met, the virtual destination is determined as the final virtual destination.

[0104] Specifically, when the attributes of the second word meet the reachability condition, the virtual destination is determined as the final virtual destination.

[0105] In this embodiment, when the attribute of the second word does not meet the condition for reaching the attribute, the backend server will send a prompt message to the frontend platform.

[0106] S360. Acquire a visual image and recognize the visual image to obtain a recognition result; wherein, the visual image refers to image information collected by a simulated camera mounted on a drone in a drone flight scenario, and the drone flight scenario refers to a virtual space constructed by image data and point cloud data in the real environment.

[0107] S370. If the recognition result indicates that the visual image does not contain the virtual destination, then adjust the perspective of the simulated drone so that the captured visual image contains the virtual destination, and generate visual image data.

[0108] S380. Generate UAV control information based on the visual image data and the virtual destination, and control the UAV to fly according to the UAV control information.

[0109] The technical solution of this invention involves segmenting a virtual destination into multiple word units; performing part-of-speech (POS) analysis on each word unit to determine the POS analysis result; determining whether a first word and a second word exist in the virtual destination based on the POS analysis result; if the first word and the second word exist in the virtual destination, determining whether the attribute of the second word meets the preset reachability attribute conditions; if it meets the conditions, determining the virtual destination as the final virtual destination; acquiring and recognizing a visual image to obtain a recognition result; if the recognition result indicates that the visual image does not contain the virtual destination, adjusting the perspective of the simulated drone to make the captured visual image contain the virtual destination, generating visual image data; generating drone control information based on the visual image data and the virtual destination, and controlling the drone's flight based on the drone control information. By implementing this technical solution, invalid text without actual directional meaning or lacking reachability can be effectively excluded, ensuring that the final determined virtual destination is semantically clear and compliant with attributes, providing an accurate and reliable data foundation for subsequent path planning operations based on the virtual destination, significantly improving the accuracy and efficiency of related applications, and reducing resource waste caused by invalid or erroneous target information.

[0110] Example 4

[0111] Figure 4This is a flowchart of a virtual simulation process for drone autopilot provided in Embodiment 4 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the drone control information generation process. Figure 4 As shown, the method includes:

[0112] S410. Obtain the input driving command and perform semantic analysis on the driving command to obtain the virtual destination.

[0113] S420. Acquire a visual image and recognize the visual image to obtain a recognition result; wherein, the visual image refers to image information collected by a simulated camera mounted on a drone in a drone flight scenario, and the drone flight scenario refers to a virtual space constructed by image data and point cloud data in the real environment.

[0114] S430. If the recognition result indicates that the visual image does not contain the virtual destination, then adjust the perspective of the simulated drone so that the captured visual image contains the virtual destination, and generate visual image data.

[0115] S440. Input the visual image data and the virtual destination into a pre-determined UAV control model and output initial UAV control information; wherein, the UAV control information includes flight direction and distance; the distance is the distance between the current frame camera and the virtual destination.

[0116] The initial UAV control information includes the initial flight direction and initial distance.

[0117] In this scheme, visual image data and virtual destination are input into a pre-determined UAV control model, which then analyzes and predicts the visual images and virtual destination to generate initial UAV control information.

[0118] The UAV control model is trained based on various pre-acquired visual image data and virtual destinations. The construction of the UAV control model is supported by a large-scale model.

[0119] S450. Adjust the initial UAV control information to obtain UAV control information.

[0120] In this embodiment, since the UAV control model has the characteristic of non-unique output, even if the input parameters are exactly the same, the output results obtained in each calculation may be different. In order to ensure that the final UAV control information is accurate and stable, the initial UAV control information needs to be adjusted and optimized in a targeted manner.

[0121] The UAV control information consists of the target flight direction and the target distance.

[0122] Specifically, knowledge graphs can be used to adjust the initial flight direction in the initial UAV control information; and large models can be used to adjust the initial distance in the initial UAV control information.

[0123] Optionally, adjusting the initial UAV control information to obtain UAV control information includes:

[0124] Based on the initial flight direction of the UAV, knowledge related to adjusting the flight direction is obtained from a knowledge graph; wherein, the knowledge graph includes nodes related to the UAV's flight direction;

[0125] By analyzing the knowledge related to adjusting the flight direction, a control strategy for adjusting the flight direction is determined, and the initial flight direction is adjusted according to the control strategy to obtain the target flight direction.

[0126] In this embodiment, the core operation process of the knowledge graph in the UAV flight direction standardization scenario is as follows: First, a unified standard for flight direction is defined, which consists of two main categories: basic directions and composite directions. The basic directions cover the six core single-dimensional directions of the UAV: ​​forward, backward, up, down, left, and right. Composite directions are formed by combining the basic directions, including four diagonal dimensions: upper left, upper right, lower left, and lower right, together constructing a complete UAV flight direction standardization system.

[0127] In this solution, a knowledge graph is a structured knowledge storage model. In the specific domain of UAV flight direction, its core components are nodes related to the flight direction. The nodes are connected to each other through well-defined relationships, ultimately forming a knowledge network.

[0128] Among them, the nodes related to flight direction include basic attribute nodes and adjustment rule nodes. Basic attribute nodes include the UAV's own parameters and initial flight direction parameters; adjustment rule nodes include known rules for direction calibration, for example, the known rule for direction calibration is to adjust the UAV angle by 45°.

[0129] In this embodiment, the initial flight direction of the UAV is the core, and knowledge directly related to adjusting from the initial flight direction to the standard direction is selected from the knowledge graph.

[0130] Among them, the control strategy is to transform the knowledge analysis results related to flight direction adjustment into a set of operation instructions that the UAV can directly recognize and execute.

[0131] Furthermore, the knowledge related to adjusting the flight direction is analyzed to determine the control strategy for adjustment. The flight control system of the UAV executes the control strategy and adjusts the initial flight direction according to the control strategy to obtain the target flight direction.

[0132] Adjusting initial drone control information based on knowledge graphs can help drones quickly and reasonably adjust their flight direction in complex environments, effectively improving the safety, flexibility, and mission execution efficiency of drone flights. It also reduces the risks of flight deviations, mission delays, and even equipment damage caused by improper direction adjustments, providing strong support for the stable application of drones in various fields such as aerial photography, logistics, and inspection.

[0133] Optionally, adjusting the initial UAV control information to obtain UAV control information further includes:

[0134] Obtain a first distance and a second distance; wherein, the first distance is the actual distance between the camera in the current frame and the virtual destination; and the second distance is the distance between the camera in the previous frame and the virtual destination.

[0135] Based on the first distance, the second distance, and the drone speed in the previous frame, a third distance is predicted using the drone control model; wherein, the third distance is the predicted distance between the camera in the current frame and the virtual destination;

[0136] If the third distance is greater than the first distance, then the first distance is taken as the target distance.

[0137] In this embodiment, a large model is used to calculate the first distance between the current frame camera and the virtual destination, and to calculate the second distance between the previous frame camera and the virtual destination.

[0138] In this scheme, the first distance, the second distance, and the drone speed of the previous frame are input into the drone control model. Based on the drone control model, the first distance, the second distance, and the drone speed of the previous frame are predicted, and the third distance between the camera and the virtual destination in the current frame is output.

[0139] Furthermore, the third distance is compared with the first distance. If the third distance is greater than the first distance, the first distance is taken as the target distance.

[0140] By using a comparative screening mechanism, misjudgments of target distance caused by model prediction bias or instantaneous data fluctuations can be avoided. This provides accurate distance data for subsequent control operations such as path planning and speed adjustment of the UAV, ensuring the control accuracy and operational stability of the UAV as it moves toward the virtual destination.

[0141] S460. Control the drone to fly according to the drone control information.

[0142] The technical solution of this invention involves acquiring input driving commands and performing semantic analysis to obtain a virtual destination; acquiring and recognizing visual images to obtain recognition results; if the recognition result indicates that the visual image does not contain the virtual destination, adjusting the perspective of the simulated drone to ensure that the captured visual image includes the virtual destination, thus generating visual image data; inputting the visual image data and the virtual destination into a pre-determined drone control model to output initial drone control information; adjusting the initial drone control information to obtain drone control information; and controlling the drone's flight based on the drone control information. By implementing this technical solution and generating drone control information through simulation, the accuracy, environmental adaptability, and intelligence level of drone flight control are improved, giving it greater robustness and generalization.

[0143] Example 5

[0144] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0145] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0146] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a virtual simulation method for drone autopilot.

[0148] In some embodiments, a drone autopilot virtual simulation method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the drone autopilot virtual simulation method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute a drone autopilot virtual simulation method by any other suitable means (e.g., by means of firmware).

[0149] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0154] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0155] According to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a drone program comprising a drone program carried on a non-transitory computer-readable medium, the drone program containing program code for performing the methods shown in the flowcharts. In such embodiments, the drone program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0156] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0157] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An unmanned aerial vehicle automatic driving virtual simulation method, characterized in that, The method comprises the following steps: acquiring input driving instructions and performing semantic analysis on the driving instructions to obtain a virtual destination; acquiring a visual image and performing recognition on the visual image to obtain a recognition result; wherein the visual image refers to image information collected by a simulated camera mounted on a simulated unmanned aerial vehicle in an unmanned aerial vehicle flight scene, and the unmanned aerial vehicle flight scene refers to a virtual space constructed by image data and point cloud data in a real environment; if the recognition result indicates that the virtual destination is not contained in the visual image, adjusting the visual angle of the simulated unmanned aerial vehicle so that the visual image captured contains the virtual destination to generate visual image data; generating unmanned aerial vehicle control information according to the visual image data and the virtual destination, and controlling the unmanned aerial vehicle to fly according to the unmanned aerial vehicle control information.

2. The method of claim 1, wherein, The method for constructing the unmanned aerial vehicle flight scene comprises the following steps: acquiring image data and first point cloud data of a target region; wherein the image data is collected from multiple angles by a camera mounted on an unmanned aerial vehicle; and the target region is a region existing in a real environment; generating a first model based on the image data; generating a second model according to point cloud elevation values in the image data; wherein the point cloud elevation values represent the vertical height of each point in the point cloud relative to a preset reference surface; superimposing the second model and the first point cloud data onto the first model to construct an unmanned aerial vehicle flight scene corresponding to the real environment.

3. The method of claim 2, wherein, The method for generating a first model based on the image data comprises the following steps: preprocessing the image data to generate second point cloud data with point cloud elevation values; constructing a triangular mesh based on the second point cloud data; wherein the triangular mesh is composed of multiple triangular facets, and the vertex of each triangular facet is a point in the second point cloud data; determining the image region of each triangular facet in the image data; mapping the texture information of the image region to the triangular facet to generate the first model.

4. The method of claim 2, wherein, The method for generating a second model according to the point cloud elevation values in the image data comprises the following steps: classifying the point cloud in the first model to obtain third point cloud data of different categories; respectively segmenting the third point cloud data of different categories to identify point cloud data corresponding to different objects in each category; extracting the point cloud elevation values of each object; constructing a second model according to the point cloud elevation values of the objects.

5. The method of claim 2, wherein, The method for superimposing the second model and the first point cloud data onto the first model to construct an unmanned aerial vehicle flight scene corresponding to the real environment comprises the following steps: superimposing the second model and the first point cloud data onto the first model to generate a basic model; determining the basic model as the unmanned aerial vehicle flight scene corresponding to the real environment; or adding meteorological data to the basic scene, and determining the environment constructed with the basic scene and the meteorological conditions as the unmanned aerial vehicle flight scene corresponding to the real environment.

6. The method of claim 1, wherein, The method for acquiring input driving instructions and performing semantic analysis on the driving instructions to obtain a virtual destination comprises the following steps: performing word segmentation on the virtual destination to obtain multiple word units; performing part-of-speech analysis on each word unit to determine a part-of-speech analysis result corresponding to each word unit; based on the part-of-speech analysis result, determining whether the first word and the second word exist in the virtual destination; wherein the first word is a word representing action execution, and the second word is a word representing a virtual destination; if the first word and the second word exist in the virtual destination, determining whether an attribute of the second word meets a preset reachable attribute condition; if the attribute meets the preset reachable attribute condition, determining the virtual destination as a final virtual destination.

7. The method of claim 1, wherein, generating drone control information based on the visual image data and the virtual destination, including: inputting the visual image data and the virtual destination into a predetermined drone control model to output initial drone control information; wherein the drone control information includes a flight direction and a distance; the distance is a distance between a current frame camera and the virtual destination; adjusting the initial drone control information to obtain drone control information.

8. The method of claim 7, wherein, The adjustment of the initial drone control information to obtain drone control information includes: obtaining knowledge related to the adjustment of the flight direction from a knowledge graph according to the initial flight direction of the drone; wherein the knowledge graph includes nodes related to the flight direction of the drone; by analyzing the knowledge related to the adjustment of the flight direction, determining a control strategy for adjusting the flight direction, and adjusting the initial flight direction according to the control strategy to obtain a target flight direction.

9. The method of claim 7, wherein, The adjustment of the initial drone control information to obtain drone control information also includes: obtaining a first distance and a second distance; wherein the first distance is a real distance between a current frame camera and a virtual destination; the second distance is a distance between a last frame camera and the virtual destination; based on the first distance, the second distance, and a last frame speed of the drone, predicting a third distance based on a drone control model; wherein the third distance is a predicted distance between the current frame camera and the virtual destination; if the third distance is greater than the first distance, the first distance is taken as a target distance.

10. A drone comprising a drone program, wherein the drone program, when executed by a processor, implements an automatic driving virtual simulation method of a drone according to any one of claims 1-9.

Images