Unmanned aerial vehicle body data generation method, system and device based on multi-modal large model and AirSim
Through the AirSim simulator and multimodal big model, virtual drones fly in a simulation environment and generate embodied data, solving the problem of low data acquisition efficiency during drone inspection, and achieving efficient and convenient data acquisition and Intelligent flight route planning.
Patent Information
- Application Number
- CN202411138013.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-29
AI Technical Summary
The existing drone inspections are slow to collect embossed data, high cost and low efficiency, making it difficult to obtain a large amount of embossed data, which affects the training of natural language models and the intelligent planning of drone flight routes.
Using a drone embodied data generation method based on multimodal large model and AirSim, a simulated flight environment is built through the AirSim simulator, a virtual drone, a target power equipment and an obstacle model is built, and a control instruction set of virtual drone is generated. Make virtual drones fly in a simulation environment, obtain multi-view data and attitude information in real time, and build multi-modal embodied data.
No need for manual manipulation of drones improves data acquisition efficiency, reduces costs, shortens acquisition time, and provides a large number of reliable embodied data sets for natural language model training and intelligent planning of drone flight routes.
Smart Images

Figure CN119048717A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of embodied intelligence, and in particular to a method, system and device for generating embodied data of a drone based on a multimodal large model and AirSim. Background Art
[0002] With the development of civil unmanned aerial vehicle technology and the gradual advancement of new power system construction, we will promote the integration of drones with basic operation applications, promote three-dimensional inspection of power transmission, transformation and distribution, realize remote intelligent inspection, and improve production operation efficiency.
[0003] At present, drone inspections are mainly carried out by flying to power equipment along manually preset flight routes. However, in actual distribution network inspection scenarios, flying according to manually preset flight routes lacks flexibility and operational robustness, which also results in poor inspection quality. Therefore, it is proposed to intelligently plan the flight routes of drones through natural language models.
[0004] The construction of natural language models requires training and testing with a large amount of embodied data to optimize the natural language models, thereby improving the flexibility of drone operations and improving the quality of inspections. As for the acquisition of embodied data, currently, data can only be collected by manually controlling drones to fly inspections in power scenarios, which makes embodied data collection slow and costly. In addition, it takes a long time to collect data, and it is difficult to obtain a large amount of embodied data, resulting in low data collection efficiency.
[0005] Therefore, a method, system and device for generating UAV embodied data based on a multimodal large model and AirSim are proposed. Summary of the invention
[0006] To solve the above problems, the present invention provides a method, system and device for generating embodied data of unmanned aerial vehicles based on a multimodal large model and AirSim, so as to solve the problems of slow embodied data collection, high cost and low collection efficiency in traditional technologies.
[0007] The present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising:
[0008] Build a simulated flight environment based on the AirSim simulator;
[0009] In the simulated flight environment, build virtual drones, target power equipment, and obstacle models;
[0010] Generate a set of control instructions for a virtual drone through a large multimodal model;
[0011] The virtual drone flies to the target power equipment in a simulated flight environment according to the control instruction set to perform inspections, and acquires multi-view data, drone attitude information, inspection videos and corresponding control instructions in real time to construct multimodal embodied data.
[0012] Preferably, the present invention provides a method for generating embodied data of a drone based on a multimodal large model and AirSim, the steps of: building a simulated flight environment based on the AirSim simulator; comprising:
[0013] Obtain environmental data for power scenarios;
[0014] Read environmental data through the data interface of the AirSim simulator and build the initial simulated flight environment;
[0015] The real scene model in the initial simulated flight environment is structurally modified by model modification software to obtain a simulated flight environment; optionally, the method includes:
[0016] The suspended objects, redundant objects and real scene models with incomplete model structures in the initial simulated flight environment are repaired.
[0017] Preferably, the present invention provides a method for generating embodied data of a UAV based on a multimodal large model and AirSim, the steps of: constructing a virtual UAV, a target power equipment and an obstacle model in a simulated flight environment; comprising:
[0018] By acquiring equipment parameters and appearance images of the UAV, a virtual UAV is constructed in the simulated flight environment;
[0019] By acquiring specification parameters of various types of power facilities and equipment, target power equipment is constructed in the simulated flight environment;
[0020] Constructing an obstacle model in the vicinity of the target power equipment in the simulated flight environment; optionally, including:
[0021] Use tree models to create barriers around transmission lines.
[0022] Preferably, the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to the control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; including:
[0023] Randomly select the starting point location information of the virtual drone, and set the target power equipment as the end point location information;
[0024] The control instruction data set plans the predicted flight path of the virtual UAV according to the obstacle model in the simulated flight environment and the safe flight requirements of the UAV in the power scenario;
[0025] The virtual drone flies according to the predicted flight path, and records the first-person perspective video and drone attitude information of the virtual drone in the simulated flight environment in real time;
[0026] The virtual drone flies to the target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains the inspection video;
[0027] Acquire a third-person perspective video in real time according to the flight process and inspection process of the virtual drone in the simulated flight environment;
[0028] The first perspective video includes: starting point video information, preset shooting point video information and landing point video information;
[0029] The multimodal embodied data includes: predicted flight path, first-person perspective video, drone attitude information, inspection video, third-person perspective video and corresponding control instruction data set.
[0030] Preferably, the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, the steps of: a virtual drone flies according to the predicted flight path, and records the first-person perspective video and drone attitude information in the simulated flight environment in real time; including:
[0031] When the virtual drone is flying according to the predicted flight path, it performs emergency obstacle avoidance and adjusts the predicted flight path when an obstacle is found;
[0032] After the virtual drone arrives at the target power equipment, obtaining an actual flight path of the virtual drone;
[0033] The multimodal embodied data also includes an actual flight path.
[0034] Preferably, the method for generating drone embodied data based on a multimodal large model and AirSim provided by the present invention comprises the steps of: the virtual drone flies to the target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains the inspection video; comprising:
[0035] Selecting the best shooting point according to the location of the target power equipment;
[0036] When the virtual drone flies to the target power equipment, the virtual drone adjusts its flight position to an optimal shooting point to shoot and obtain an inspection video.
[0037] Preferably, the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to the control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; and also includes:
[0038] After the virtual drone flies to the target power equipment in the simulated flight environment according to the control instruction set, detecting whether the virtual drone successfully avoids obstacles during the flight process;
[0039] After detecting that the obstacle avoidance fails during the flight of the virtual drone, the control instruction set is optimized, and the virtual drone is reset to the starting point position for another flight.
[0040] Preferably, the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to the control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; and also includes:
[0041] The control instruction set provides an external control interface for a user to control the virtual drone to fly in the simulated flight environment and obtain user control instructions;
[0042] The multimodal embodied data also includes user control instructions.
[0043] Preferably, the present invention provides a drone embodied data generation system based on a multimodal large model and AirSim, comprising:
[0044] The simulation environment building module is used to build a simulated flight environment based on the AirSim simulator;
[0045] A model building module is used to build virtual drones, target power equipment, and obstacle models in a simulated flight environment;
[0046] The instruction building module is used to generate a set of control instructions for a virtual drone through a multi-modal large model;
[0047] The data generation module is used to control the virtual drone to fly to the target power equipment for inspection in a simulated flight environment according to the control instruction set, and to obtain multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multi-modal embodied data.
[0048] Preferably, the present invention provides a drone embodied data generation device based on a multimodal large model and AirSim, comprising:
[0049] A data processor and a memory, wherein the memory stores a computer program that can be executed on the data processor, and the data processor is used to implement the method according to any one of claims 1 to 8 when executing the computer program.
[0050] Compared with traditional technologies, the beneficial effects of the present invention are as follows: the present invention proposes a method, system and device for generating embodied data of unmanned aerial vehicles based on a multimodal large model and AirSim, which does not require manual control of the unmanned aerial vehicle for data collection. The multimodal embodied data can be acquired by performing flight inspections by virtual unmanned aerial vehicles in a simulated flight environment. While improving the efficiency of data collection, the safety risks of data collection by unmanned aerial vehicles in traditional technologies are avoided, the data collection time is shortened, and the data collection cost is reduced, so as to realize the convenient acquisition of a large amount of multimodal embodied data. While providing a reliable data set for the natural language model for training and testing, the intelligent planning of the flight route of the unmanned aerial vehicle according to the natural language model is further realized. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0052] Figure 1 It is a flowchart of a method for generating embodied data of UAV based on a multimodal large model and AirSim;
[0053] Figure 2 The figure is a structural diagram of a UAV embodied data generation system based on a multimodal large model and AirSim. DETAILED DESCRIPTION
[0054] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.
[0055] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.
[0056] Embodiment 1:
[0057] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, referring to Figure 1 ,include:
[0058] Build a simulated flight environment based on the AirSim simulator;
[0059] In the simulated flight environment, build virtual drones, target power equipment, and obstacle models;
[0060] Generate a set of control instructions for a virtual drone through a large multimodal model;
[0061] The virtual drone flies to the target power equipment in a simulated flight environment for inspection according to the control command set, and obtains multi-view data, drone attitude information, inspection video and corresponding control commands in real time to construct multimodal embodied data.
[0062] In the above embodiments, a simulated flight environment is built based on the AirSim simulator, and a virtual drone, target power equipment and obstacle models are constructed in the simulated flight environment; a control instruction set for the virtual drone is generated through a multimodal large model; the virtual drone flies to the target power equipment in the simulated flight environment according to the control instruction set for inspection, and multi-perspective data, drone attitude information, inspection video and corresponding control instructions of the virtual drone during flight are obtained in real time to obtain multimodal embodied data.
[0063] In the above embodiments, the multimodal large model can be implemented as ChatGPT, Claude, and Wenxinyiyan.
[0064] The beneficial effects of the above technology are: by controlling the virtual drone to fly in a simulated flight environment built based on the AirSim simulator according to a set of control instructions, inspections are carried out on the target power equipment, thereby realizing the acquisition of multimodal embodied data; the above method does not require manual control of the drone for data collection, and the acquisition of multimodal embodied data can be realized by flying and inspecting the virtual drone in a simulated flight environment. While improving the data collection efficiency, it avoids the safety risks of data collection by drones in traditional technologies, shortens the data collection time, and reduces the data collection cost, so as to realize the convenient acquisition of a large amount of multimodal embodied data, and while providing a reliable data set for the natural language model for training and testing, it further realizes the intelligent planning of the drone flight route according to the natural language model.
[0065] Embodiment 2:
[0066] The embodiment of the present invention provides a method for generating embodied data of a drone based on a multi-modal large model and AirSim, comprising the steps of: building a simulated flight environment based on an AirSim simulator; comprising:
[0067] Obtain environmental data for power scenarios;
[0068] Read environmental data through the data interface of the AirSim simulator and build the initial simulated flight environment;
[0069] The real scene model in the initial simulated flight environment is structurally modified by the model modification software to obtain the simulated flight environment; optionally, the method includes:
[0070] The real-life model with suspended objects, redundant objects, and incomplete model structure in the initial simulated flight environment is repaired.
[0071] In the above embodiments, the AirSim simulator is an open source software developed by Microsoft Corporation, which can be used to build a simulation environment for unmanned vehicles and drones, and can achieve physical and visual real simulation.
[0072] In the above embodiments, the environmental data of the power scene is read through the data interface of the AirSim simulator to build an initial simulated flight environment; and the real-scene model in the initial simulated flight environment is structurally trimmed through the modeling software to obtain the simulated flight environment.
[0073] The beneficial effects of the above technology are: through the AirSim simulator, the initial simulated flight environment is built, and the real-scene model in the initial simulated flight environment is refined through the model repair software, thereby effectively improving the texture clarity of the simulated flight environment and avoiding the interference of suspended objects and redundant objects in the simulated flight environment on the flight of the virtual drone.
[0074] Embodiment 3:
[0075] The embodiment of the present invention provides a method for generating embodied data of a UAV based on a multimodal large model and AirSim, comprising the steps of: constructing a virtual UAV, a target power equipment and an obstacle model in a simulated flight environment; comprising:
[0076] By acquiring the equipment parameters and appearance images of the UAV, a virtual UAV is constructed in a simulated flight environment;
[0077] By obtaining the specification parameters of various power facilities and equipment, the target power equipment is constructed in the simulated flight environment;
[0078] Construct an obstacle model in the vicinity of the target power equipment in the simulated flight environment; optional, including:
[0079] Use tree models to create barriers around transmission lines.
[0080] In the above embodiments, a virtual drone is constructed in a simulated flight environment by acquiring equipment parameters and appearance images of the drone; and target power equipment is constructed in a simulated flight environment by acquiring specification parameters of various types of power facility equipment.
[0081] In the above embodiments, an obstacle model is constructed in the vicinity of the target electric power equipment.
[0082] In the above embodiments, the power facilities and equipment include cables, insulators, and pole towers.
[0083] Embodiment 4:
[0084] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to a control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; comprising:
[0085] Randomly select the starting point location information of the virtual drone, and set the target power equipment as the end point location information;
[0086] The control command data set plans the predicted flight path of the virtual drone based on the obstacle model in the simulated flight environment and the safety flight requirements of the drone in the power scenario;
[0087] The virtual drone flies according to the predicted flight path and records the first-person video and drone attitude information of the virtual drone in the simulated flight environment in real time;
[0088] The virtual drone flies to the target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains the inspection video;
[0089] According to the flight process and inspection process of the virtual drone in the simulated flight environment, the third-person perspective video is obtained in real time;
[0090] First-person video, including: starting point video information, preset shooting point video information, and landing point video information;
[0091] Multimodal embodied data, including: predicted flight path, first-person video, drone attitude information, inspection video, third-person video, and corresponding control instruction data sets.
[0092] In the above embodiments, the multi-perspective data includes a first-perspective video and a third-perspective video.
[0093] In the above embodiments, there is an obstacle model on the straight line connecting the starting position and the end position of the virtual drone, and the starting position is at a certain distance from the obstacle, so that the virtual drone has an initial velocity before encountering the obstacle model.
[0094] In the above embodiments, the control instruction data set plans the predicted flight path of the virtual drone based on the starting position information, end position information and obstacle model of the virtual drone and the safe flight requirements of the drone in the power scenario, and the virtual drone flies according to the predicted flight path.
[0095] In the above embodiments, the safety flight requirements for drones in power scenarios include power safety regulations such as the "Electric Power Safety Work Regulations" and the "State Grid Corporation Electric Power Safety Work Regulations", which describe in detail the safety flight requirements for drones in power scenarios.
[0096] In the above embodiments, during the flight of the virtual drone, the first-person perspective video and drone attitude information are recorded in real time, and when the drone flies to the target power equipment, the electronic devices and power facilities of the target power equipment are inspected to obtain the inspection video.
[0097] In the above embodiments, third-person perspective videos of the flight process and inspection process of the virtual drone in the simulated flight environment are obtained.
[0098] The beneficial effects of the above technology are: through the control instruction data set, the flight path of the virtual UAV in the simulated flight environment is planned, and through the flight and inspection process of the virtual UAV in the simulated flight environment, the first-person video, UAV attitude information, inspection video, and third-person video are obtained, thereby realizing the acquisition of multimodal embodied data.
[0099] Embodiment 5:
[0100] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising the steps of: a virtual drone flies according to a predicted flight path, and records first-person perspective video and drone attitude information in a simulated flight environment in real time; comprising:
[0101] When the virtual drone is flying according to the predicted flight path, it will perform emergency obstacle avoidance and adjust the predicted flight path when it finds obstacles;
[0102] After the virtual UAV arrives at the target power equipment, the actual flight path of the virtual UAV is obtained;
[0103] Multimodal embodied data, also including actual flight paths.
[0104] In the above embodiments, when the virtual drone is flying, it finds an obstacle and performs emergency obstacle avoidance, adjusts the predicted flight path, and obtains the actual flight path of the virtual drone after arriving at the target power equipment.
[0105] The beneficial effects of the above technology are: by enabling the virtual drone to avoid obstacles during flight, the intelligent flight of the virtual drone is realized, and the actual flight path of the virtual drone is obtained, thereby realizing the acquisition of multimodal embodied data.
[0106] Embodiment 6:
[0107] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising the steps of: a virtual drone flies to a target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains an inspection video; comprising:
[0108] Select the best shooting point according to the location of the target power equipment;
[0109] When the virtual drone flies to the target power equipment, the virtual drone adjusts its flight position to the best shooting point to shoot and obtain the inspection video.
[0110] In the above embodiments, the best shooting point is selected according to the location of the target power equipment, and the virtual drone flies to the best shooting point to shoot the target power equipment and obtain the inspection video.
[0111] The beneficial effect of the above technology is that by selecting the best shooting point and controlling the virtual drone to fly to the best shooting point for shooting, while obtaining the inspection video, the inspection shooting quality of the target power equipment is effectively improved.
[0112] Embodiment 7:
[0113] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to a control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; and further comprising:
[0114] After the virtual drone flies to the target power equipment in the simulated flight environment according to the control command set, it is detected whether the virtual drone successfully avoids obstacles during the flight;
[0115] After detecting that the virtual drone fails to avoid obstacles during flight, the control instruction set is optimized and the virtual drone is reset to the starting point for another flight.
[0116] In the above embodiments, after the virtual drone flies to the target power equipment according to the control instruction set, it is detected whether the virtual drone collides with an obstacle during the flight. After detecting that the virtual drone fails to avoid the obstacle, the control instruction set is optimized and the virtual drone is reset to the starting position for another flight.
[0117] In the above embodiments, it is detected that the virtual drone does not collide with any obstacle during the flight process, and the actual flight path of the virtual drone is obtained.
[0118] The beneficial effects of the above technology are: after the virtual drone flies to the target power equipment, it is detected whether the virtual drone collides with an obstacle during the flight. After detecting that the obstacle avoidance is successful, the actual flight path of the virtual drone is obtained. After detecting that the obstacle avoidance fails, the control instruction set is optimized so that the actual flight path of the virtual drone can effectively avoid obstacles. At the same time, the optimization of the control instruction set generated by the multimodal large language model is realized.
[0119] Embodiment 8:
[0120] The embodiment of the present invention provides a method for generating drone embodied data based on a multimodal large model and AirSim, comprising the steps of: a virtual drone flies to a target power equipment in a simulated flight environment according to a control instruction set to perform an inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data; and further comprising:
[0121] The control instruction set provides an external control interface for users to control the virtual drone to fly in a simulated flight environment and obtain user control instructions;
[0122] Multimodal embodied data also includes user control instructions.
[0123] In the above embodiments, the control instruction set provides an external control interface, and the user controls the virtual drone in the simulated flight environment to perform flight operations through the external control interface.
[0124] The beneficial effect of the above technology is that the user can control the virtual drone in the simulated flight environment through the external control interface and generate corresponding user control instructions.
[0125] Embodiment 9:
[0126] The embodiment of the present invention provides a drone embodied data generation system based on a multimodal large model and AirSim, referring to Figure 2 ,include:
[0127] The simulation environment building module is used to build a simulated flight environment based on the AirSim simulator;
[0128] A model building module is used to build virtual drones, target power equipment, and obstacle models in a simulated flight environment;
[0129] The instruction building module is used to generate a set of control instructions for a virtual drone through a multi-modal large model;
[0130] The data generation module is used to control the virtual drone to fly to the target power equipment for inspection in a simulated flight environment according to a set of control instructions, and to obtain multi-perspective data, drone attitude information, inspection video and corresponding control instructions in real time to construct multimodal embodied data.
[0131] In the above embodiments, the simulation environment building module builds a simulated flight environment based on the AirSim simulator, and builds a virtual drone, target power equipment and obstacle model in the simulated flight environment through the model building module; the instruction building module generates a control instruction set for the virtual drone through a multimodal large model; the data generation module controls the virtual drone to fly to the target power equipment for inspection in the simulated flight environment according to the control instruction set, and constructs multimodal embodied data by acquiring multi-perspective data, drone attitude information, inspection video and corresponding control instructions during the flight of the virtual drone in real time.
[0132] The beneficial effects of the above technology are: the data generation module in the system controls the virtual drone to fly in a simulated flight environment built based on the AirSim simulator according to a set of control instructions, and inspects the target power equipment, thereby realizing the acquisition of multimodal embodied data; the above system does not require manual control of the drone for data collection. By flying and inspecting the virtual drone in a simulated flight environment, it can realize the acquisition of multimodal embodied data. While improving the data collection efficiency, it avoids the safety risks of data collection by drones in traditional technologies, shortens the data collection time, and reduces the data collection cost, so as to realize the convenient acquisition of a large amount of multimodal embodied data. While providing a reliable data set for the natural language model for training and testing, it further realizes the intelligent planning of the drone flight route according to the natural language model.
[0133] Embodiment 10:
[0134] The embodiment of the present invention provides a device for generating drone embodied data based on a multimodal large model and AirSim, comprising:
[0135] A data processor and a memory, wherein a computer program is stored in the memory and can be executed on the data processor, and the data processor is used to implement the method of any of the above embodiments when executing the computer program.
[0136] In the above embodiments, the memory stores a computer program, and when the data processor executes the computer program, any of the above embodiments is implemented to output multimodal embodied data.
[0137] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.
[0138] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0139] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.
[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for generating drone embodied data based on a multimodal large model and AirSim, characterized in that: include: Build a simulated flight environment based on the AirSim simulator; In the simulated flight environment, build virtual drones, target power equipment, and obstacle models; Generate a set of control instructions for a virtual drone through a large multimodal model; The virtual drone flies to the target power equipment in a simulated flight environment according to the control instruction set to perform inspections, and acquires multi-view data, drone attitude information, inspection videos and corresponding control instructions in real time to construct multimodal embodied data.
2. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 1, characterized in that: The steps: building a simulated flight environment based on the AirSim simulator; including: Obtain environmental data for power scenarios; Read environmental data through the data interface of the AirSim simulator and build the initial simulated flight environment; The real scene model in the initial simulated flight environment is structurally modified by model modification software to obtain a simulated flight environment.
3. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 1, characterized in that: The steps: constructing a virtual UAV, target power equipment and obstacle model in a simulated flight environment; including: By acquiring equipment parameters and appearance images of the UAV, a virtual UAV is constructed in the simulated flight environment; By acquiring specification parameters of various types of power facilities and equipment, target power equipment is constructed in the simulated flight environment; An obstacle model is constructed in the vicinity of the target power equipment in the simulated flight environment.
4. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 1, characterized in that: The steps include: the virtual drone flies to the target power equipment in a simulated flight environment according to the control instruction set to perform inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multi-modal embodied data; including: Randomly select the starting point location information of the virtual drone, and set the target power equipment as the end point location information; The control instruction data set plans the predicted flight path of the virtual UAV according to the obstacle model in the simulated flight environment and the safe flight requirements of the UAV in the power scenario; The virtual drone flies according to the predicted flight path, and records the first-person perspective video and drone attitude information of the virtual drone in the simulated flight environment in real time; The virtual drone flies to the target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains the inspection video; According to the flight process and inspection process of the virtual UAV in the simulated flight environment, a third-person perspective video is acquired in real time; The first perspective video includes: starting point video information, preset shooting point video information and landing point video information; The multimodal embodied data includes: predicted flight path, first-person perspective video, drone attitude information, inspection video, third-person perspective video and corresponding control instruction data set.
5. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 4, characterized in that: The step of: the virtual drone flies according to the predicted flight path, and records the first-person perspective video and drone attitude information of the virtual drone in the simulated flight environment in real time; includes: When the virtual drone is flying according to the predicted flight path, it performs emergency obstacle avoidance and adjusts the predicted flight path when an obstacle is found; After the virtual drone arrives at the target power equipment, obtaining an actual flight path of the virtual drone; The multimodal embodied data also includes an actual flight path.
6. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 4, characterized in that: The step: the virtual drone flies to the target power equipment, inspects the electronic devices and power facilities of the target power equipment, and obtains the inspection video; includes: Selecting the best shooting point according to the location of the target power equipment; When the virtual drone flies to the target power equipment, the virtual drone adjusts its flight position to an optimal shooting point to shoot and obtain an inspection video.
7. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 4, characterized in that: The steps include: the virtual drone flies to the target power equipment in a simulated flight environment according to the control instruction set to perform inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multi-modal embodied data; and further comprising: After the virtual drone flies to the target power equipment in the simulated flight environment according to the control instruction set, detecting whether the virtual drone successfully avoids obstacles during the flight process; After detecting that the obstacle avoidance fails during the flight of the virtual drone, the control instruction set is optimized, and the virtual drone is reset to the starting point position for another flight.
8. The method for generating drone embodied data based on a multimodal large model and AirSim according to claim 4, characterized in that: The steps include: the virtual drone flies to the target power equipment in a simulated flight environment according to the control instruction set to perform inspection, and acquires multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multi-modal embodied data; and further comprising: The control instruction set provides an external control interface for a user to control the virtual drone to fly in the simulated flight environment and obtain user control instructions; The multimodal embodied data also includes user control instructions.
9. A drone embodied data generation system based on a multimodal large model and AirSim, characterized in that: include: The simulation environment building module is used to build a simulated flight environment based on the AirSim simulator; A model building module is used to build virtual drones, target power equipment, and obstacle models in a simulated flight environment; The instruction building module is used to generate a set of control instructions for a virtual drone through a multi-modal large model; The data generation module is used to control the virtual drone to fly to the target power equipment for inspection in a simulated flight environment according to the control instruction set, and to obtain multi-view data, drone attitude information, inspection video and corresponding control instructions in real time to construct multi-modal embodied data.
10. A drone embodied data generation device based on a multimodal large model and AirSim, characterized in that: include: A data processor and a memory, wherein the memory stores a computer program that can be executed on the data processor, and the data processor is used to implement the method according to any one of claims 1 to 8 when executing the computer program.
Citation Information
Cited By
Unmanned aerial vehicle inspection method and device for power transformation equipment
CN119472766A
Unmanned aerial vehicle cooperation system based on intelligent unmanned aerial vehicle and multi-vehicle cooperation control method
CN119690134A
Unmanned aerial vehicle heuristic path planning method and device based on large language model and medium
CN120066113A
Engineering project intelligent surveying and mapping method based on remote unmanned aerial vehicle
CN120745241A
An engineering project intelligent reconnaissance surveying and mapping method based on a remote unmanned aerial vehicle
CN120745241B
