Data generation method and device, electronic equipment and computer readable storage medium
Generating multimodal samples through flight simulation solves the problems of low collection efficiency and limitation in the prior art, and achieves efficient and flexible multimodal samples acquisition and training effects.
Patent Information
- Application Number
- CN202411716910.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-07-08
AI Technical Summary
When collecting multimodal samples, the prior art requires driving the flight equipment to perform flight missions. The efficiency is low and is limited by hardware, region and other factors, making it difficult to meet training needs.
By obtaining modal types and mission configuration information, flying simulations generate multiple simulated flight data, and multi-modal samples are generated based on these data, and task planning is used to improve data acquisition efficiency and correlation.
It realizes efficient generation of multimodal samples that meet the needs of multiple training tasks without being restricted by the hardware and geographical limitations of the flight equipment, enhancing the correlation and training effect between modal samples.
Smart Images
Figure CN120277861A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular, to a data generation method, apparatus, electronic device, and computer-readable storage medium. Background Art
[0002] Multimodal machine learning is a method of learning from multimodal samples and improving itself. Multimodal machine learning relies on a large number of multimodal samples. For example, before training a multimodal model related to a flight device, it is necessary to collect a large amount of data that meets the requirements during the flight mission of the flight device and organize it to obtain multimodal samples that meet the training requirements.
[0003] However, each time multimodal samples are collected, it is necessary to drive the flight device to perform a flight mission, resulting in low collection efficiency. Moreover, due to limitations such as the hardware and geographical location of the flight device, it may be difficult to collect multimodal samples that meet the training requirements. Summary of the Invention
[0004] In view of the above, embodiments of this application provide a data generation method, apparatus, electronic device, and computer-readable storage medium, which can not only improve the acquisition efficiency of multimodal samples, but also enable the multimodal samples to meet the requirements of various training tasks for sample data.
[0005] Embodiments of this application provide a data generation method, including: Obtain the modality type configuration information of the multimodal sample to be generated and the task configuration information of the flight device, where the task configuration information is used to configure a simulated flight task; Perform flight simulation based on the task configuration information and the modality type configuration information to obtain various simulated flight data; Wherein, the modality types of the various simulated flight data belong to the modality types configured in the modality type configuration information; Generate the multimodal sample based on the various simulated flight data.
[0006] Embodiments of this application can perform flight simulation based on task configuration information and modality type configuration information, which not only makes the acquisition of multimodal samples no longer restricted by the hardware, geographical location, climate, etc. of the flight device, improves the acquisition efficiency of multimodal samples, but also enables the generated multimodal samples to flexibly meet various types of training tasks.
[0007] In some embodiments, generating the multimodal sample based on the various simulated flight data includes: Obtain the simulation generation time sequence of the various simulated flight data; Bind the generation time sequence and the multiple simulated flight data to obtain the multimodal sample.
[0008] Embodiments of the present application can enable various simulation flight data in multi-modal samples to have a collaborative connection in time, which is beneficial to strengthening the correlation between various modal samples.
[0009] In some embodiments, flight simulation is performed based on the task configuration information and the modal type configuration information to obtain various simulation flight data, including: Perform task planning on the flight simulation process of the flight device based on a preset task planning model, the task configuration information, and the modal type configuration information to obtain the task planning information of the flight device; Among them, the task planning model is created based on a large language model; Determine the various simulation flight data based on the task planning information.
[0010] Embodiments of the present application process the task configuration information and the modal type configuration information based on a preset task planning model, which can realize the task planning of the flight simulation process, obtain smaller and more easily processed subtasks, thereby facilitating the processing of complex simulation flight tasks and modal types. In addition, embodiments of the present application use a large language model as the task planning model, and the large language model has excellent planning capabilities. Therefore, the subtasks planned based on this are more easily processed by electronic devices.
[0011] In some embodiments, the task planning information includes the maneuver operation sequence of the flight device; the various simulation flight data includes the trajectory sequence of the flight device; The determining the various simulation flight data based on the task planning information includes: Perform flight path planning based on a preset trajectory planning model and the maneuver operation sequence to obtain the trajectory sequence.
[0012] In some embodiments, the various simulation flight data includes the payload data of the flight device; The task planning information includes payload feature descriptions and a task execution sequence, and the task execution sequence is used to describe each payload task to be executed by the flight device; The determining the various simulation flight data based on the task planning information includes: Determine the payload maneuver operation sequence based on the task execution sequence; Generate the payload data based on the payload maneuver operation sequence, the payload feature descriptions, and a preset payload data generator.
[0013] In some embodiments, the mission planning information includes the mission scenario sequence of the flying device; generating the payload data based on the payload maneuvering operation sequence, the payload feature description, and a preset payload data generator includes: Generating payload features based on the payload maneuvering operation sequence, the payload feature description, and the payload data generator; Rendering the payload features based on the mission scenario sequence to obtain the payload data.
[0014] In some embodiments, after generating the multi-modal samples based on the multiple simulation flight data, it further includes: Obtaining a multi-modal model to be trained; Training the multi-modal model based on the multi-modal samples to obtain a trained multi-modal model.
[0015] An embodiment of the present application further provides a data generation device, including: An acquisition module, configured to acquire the modal type configuration information of the multi-modal samples to be generated and the mission configuration information of the flying device, where the mission configuration information is used to configure the simulation flight mission; A simulation module, configured to perform flight simulation based on the mission configuration information and the modal type configuration information to obtain multiple simulation flight data; Wherein, the modal types of the multiple simulation flight data belong to the modal types configured in the modal type configuration information; A generation module, configured to generate the multi-modal samples based on the multiple simulation flight data.
[0016] An embodiment of the present application further provides an electronic device, where the electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the above data generation method.
[0017] An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and when the computer instructions run on an electronic device, the electronic device is made to execute the above data generation method. Description of the Drawings
[0018] Figure 1 It is a step flowchart of a data generation method according to an embodiment of the present application.
[0019] Figure 2 It is a system architecture diagram of a data generation system according to an embodiment of the present application.
[0020] Figure 3It is a sub-step flowchart of step 102 provided according to an embodiment of the present application.
[0021] Figure 4 It is a schematic structural diagram of a data generation device provided according to an embodiment of the present application.
[0022] Figure 5 It is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. Detailed implementation manners
[0023] In order to more clearly understand the above objects, features and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation manners. It should be noted that, without conflict, the implementation manners of the present application and the features in the implementation manners can be combined with each other.
[0024] Many specific details are set forth in the following description in order to provide a thorough understanding of the present application. The described implementation manners are only a part of the implementation manners of the present application, rather than all of the implementation manners.
[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application herein are only for the purpose of describing specific implementation manners, and are not intended to limit this application.
[0026] Further, it should be noted that in this article, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.
[0027] "At least one" in this application means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.
[0029] The source or form of each type of information can be referred to as a modality. For example, humans have tactile, auditory, visual, and olfactory senses; information media include speech, video, text, etc.; and various sensors such as radar, infrared, and accelerometers, each of the above can be referred to as a modality. Multimodality refers to multiple sources or forms of information.
[0030] Multimodal machine learning is a method of learning from multimodal samples and improving itself. Multimodal machine learning relies on a large number of multimodal samples. For example, before training a multimodal model related to a flight device, it is necessary to collect a large amount of data that meets the requirements during the flight mission of the flight device and organize it to obtain multimodal samples that meet the training requirements.
[0031] However, each time multimodal samples are collected, it is necessary to drive the flight device to perform a flight mission, resulting in low collection efficiency. Moreover, due to factors such as geographical location and the hardware of the flight device, it may be difficult to collect multimodal samples that meet the training requirements.
[0032] In some embodiments, each unimodal data generator can be used to generate respective unimodal data, and then, the set of the unimodal data can be used as multimodal samples.
[0033] For example, each unimodal data generator can be used to generate unimodal samples of various different modality types such as visible light images, infrared images, Synthetic Aperture Radar (SAR) images, and flight trajectories, and then, these unimodal data can be integrated as multimodal samples.
[0034] The above embodiments do not require driving the flight device to perform a flight mission to collect multimodal samples. Instead, generated unimodal data is used to obtain multimodal samples, so that the collection of data is not restricted by factors such as the hardware of the flight device, geographical location, and climate.
[0035] However, during the actual acquisition process of the flight device, there is a mutual correlation between the single-modal data collected, and there is a cooperative relationship between them. For example, there are correlations in the flight time sequence, pose, field of view range, and environmental state of the single-modal data. However, the above-mentioned single-modal data generators have no connection with each other. Therefore, the single-modal data generated by them lack correlation with each other, resulting in the multi-modal samples obtained by this method being unable to reflect the correlation and cooperative information between the sample data. Furthermore, the multi-modal model cannot learn the cooperative relationship between the modal data in the multi-modal sample, making it difficult to achieve good training results.
[0036] In view of the above, an embodiment of the present application provides a data generation method.
[0037] This data generation method can perform flight simulation according to the task configuration information and the modal type configuration information, which not only makes the acquisition of multi-modal samples no longer restricted by the hardware, region, etc. of the flight device, improves the acquisition efficiency of multi-modal samples, but also enables the generated multi-modal samples to flexibly meet various types of training tasks.
[0038] The above data generation method can be applied in one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to processors, microprogrammed control units (MCUs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc. The electronic device can be a portable electronic device (such as a mobile phone, tablet computer), a personal computer, a server, etc.
[0039] Figure 1 It is a flowchart of the steps of an embodiment of the data generation method of the present application. Figure 2 It is a flowchart of the steps of an embodiment of the data generation system of the present application.
[0040] Refer to Figure 1 and Figure 2 As shown, this data generation method may include the following steps.
[0041] Step 101, obtain the modal type configuration information of the multi-modal sample to be generated and the task configuration information of the flight device.
[0042] The flight device can be a drone, an airship, etc., and the embodiments of the present application do not limit this.
[0043] The modality type configuration information is used to configure the modality types of the sample data included in the multi-modal sample. For example, the modality types may include visible light images, infrared images, SAR radar images, flight trajectories, measurement data of an Inertial Measurement Unit (IMU), etc., but are not limited thereto.
[0044] The task configuration information is used to configure the simulation flight task.
[0045] For example, the simulation flight task may include aerial photography, plant protection, express delivery, disaster relief, wildlife observation, mapping, power line inspection, disaster relief, image shooting, battlefield reconnaissance mission, equipment inspection, etc., but is not limited thereto.
[0046] In some embodiments, the electronic device may further obtain environment configuration information, which is used to configure the flight environment when the flight device executes the simulation flight task, so as to facilitate the subsequent generation of multi-modal samples matching the flight environment.
[0047] For example, the environment configuration information may include various meteorological environments such as rainy days, foggy days, sunny days, strong winds, gentle winds, cloudy days, terrain environments, locations, etc., but is not limited thereto.
[0048] In the embodiments of the present application, the configuration contents of the modality type configuration information, the task configuration information of the flight device, and the environment configuration information can all be configured according to actual application requirements. The user can perform information configuration targeted at the function of the multi-modal model to be trained and the effects to be achieved.
[0049] For example, the electronic device may provide a user interface for the user to configure the flight environment, modality types, and simulation flight task. The electronic device can obtain the environment configuration information, task configuration information, modality type configuration information, etc. input by the user from the user interface. For example, a flight simulation software is integrated in the electronic device, and the flight simulation software can be a Figure 2 data generation system as shown. The configuration unit in the data generation system can obtain these configuration information.
[0050] Step 102: Perform flight simulation based on the task configuration information and the modality type configuration information to obtain various simulation flight data.
[0051] That is, the electronic device simulates the flight data generated by the flight device during the execution of the simulation task based on the task configuration information and the modality type configuration information, and the flight data is the simulation flight data.
[0052] For example, a flight simulation software is integrated in the electronic device, and the flight simulation software can be Figure 2The data generation system shown, the flight simulation software can simulate the flight data generated by the flight equipment during the execution of the simulation task to obtain a variety of simulated flight data.
[0053] The simulated flight data may include a trajectory sequence generated by the flight equipment during the execution of the simulation task, payload data generated by the payload equipment carried by the flight equipment, and so on.
[0054] For example, assuming the payload equipment is a camera, the payload data may refer to visible light images captured by the camera; assuming the payload equipment is an IMU, the payload data may be the measurement data of the IMU, and assuming the payload equipment is a radar, the payload data may be radar images.
[0055] The modal types of the above-mentioned various simulated flight data belong to the modal types configured in the modal type configuration information.
[0056] For example, when the modal types indicated by the modal type configuration information include visible light images and radar images, the electronic device will simulate the visible light images and radar images generated by the flight equipment during the execution of the simulation task to obtain a variety of simulated flight data.
[0057] For another example, when the modal types indicated by the modal type configuration information include the flight equipment trajectory and visible light images, the electronic device will simulate the trajectory sequence and visible light images generated by the flight equipment during the execution of the simulation task to obtain a variety of simulated flight data.
[0058] In some embodiments, step 102 can also perform flight simulation in combination with the environment configuration information. Specifically, step 102 may include: performing flight simulation based on the task configuration information, modal type configuration information, and environment configuration information to obtain a variety of simulated flight data. For example, the user can input the task configuration information, modal type configuration, and environment configuration information in the user interface so that the flight simulation software can perform flight simulation.
[0059] When performing flight simulation in the embodiments of the present application, the environment configuration information can also be referred to, which can make the generated multi-modal samples more in line with the requirements of the training task.
[0060] In some embodiments, the steps of flight simulation may include: the electronic device performs task planning on the flight simulation process of the flight equipment based on a preset task planning model, task configuration information, and modal type configuration information to obtain the task planning information of the flight equipment. For example, the task planning model can perform type decomposition based on the modal type configuration information and task division based on the task configuration information to obtain each task planning information; determining the various simulated flight data based on the task planning information.
[0061] When the user is still configuring the environment of the simulated flight mission, determining multiple types of simulated flight data based on the mission planning information may include: determining multiple types of simulated flight data based on the mission planning information and the environment configuration information.
[0062] The mission planning information may include simulation subtasks.
[0063] The simulation subtasks may include, but are not limited to, maneuver operation sequences, task execution sequences, descriptions of the characteristics of each payload, and sequences of task scenarios to be simulated, etc.
[0064] The maneuver operation sequence is used to describe the maneuver actions performed by the flight device during flight and the execution order of the maneuver actions. For example, the first step is takeoff, the second step is cruising, the third step is turning, the fourth step is diving, etc.
[0065] The maneuver operation sequence may also include the starting point and the ending point of the section that the flight device has to pass through during the execution of the maneuver action.
[0066] The task execution sequence is used to describe the various payload tasks to be executed by each payload device in the flight device and the execution order.
[0067] For example, assuming that the simulated flight mission includes a reconnaissance mission, a rescue mission, and a shooting mission, the electronic device can obtain the payload devices involved in the flight device when performing the reconnaissance mission and the payload tasks of the payload devices, such as the gimbal task; obtain the payload devices involved in the flight device when performing the rescue mission and the payload tasks of the payload devices, such as the rescue material delivery task of the object delivery device; obtain the payload devices involved in the flight device when performing the shooting and the payload tasks, such as the photographing task of the camera, so as to obtain the task execution sequence.
[0068] Payload characteristic description: used to represent the payload characteristic type of the payload data. For example, if the payload data includes an infrared image, the payload characteristic description includes the infrared distribution characteristic. The electronic device can disassemble the modal type configuration information according to a preset mission planning model to obtain the payload characteristic description in the mission planning information.
[0069] The task scenario sequence represents the changes in the surrounding scenes and environment as the pose and position of the flight device change during the execution of the simulated flight mission by the flight device.
[0070] The task scenario sequence can be determined based on the task scenario characteristics output by the mission planning model.
[0071] The task scenario characteristic refers to the task scenario of the simulated flight mission. That is, the electronic device can parse the task configuration information based on a preset mission planning model to obtain the task scenario characteristic.
[0072] For example, assume that the task configuration information includes performing a rescue mission in a forest. The task planning model will parse this task configuration information, generate the characteristics of the forest scene, and generate a series of objects and targets to be rescued, such as the positions of trapped persons, in the forest scene.
[0073] In some other embodiments, the task scene sequence can also be determined according to the environmental configuration information and the task scene characteristics, where the environmental configuration information is used to configure the flight environment of the simulation flight task, such as climate, terrain, etc.
[0074] Specifically, the electronic device can integrate the environmental configuration information into the task scene characteristics to obtain the changes in the surrounding scene and environment as the pose and position of the flight device change during the execution of the simulation flight task.
[0075] In this embodiment, by disassembling the task configuration information and the modal type configuration information based on a preset task planning model, smaller and more easily processed subtasks can be obtained, thus facilitating the processing of complex simulation flight tasks and data collection tasks.
[0076] Furthermore, the task planning model can be created based on a large language model.
[0077] In the embodiment of the present application, a large language model is used as the task planning model. The large language model has excellent planning and understanding capabilities. Therefore, the subtasks disassembled based on this are more easily processed by the electronic device.
[0078] Furthermore, when the simulation flight data includes the trajectory sequence of the flight device, the task planning information can include the maneuver operation sequence of the flight device.
[0079] The electronic device can perform flight path planning based on a preset trajectory planning model and the maneuver operation sequence to obtain the trajectory sequence.
[0080] The maneuver operation sequence includes the starting point and the ending point of each section. Therefore, a preset trajectory planning model and the starting point and the ending point can be used to perform path planning for each section, thereby obtaining the trajectory sequence corresponding to the execution of the simulation flight task.
[0081] In some embodiments, when the multiple simulation flight data includes the payload data of the flight device, the task planning information can include the payload feature description and the task execution sequence, and the task execution sequence is used to describe the various payload tasks to be executed by the flight device.
[0082] The payload maneuver operation sequence refers to the maneuver operations required for the payload device to perform the payload task and the order of these maneuver operations.
[0083] The electronic device can determine a payload maneuver operation sequence based on the task execution sequence; and generate the payload data based on the payload maneuver operation sequence, the payload feature description, and a preset payload data generator.
[0084] For example, the electronic device generates the payload feature based on the payload maneuver operation sequence, the payload feature description, and a preset payload data generator, and then generates the payload data based on the payload feature.
[0085] Among them, the payload feature is used to describe the feature of the payload data, and the payload data refers to the simulation data generated by each payload device.
[0086] The payload feature description is used to represent the type of payload feature of the payload data.
[0087] The payload data generator can adopt payload data simulation technology, which can generate corresponding payload features based on the payload feature description, the payload maneuver sequence, etc.
[0088] Furthermore, the above generation of the payload data based on the payload maneuver operation sequence, the payload feature description, and a preset payload data generator may include: generating each payload feature based on the payload maneuver operation sequence, each payload feature description, and each preset payload data generator; rendering the payload features based on the task scenario sequence to obtain the payload data.
[0089] In the embodiments of the present application, by obtaining the task scenario sequence and rendering the payload features based on the scenario change sequence, payload data that can present the flight environment can be obtained. For example, the payload data of the image type, and the payload data of the image type can be a visualization image, an infrared image, a radar image, a video, etc., but not limited thereto.
[0090] In some embodiments, when the simulated flight data includes a trajectory sequence and payload data, the payload data can be generated in combination with the trajectory sequence, the payload data generator, etc., so as to establish an association relationship between the payload data and the trajectory sequence, so that the multi-modal samples can reflect the payload data generated at each position in the trajectory sequence.
[0091] Specifically, referring to Figure 2 and Figure 3 as shown, step 102 may include: Step 301, perform task planning on the flight simulation process of the flight device based on a preset task planning model, task configuration information, and modal type configuration information to obtain the task planning information of the flight device.
[0092] In some embodiments, the task planning information may include the maneuver operation sequence and task execution sequence of the flight device.
[0093] Among them, the maneuver operation sequence is used to describe the maneuver actions performed by the flight device during flight and the execution order of the maneuver actions.
[0094] The mission execution sequence is used to describe the various payload tasks to be executed by each payload device in the flight device and the execution order.
[0095] The payload device refers to various devices and items carried on the flight device, which are used to perform specific tasks or complete specific functions. The payload device can include two types: sensor payload and operation payload.
[0096] The sensor payload refers to sensors used to collect data, sense the environment or conduct monitoring, such as cameras, infrared sensors, radars, etc. These payload devices can provide information such as images, videos, sounds, thermal energy, etc., so as to support the execution of flight tasks and decision-making.
[0097] The operation payload refers to devices used to perform specific tasks or functions, such as grasping devices, launchers, communication systems, etc. These payload devices can be used in multiple fields such as rescue operations, search tasks, cargo transportation, agricultural spraying, etc.
[0098] For example, the mission planning model can disassemble the mission based on the mission configuration information and the modal type configuration information, and decompose it into subtasks that the flight device can understand. For example, the first step is takeoff, the second step is cruise, the third step is turning, the fourth step is diving, the fifth step is camera shooting, and the sixth step is pan-tilt recovery.
[0099] In some embodiments, the mission planning information may further include: payload feature description and scene change sequence.
[0100] Payload feature description: used to represent the payload feature type of the payload data.
[0101] For example, if the payload data includes an infrared image, the payload feature description includes the infrared distribution feature; if the payload data includes a SAR image, the payload feature description includes the SAR feature; if the payload data includes a visible light image, the payload feature description includes the feature map of the visible light image; if the payload data includes a laser image, the payload feature description includes the lidar imaging feature, etc.; if the payload data includes IMU measurement data, the payload feature description includes the feature of the IMU measurement data.
[0102] The number and type of the above-generated payload feature descriptions are determined according to the modal types configured by the modal type configuration information, so that the generated payload data can meet the requirements of the modal types configured in the modal type configuration information.
[0103] The scene change sequence represents the changes in the surrounding scene and environment during the process of the flight device performing the simulated flight mission, along with the changes in the pose and position of the flight device.
[0104] In some embodiments, the task scenario sequence may be determined based on the task scenario features output by the task planning model.
[0105] The task scenario features are used to represent the task scenario of the simulated flight task.
[0106] For example, in the case where the flight task includes an aerial photography task, the task scenario may include the object to be photographed, etc.; in the case where the flight task includes express delivery, the task scenario may include the pick-up address, the delivery address of the express, and the type of the express, etc. of the express delivery scenario; in the case where the flight task includes a rescue task, the task scenario may include a fire rescue scenario, a field search and rescue scenario, etc., and may also include the location of the trapped person, etc.; in the case where the flight task includes an inspection task, the task scenario may include whether there are illegal buildings around, etc. of the flight task scenario.
[0107] In some other embodiments, the task scenario sequence may also be determined based on the task scenario features and the environment configuration information, where the environment configuration information is used to configure the flight environment of the simulated flight task, such as climate, terrain, etc. That is, the task scenario sequence of this embodiment may include the fusion features of the environment configuration and the task scenario features, and their change order. The electronic device may fuse the environment configuration information in the task scenario features to obtain the changes in the surrounding scenarios and environment as the pose and position of the flight device change during the execution of the simulated flight task by the flight device.
[0108] After the electronic device obtains the task planning information, it may determine the multiple simulated flight data based on the task planning information. For example, perform the following steps 302 to 304.
[0109] Step 302, perform flight path planning based on a preset trajectory planning model and a maneuver operation sequence to obtain a trajectory sequence.
[0110] The preset trajectory planning model (i.e., the flight trajectory generator as shown in Figure 2 ) may adopt a trajectory planning algorithm, such as the Dijkstra algorithm, the A* algorithm, etc., but is not limited thereto, and the embodiments of the present application do not limit this.
[0111] Step 303, determine the payload maneuver operation sequence based on the task execution sequence.
[0112] The payload maneuver operation sequence refers to the maneuver operations required for the payload device to perform the payload task and the order of these maneuver operations.
[0113] In some embodiments, the electronic device may store the mapping relationship between each payload task and the payload maneuver operation sequence, and the electronic device may obtain the payload maneuver operation sequence corresponding to each payload task in the task execution sequence based on these mapping relationships.
[0114] For example, for a payload mission with a camera shooting mission, the payload maneuver operation sequence may include operations such as zooming in and out of the camera's focal length, and shooting operations.
[0115] In some other embodiments, the electronic device may also use machine learning techniques such as reinforcement learning to determine the payload maneuver operation sequence corresponding to each payload mission in the task execution sequence.
[0116] The payload maneuver operation sequence can be coordinated with the maneuver operation sequence, that is, when a certain maneuver operation is performed in the flying device, the payload device performs the corresponding maneuver operation.
[0117] For example, the maneuver operation sequence includes diving and moving forward, the payload mission includes the camera shooting mission, and the payload maneuver operation sequence can zoom in and out of the camera's focal length in coordination with diving and moving forward.
[0118] Step 304, generate payload data based on the payload maneuver operation sequence, payload feature description, trajectory sequence, and a preset payload data generator.
[0119] Payload data refers to the simulation data generated by each simulated payload device.
[0120] Refer again to Figure 2 As shown, the payload data generator may include, but is not limited to, an infrared data feature generator, an SAR radar data feature generator, a video data feature generator, a lidar feature generator, etc.
[0121] In some embodiments, step 304 may include: Step 3041, generate payload features based on the payload maneuver operation sequence, payload feature description, trajectory sequence, and a preset payload data generator.
[0122] The payload data generator can combine the payload maneuver operation sequence, each payload feature description, and the trajectory sequence to generate the payload features collected at each position in the flight trajectory sequence.
[0123] The payload features are used to describe the features of the payload data.
[0124] Step 3042, render the payload features based on the task scenario sequence to obtain the payload data.
[0125] The modal type of the payload data belongs to the modal types configured in the modal type configuration information.
[0126] In the case where the payload data is image data or video data, its payload feature is an image feature, and the electronic device can render the payload feature based on the task scenario sequence to generate a realistic image, thereby obtaining the payload data.
[0127] For example, the payload data may include, but is not limited to, IMU data, infrared images, videos, etc.
[0128] Step 103: Generate multimodal samples based on multiple simulation flight data.
[0129] In some embodiments, step 103 may include: obtaining the generation time sequence of the multiple simulation flight data; binding the generation time sequence and the multiple simulation flight data to obtain the multimodal samples.
[0130] The generation time sequence is the time order in which multiple simulation flight data are generated as the simulation flight task is executed.
[0131] For example, a time synchronization unit as shown in Figure 2 may be used to unify the time, that is, to unify multiple simulation flight data to the same time period and attach a time tag, which can reflect the generation time sequence of the simulation flight data, so that each sample in the multimodal samples carries time sequence information, and thus the correlation between them can be established.
[0132] In some embodiments, after step 103, it may further include: obtaining a multimodal model to be trained; training the multimodal model based on the multimodal samples to obtain a trained multimodal model.
[0133] The multimodal model is a model capable of processing multimodal data generated by a flight device.
[0134] For example, the multimodal model may be a real-time driving path planning model for a flight device, and the path planning model adjusts the trajectory, pose, speed, etc. of the flight device according to the real-time collected multimodal data. The function of the multimodal model is only an example, and the embodiments of the present application do not limit this.
[0135] The embodiments of the present application can perform flight simulation according to the task configuration information and the modality type configuration information, which not only makes the acquisition of multimodal samples no longer restricted by the hardware, location, climate, etc. of the flight device, improves the acquisition efficiency of multimodal samples, but also enables users to flexibly configure according to the training tasks of the multimodal model, so that the generated multimodal samples can flexibly meet the training requirements of various types of training tasks.
[0136] In addition, the embodiments of the present application can make the multiple simulation flight data in the multimodal samples have a cooperative connection in time, realize the matching of the time sequence and payload data such as multi-band image data, and is beneficial to strengthening the correlation between each modal sample.
[0137] Based on the same idea as the data generation method in the above embodiment, the present application also provides a data generation device, which can be used to execute the above data generation method. For ease of explanation, the structural diagram of the data generation device embodiment only shows the parts related to the embodiment of the present application. It can be understood by those skilled in the art that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.
[0138] like Figure 4 As shown, the data generating device includes an acquisition module 401, a simulation module 402 and a generation module. In some embodiments, the above modules can be programmable software instructions stored in a memory and can be called and executed by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware solidified in the processor.
[0139] The acquisition module 401 is used to acquire the modality type configuration information of the multimodal samples to be generated and the mission configuration information of the flight equipment, and the mission configuration information is used to configure the simulation flight mission.
[0140] The simulation module 402 is used to perform flight simulation based on the mission configuration information and the mode type configuration information to obtain a variety of simulated flight data.
[0141] Among them, the modal types of the multiple simulated flight data belong to the modal types configured in the modal type configuration information.
[0142] The generating module 403 is used to generate the multi-modal samples based on the multiple simulated flight data.
[0143] Figure 5 This is a schematic diagram of an embodiment of an electronic device of the present application.
[0144] The electronic device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, the steps in the above data generation method embodiment are implemented, for example Figure 1 Steps 101 to 103 are shown.
[0145] Exemplarily, the computer program 40 can also be divided into one or more modules / units, which are stored in the memory 20 and executed by the processor 30. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 40 in the electronic device 100. For example, it can be divided into Figure 4The acquisition module 401, simulation module 402, and generation module shown.
[0146] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 100, and does not constitute a limitation on the electronic device 100. It may include more or fewer components than shown, or combine some components, or different components. For example, the electronic device 100 may also include input / output devices, network access devices, buses, etc.
[0147] The processor 30 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, a single-chip microcomputer, or the processor 30 may also be any conventional processor, etc.
[0148] The memory 20 can be used to store the computer program 40 and / or modules / units. The processor 30 realizes various functions of the electronic device 100 by running or executing the computer program and / or modules / units stored in the memory 20, and by calling the data stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the electronic device 100 (such as audio data, etc.). In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
[0149] If the integrated module / unit of the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of the present application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.
[0150] In several embodiments provided by the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the above-described electronic device embodiments are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation.
[0151] In addition, each functional unit in the various embodiments of the present application can be integrated in the same processing unit, or each unit can exist physically alone, or two or more units can be integrated in the same unit. The above-mentioned integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional modules.
[0152] For those skilled in the art, it is obvious that the present application is not limited to the details of the above-described exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. In addition, obviously, the word "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or electronic devices stated in the electronic device claims can also be implemented by the same unit or electronic device through software or hardware. The words such as first and second are used to represent names and do not represent any specific order.
[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.
Claims
1. A data generation method, characterized in that, Including: Obtain the modality type configuration information of the multi-modal sample to be generated and the task configuration information of the flight device, where the task configuration information is used to configure the simulation flight task; Perform flight simulation based on the task configuration information and the modality type configuration information to obtain various simulation flight data; Among them, the modality types of the various simulation flight data belong to the modality types configured in the modality type configuration information; Generate the multi-modal sample based on the various simulation flight data.
2. The data generation method according to claim 1, wherein The generating the multi-modal sample based on the various simulation flight data includes: Obtain the generation time sequences corresponding to the various simulation flight data respectively; Bind the generation time sequences and the various simulation flight data to obtain the multi-modal sample.
3. The data generation method according to claim 1 or 2, characterized in that, The performing flight simulation based on the task configuration information and the modality type configuration information to obtain various simulation flight data includes: Perform task planning on the flight simulation process of the flight device based on a preset task planning model, the task configuration information, and the modality type configuration information to obtain the task planning information of the flight device; Among them, the task planning model is created based on a large language model; Determine the various simulation flight data based on the task planning information.
4. The data generation method according to claim 3, wherein The task planning information includes the maneuver operation sequence of the flight device; The various simulation flight data include the trajectory sequence of the flight device; The determining the various simulation flight data based on the task planning information includes: Perform flight path planning based on a preset trajectory planning model and the maneuver operation sequence to obtain the trajectory sequence.
5. The data generation method according to claim 3, wherein The various simulation flight data include the payload data of the flight device; The task planning information includes payload feature description and task execution sequence, and the task execution sequence is used to describe the various payload tasks to be executed by the flight device; The determining the various simulation flight data based on the task planning information includes: Determine the payload maneuver operation sequence based on the task execution sequence; Generate the payload data based on the payload maneuver operation sequence, the payload feature description, and a preset payload data generator.
6. The data generation method according to claim 5, wherein The task planning information further includes the task scenario sequence of the flight device, and the generating the payload data based on the payload maneuver operation sequence, the payload feature description, and a preset payload data generator includes: Generate payload features based on the payload maneuver operation sequence, the payload feature description, and the payload data generator; Render the payload features based on the task scenario sequence to obtain the payload data.
7. The data generation method according to claim 1, wherein After generating the multi-modal sample based on the various simulation flight data, it further includes: Obtain the multi-modal model to be trained; Train the multi-modal model based on the multi-modal sample to obtain a trained multi-modal model.
8. A data generation device, characterized in that, Including: An acquisition module, configured to obtain the modality type configuration information of the multi-modal sample to be generated and the task configuration information of the flight device, where the task configuration information is used to configure the simulation flight task; A simulation module, configured to perform flight simulation based on the task configuration information and the modality type configuration information to obtain various simulation flight data; Among them, the modal types of the multiple simulation flight data belong to the modal types configured in the modal type configuration information; A generation module, configured to generate the multi-modal samples based on the multiple simulation flight data.
9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the data generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute the data generation method according to any one of claims 1 to 7.