A training data generation method and related apparatus
By generating a solid-color background image of a drone model in a 3D virtual space and synthesizing it into a target background image, the high cost and low efficiency of drone recognition model training data acquisition are solved, achieving efficient and accurate training data generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING LIZHENG TECH CO LTD
- Filing Date
- 2026-04-08
- Publication Date
- 2026-06-12
Smart Images
Figure CN122200237A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target detection technology, and in particular to a training data generation method and related apparatus. Background Technology
[0002] When training drone recognition models, changes in the drone's appearance due to model iterations or other reasons often necessitate the re-collection of training data. Currently, generating training data requires a team to operate the drone extensively in various real-world scenarios (such as open skies, complex buildings, and wilderness environments), simultaneously capturing photos or videos from multiple angles using cameras and video recorders. Then, professionals must manually visually examine each frame of the images or videos, marking the drone's location in each image where it appears. This method is not only time-consuming and labor-intensive but also costly and inefficient. Summary of the Invention
[0003] In view of the above problems, this application provides a training data generation method and related apparatus to reduce manual labor and improve generation efficiency. The specific solution is as follows:
[0004] The first aspect of this application provides a method for generating training data, including:
[0005] The target drone model is placed in the three-dimensional virtual space according to the first target state parameters, and a virtual camera is placed in the three-dimensional virtual space according to the first virtual camera parameters to obtain the drone deployment scene;
[0006] In the drone deployment scene, the rendering engine is invoked to generate a foreground image based on the second target state parameters, the second virtual camera parameters, and the environmental parameters of the target drone model. This results in a foreground image containing only a solid color background of the target drone model, and the position information of the target drone model in the image coordinate system and the world coordinate system is recorded.
[0007] The foreground image is processed to obtain a mask of the target drone model. The foreground image is then adjusted according to the parameters of the second virtual camera and the lighting information of the target background image. The adjusted foreground image is then composited into the target background image according to the mask, and the category of the target background image is recorded.
[0008] In one possible implementation, the step of placing a target UAV model in a three-dimensional virtual space according to a first target state parameter, and placing a virtual camera in the three-dimensional virtual space according to a first virtual camera parameter, to obtain a UAV deployment scene, includes:
[0009] Based on the position distribution, attitude angle information, and relative size of the target drone model, the target drone model is placed in the three-dimensional virtual space; and based on the position and orientation of the virtual camera, the virtual camera is set in the three-dimensional virtual space.
[0010] In one possible implementation, the step of invoking a rendering engine in the drone deployment scene to generate a foreground image based on the second target state parameters, second virtual camera parameters, and environmental parameters of the target drone model, resulting in a foreground image containing only a solid-color background of the target drone model, including:
[0011] Based on the material properties, lighting conditions, and internal parameters of the target drone model, ray tracing or rasterization rendering is performed to obtain the foreground image.
[0012] In one possible implementation, processing the foreground image to obtain a mask for the target drone model includes:
[0013] The mask is obtained by extracting the drone image region from the foreground image using chroma keying based on the solid color background of the foreground image; or...
[0014] Based on the depth map information of the foreground image, the image region of the UAV is extracted through depth testing to obtain the mask.
[0015] In one possible implementation, adjusting the foreground image based on the second virtual camera parameters and the lighting information of the target background image, and then compositing the adjusted foreground image into the target background image based on the mask, includes:
[0016] The perspective transformation of the drone region in the foreground image is performed based on the internal parameters of the virtual camera, so that the perspective effect of the target drone model is consistent with that of the target background image;
[0017] The brightness, contrast, highlights, and shadows of the foreground image are adjusted according to the lighting direction and color temperature of the background image.
[0018] The adjusted foreground image is composited into the target background image according to the mask, and post-processing effects are added.
[0019] In one possible implementation, the training data generation method further includes:
[0020] Based on the location information, the parameters of the second virtual camera, and the parameters of the first virtual camera, the coordinate bounding box of the UAV in the two-dimensional image plane is determined, and the coordinate bounding box is associated with the category of the target background image and then stored.
[0021] A second aspect of this application provides a training data generation apparatus, comprising:
[0022] The scene layout module is used to place a target drone model in a three-dimensional virtual space according to the first target state parameters, and to place a virtual camera in the three-dimensional virtual space according to the first virtual camera parameters, so as to obtain a drone layout scene;
[0023] An image generation module is used to invoke the rendering engine in the drone deployment scene, generate a foreground image based on the second target state parameters, second virtual camera parameters, and environmental parameters of the target drone model, obtain a foreground image containing only a solid-color background of the target drone model, and record the position information of the target drone model in the image coordinate system and the world coordinate system; and,
[0024] An image compositing module is used to process the foreground image to obtain a mask of the target drone model, adjust the foreground image according to the parameters of the second virtual camera and the lighting information of the target background image, composite the adjusted foreground image into the target background image according to the mask, and record the category of the target background image.
[0025] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the training data generation method of the first aspect or any implementation thereof.
[0026] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:
[0027] The memory is used to store computer programs;
[0028] The processor is used to execute the computer program so that the electronic device can implement the training data generation method of the first aspect or any implementation thereof.
[0029] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the training data generation method of the first aspect or any implementation thereof.
[0030] By employing the aforementioned technical solution, the training data generation method provided in this application obtains a drone deployment scene by placing a target drone model in a three-dimensional virtual space according to first target state parameters and placing a virtual camera according to first virtual camera parameters. Further, within the drone deployment scene, a rendering engine is invoked to generate a foreground image based on the second target state parameters of the target drone model, the second virtual camera parameters, and environmental parameters. This results in a foreground image containing only a solid-color background of the target drone model, and the position information of the target drone model in the image coordinate system and the world coordinate system is recorded. Finally, the foreground image is processed to obtain a mask of the target drone model. The foreground image is then adjusted according to the second virtual camera parameters and the lighting information of the target background image. The adjusted foreground image is then composited into the target background image based on the mask, and the category of the target background image is recorded. This achieves a digital image generation process encompassing drone model acquisition, virtual camera scene deployment in virtual three-dimensional space, and foreground and background image synthesis, effectively reducing manual on-site operations and improving the efficiency of training data acquisition. Attached Figure Description
[0031] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0032] Figure 1 An architecture diagram of a training data generation system provided in this application;
[0033] Figure 2 A flowchart of a training data generation method provided in this application;
[0034] Figure 3 A structural diagram of a training data generation device provided in this application;
[0035] Figure 4 This is a structural diagram of an electronic device provided in this application. Detailed Implementation
[0036] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0037] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0038] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0039] See Figure 1 , Figure 1 A schematic diagram of the architecture of a training data generation system is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the training data generation method provided in the embodiments of this application to one or more terminals.
[0040] The terminal 100 may be equipped with a training data generation application. The application and webpage can provide an interface. The terminal 100 can receive the relevant configuration file entered by the user on the training data generation interface and send the configuration file to the server 200. The server 200 can obtain the processing result based on the received configuration file and return the processing result to the terminal 100.
[0041] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received configuration file on its own, without the need for the server to cooperate. This application embodiment is not limited to this.
[0042] The following description Figure 1 The product form of the mid-terminal 100;
[0043] The terminal 100 in this application embodiment can be a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), etc., and this application embodiment does not impose any restrictions on it.
[0044] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.
[0045] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0046] Among them, the input device can receive input data, etc.
[0047] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display the interface for generating training data, processing results, etc.
[0048] The memory can be used to store software code related to the training data generation method, the processor can execute the steps of the training data generation method, and can also schedule other units (such as the input unit and display unit mentioned above) to achieve the corresponding functions.
[0049] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.
[0050] In this embodiment of the application, the radio frequency unit can send the configuration file to the server 200 and receive the processing result sent by the server 200.
[0051] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.
[0052] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.
[0053] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.
[0054] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.
[0055] The memory can be used to store software code related to the training data generation method, the processor can execute the steps of the chip's training data generation method, and can also schedule other units to achieve the corresponding functions.
[0056] Currently, to train target detection models for drone identification, the industry mainly relies on the following physical data collection and manual processing procedures:
[0057] Field data collection: For each new model of drone that needs to be identified, the team will operate the drone and conduct a large number of flights in various real-world scenarios (such as open skies, complex buildings, and outdoor environments), while simultaneously using cameras or video cameras to capture photos or video footage from multiple angles.
[0058] Manual data annotation: After data collection, professionals need to examine massive amounts of images or videos frame by frame. By visually inspecting each image in which a drone appears, they must precisely mark the drone's location with a rectangle and assign it the correct category label (such as "quadcopter drone"). This process is usually done with annotation software, but it is still essentially a labor-intensive task.
[0059] Direct reuse and fine-tuning of datasets: When a new model appears, due to changes in appearance, size, and scale, the old dataset is usually unusable directly. Practitioners often have to repeat steps 1 and 2 above to create a new, independent dataset for the new model.
[0060] The existing solutions that rely on entity acquisition and manual annotation have the following inherent and significant drawbacks:
[0061] The economic costs are high: each data collection requires coordination of equipment, personnel, and venues, and incurs costs related to drone battery life and equipment wear and tear. Manual annotation is an even heavier expense, charged per image, and for projects requiring tens or even hundreds of thousands of annotated images, the total cost is extremely considerable.
[0062] The time cycle is lengthy: from planning and data collection, waiting for suitable weather and environment, actual flight shooting, to the subsequent lengthy annotation process, the cycle of preparing a qualified training dataset for a new aircraft model often takes several weeks or even months, which seriously slows down the speed of artificial intelligence model development and product iteration.
[0063] Poor scalability and flexibility: The solution is entirely constrained by the physical world. It cannot quickly acquire data from extreme, dangerous, or rare scenarios (such as drones in thunderstorms or near high-voltage power line towers). If the size of the drone changes, it must be started from scratch, making it impossible to accumulate and reuse knowledge or data.
[0064] Data quality and consistency are difficult to guarantee: manual annotation is subject to subjective differences and fatigue errors, resulting in inconsistent bounding box positions and sizes, which affects model training performance. In addition, due to limitations in practical conditions, the diversity of collected data in terms of pose, lighting, and scale is often insufficient, making the trained model prone to overfitting and lacking robustness.
[0065] To address the aforementioned problems, this application provides a training data generation method. The training data generation method of this application embodiment will be described in detail below with reference to the accompanying drawings.
[0066] Reference Figure 2 , Figure 2 This application provides a flowchart illustrating a training data generation method as shown in the embodiments. Figure 2 As shown, the training data generation method provided in this application embodiment may include steps S201 to S203, which will be described in detail below.
[0067] Step S201: Place the target UAV model in the three-dimensional virtual space according to the first target state parameters, and place the virtual camera in the three-dimensional virtual space according to the first virtual camera parameters to obtain the UAV deployment scene.
[0068] Specifically, for the original 3D model of a drone, a 3D modeling tool can be used to create the drone, and then a rendering tool can be used to add materials, colors, and other rendering effects to make the resulting 3D model more realistic. Based on this, the resulting 3D model is standardized to a standard coordinate system (e.g., origin at the centroid, Z-axis upwards) and its dimensions are normalized to facilitate subsequent attitude and size transformations. This results in a standardized 3D model file of the drone with a unified format, textures, and material properties. Finally, the rendered 3D model can be stored in a 3D model library.
[0069] To construct the background image library, a large number of static images and 360° panoramic images of real scenes can be collected in advance using various optical devices (such as fixed surveillance cameras, cameras, etc.) under different times, locations, and weather conditions. For user convenience, these images can be labeled with corresponding semantic tags using image recognition models or manual methods, such as: sky, clear sky, city buildings, forest, dusk, haze, presence of birds, etc. The sky portion, being particularly crucial, can be extracted automatically or semi-automatically from the image using image recognition or manual operation, facilitating subsequent background image synthesis. Finally, images with clear quality and no severe distortion are selected and subjected to standardized preprocessing such as size normalization and color level adjustment, forming a structured database of real background images with rich semantic tags and metadata (sky boundaries).
[0070] During use, users can define the configuration file for the images to be synthesized through a graphical user interface, thereby setting the task parameters. Specifically, various parameters can be set through the configuration interface of the configuration file, such as:
[0071] Select one or more target UAV 3D models from the model library, and select corresponding background images from the background image library by label or randomly. Set virtual camera parameters, UAV target state parameters, environment and rendering parameters, sky background or non-sky background, etc.
[0072] Based on the number of images to be synthesized as set by the user, the foreground image is first generated according to the above configuration file. In the virtual three-dimensional space, the target drone model is placed according to the first target state parameters, and the virtual camera is placed according to the first virtual camera parameters to obtain the drone deployment scene.
[0073] Step S202: In the drone deployment scene, call the rendering engine to generate a foreground image based on the second target state parameters, the second virtual camera parameters, and the environmental parameters of the target drone model, so as to obtain a foreground image containing only a solid color background of the target drone model, and record the position information of the target drone model in the image coordinate system and the world coordinate system.
[0074] Specifically, based on the aforementioned drone deployment scenario, the system can obtain information such as the pose and distribution of each background image and the drones within those background images according to the random distribution rules set in the configuration file. A physically based rendering engine is then invoked, and ray tracing or rasterization rendering is performed based on the set second target state parameters, second virtual camera parameters, and environmental parameters. This generates a foreground image (RGB image) containing only the drones against a solid color background (e.g., green), along with the corresponding depth map and pose matrix. This results in a drone image rendered on a solid color background, while simultaneously recording precise rendering parameters, such as the drone's position in the image coordinate system and the world coordinate system.
[0075] Step S203: Process the foreground image to obtain a mask of the target drone model, adjust the foreground image according to the parameters of the second virtual camera and the lighting information of the target background image, composite the adjusted foreground image into the target background image according to the mask, and record the category of the target background image.
[0076] Specifically, the drone image is extracted from the foreground image, and then the drone image is adjusted according to the parameters of the virtual camera and environmental parameters such as lighting. The adjusted foreground image is then composited into the selected background image based on the mask to obtain a synthetic image in a format such as .jpg that can be used for model training.
[0077] This training data generation method fundamentally changes the way training data is produced through parameterized settings. It transforms the traditional model, which heavily relies on physical entities, the environment, and human intervention, into a completely new model where data is automatically generated entirely in the digital world through software algorithms. This effectively reduces the cost of acquiring training images and, by utilizing the precise target position and pose information naturally present during virtual rendering, automatically generates pixel-level accurate annotation files while synthesizing images. This eliminates the independent, time-consuming, and expensive manual annotation step, saving labor and improving image generation efficiency.
[0078] In some embodiments, the above-mentioned placement of a target drone model in a three-dimensional virtual space according to a first target state parameter, and placement of a virtual camera in a three-dimensional virtual space according to a first virtual camera parameter, to obtain a drone deployment scene, includes:
[0079] Based on the target drone model's position distribution, attitude angle information, and relative size, the target drone model is placed in a three-dimensional virtual space; and based on the virtual camera's position and orientation, the virtual camera is set in the three-dimensional virtual space.
[0080] Specifically, users can generate multiple images at once through a configuration file. The configuration file defines the rules for the random distribution of drones (e.g., against a sky background or a non-sky background), and instantiates a set of specific parameter values for each image to be generated (e.g., determining the specific distance, attitude, background ID, etc.). Then, in the virtual 3D space, the drone model is placed according to the instantiated parameters, and the position and orientation of the virtual camera are set. Here, the placement of the virtual camera is mainly based on its extrinsic parameters (e.g., observation distance range, pitch angle range), and the drone model is placed according to its position distribution in the scene, the range of attitude angles (pitch, yaw, roll), and the range of relative size (pixel percentage).
[0081] In the drone deployment scene, the rendering engine is invoked to generate a foreground image based on the second target state parameters, second virtual camera parameters, and environmental parameters of the target drone model. This results in a foreground image containing only a solid-color background of the target drone model, including:
[0082] Based on the material properties of the target drone model, lighting conditions, and the internal parameters of the virtual camera (such as focal length, sensor size, and resolution), ray tracing or rasterization rendering is performed to obtain the foreground image.
[0083] Specifically, by invoking a physically based rendering engine, ray tracing or rasterization rendering is performed based on the set material properties, lighting conditions, and camera parameters. This generates a foreground image (RGB image) containing only the drone with a solid color background (such as green), along with the corresponding depth map and pose matrix. All precise parameters used in this rendering are recorded, including the target's position in the image coordinate system and world coordinate system, its 3D bounding box, and distance. This data serves as the source of real-world data for subsequent drone annotation.
[0084] The specific processing steps for processing the foreground image to obtain the mask of the target drone model include:
[0085] Extract the drone image region from the foreground image using chroma keying based on the solid-color background of the foreground image to obtain a mask; or,
[0086] Based on the depth map information of the foreground image, the drone image region is extracted through depth testing to obtain a mask. Specifically, using a solid color background (such as green) or depth map information, the foreground drone image region is accurately extracted through chroma keying or depth testing to obtain its mask.
[0087] Based on this, the foreground image is adjusted according to the parameters of the second virtual camera and the lighting information of the target background image. The adjusted foreground image is then composited into the target background image using a mask. This can specifically include:
[0088] Based on the internal parameters of the virtual camera, a perspective transformation is performed on the drone area in the foreground image to make the perspective effect of the target drone model consistent with that of the target background image.
[0089] Adjust the brightness, contrast, highlights, and shadows of the foreground image based on the lighting direction and color temperature of the background image.
[0090] The adjusted foreground image is composited into the target background image based on the mask, and post-processing effects are added.
[0091] Specifically, during the geometry and lighting fusion process, perspective transformation is performed on the foreground drone based on the virtual camera's focal length and viewing angle to match its perspective relationship with the background image. Next, lighting consistency adjustment is performed: the lighting direction and color temperature of the background image are analyzed, and the brightness, contrast, highlights, and shadows of the foreground drone image are dynamically adjusted to make it appear realistically present in the background environment. Finally, the adjusted foreground image is composited onto the selected background image based on its mask. Subsequently, post-processing effects such as simulated camera noise, mild motion blur, and image compression artifacts can be added to further enhance the realism of the composite image.
[0092] To facilitate the direct use of subsequent images, relevant data on the drone's position in the composite image can be generated based on data obtained from the virtual camera and rendering engine. The coordinate bounding box of the drone in the two-dimensional image plane can be determined based on the position information, the parameters of the second virtual camera, and the parameters of the first virtual camera. The coordinate bounding box is then associated with the category of the target background image and stored.
[0093] Specifically, based on the UAV's 3D attitude, position, and camera parameters, precise 2D pixel coordinate bounding boxes of the UAV on the final synthesized 2D image plane can be obtained through projective geometry calculations. Then, the bounding box coordinates, category labels, and optional difficulty markers, attitude angles, and other information are encapsulated into a standard object detection dataset format, resulting in pixel-level precise standard annotation files that correspond one-to-one with each synthesized training image.
[0094] In practical implementation, when the number of training images generated by the user is large, such as when tens of thousands of training images need to be generated, a task queue can be used to break down the large-scale generation task into parallel small tasks, schedule computing resources (such as GPU clusters), perform image synthesis in parallel, and monitor the execution status and logs of each step, thereby improving the speed of training data generation.
[0095] As can be seen from the above, this training data generation method is based on simulation rendering using optical imaging principles. It not only synthesizes images but also precisely simulates the imaging effects (such as pixel size, blur, and noise) under specific camera parameters, distances, lighting, and atmospheric conditions through a physical rendering engine. This ensures the consistency of the physical distribution of the synthesized data with the real data, guaranteeing the effectiveness of model training. When combining virtual drones with real backgrounds, it's not a simple overlay but rather an intelligent matching and adjustment of lighting, shadows, colors, and perspective relationships through algorithms, generating visually highly realistic and indistinguishable synthetic scenes.
[0096] Through parameterized configuration, it can systematically and purposefully generate data covering extreme poses, complex backgrounds, and interference, thereby improving model robustness and overcoming the randomness and limitations of real-world shooting data. It integrates the scattered modeling, configuration, rendering, compositing, and annotation processes into a coherent automated system, realizing one-click operation from "input device model" to "output standardized dataset," greatly improving overall efficiency and stability.
[0097] The above describes a training data generation method provided by the embodiments of this application. The following describes the apparatus for performing the above training data generation method.
[0098] Please see Figure 3 , Figure 3 This is a schematic diagram of a training data generation device provided in an embodiment of this application. Figure 3 As shown, the training data generation device includes:
[0099] The scene layout module 301 is used to place a target drone model in a three-dimensional virtual space according to the first target state parameters, and to place a virtual camera in a three-dimensional virtual space according to the first virtual camera parameters, so as to obtain a drone layout scene.
[0100] The image generation module 302 is used to invoke the rendering engine in the drone deployment scene, generate a foreground image based on the second target state parameters, second virtual camera parameters, and environmental parameters of the target drone model, obtain a foreground image containing only a solid-color background of the target drone model, and record the position information of the target drone model in the image coordinate system and the world coordinate system; and,
[0101] The image compositing module 303 is used to process the foreground image to obtain a mask of the target drone model, adjust the foreground image according to the parameters of the second virtual camera and the illumination information of the target background image, composite the adjusted foreground image into the target background image according to the mask, and record the category of the target background image.
[0102] In one possible implementation, the scene placement module 301 places a target UAV model in a three-dimensional virtual space according to the first target state parameters, and places a virtual camera in the three-dimensional virtual space according to the first virtual camera parameters, thus obtaining the process of placing the UAV scene, including:
[0103] Based on the target drone model's position distribution, attitude angle information, and relative size, the target drone model is placed in a three-dimensional virtual space; and based on the virtual camera's position and orientation, the virtual camera is set in the three-dimensional virtual space.
[0104] In one possible implementation, the image generation module 302 invokes the rendering engine in the drone deployment scene to generate a foreground image based on the second target state parameters of the target drone model, the second virtual camera parameters, and environmental parameters. The process of obtaining a foreground image containing only a solid-color background of the target drone model includes:
[0105] Based on the material properties of the target drone model, lighting conditions, and the internal parameters of the virtual camera, ray tracing or rasterization rendering is performed to obtain the foreground image.
[0106] In one possible implementation, the image synthesis module 303 processes the foreground image to obtain a mask of the target UAV model, including:
[0107] Extract the drone image region from the foreground image using chroma keying based on the solid-color background of the foreground image to obtain a mask; or,
[0108] Based on the depth map information of the foreground image, the drone image region is extracted through depth testing to obtain a mask.
[0109] In one possible implementation, the image compositing module 303 adjusts the foreground image based on the parameters of the second virtual camera and the illumination information of the target background image, and then composites the adjusted foreground image into the target background image based on a mask, including:
[0110] Based on the internal parameters of the virtual camera, a perspective transformation is performed on the drone area in the foreground image to make the perspective effect of the target drone model consistent with that of the target background image.
[0111] Adjust the brightness, contrast, highlights, and shadows of the foreground image based on the lighting direction and color temperature of the background image;
[0112] The adjusted foreground image is composited into the target background image based on the mask, and post-processing effects are added.
[0113] In one possible implementation, the image synthesis module 303 is also used for:
[0114] Based on the location information, the parameters of the second virtual camera, and the parameters of the first virtual camera, the coordinate bounding box of the UAV in the two-dimensional image plane is determined, and the coordinate bounding box is associated with the category of the target background image and then stored.
[0115] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as laptops, desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0116] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0117] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0118] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the training data generation methods provided in this application.
[0119] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the training data generation methods provided in this application.
[0120] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0122] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0123] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. A method for generating training data, characterized in that, include: The target drone model is placed in the three-dimensional virtual space according to the first target state parameters, and a virtual camera is placed in the three-dimensional virtual space according to the first virtual camera parameters to obtain the drone deployment scene; In the drone deployment scene, the rendering engine is invoked to generate a foreground image based on the second target state parameters, the second virtual camera parameters, and the environmental parameters of the target drone model. This results in a foreground image containing only a solid color background of the target drone model, and the position information of the target drone model in the image coordinate system and the world coordinate system is recorded. The foreground image is processed to obtain a mask of the target drone model. The foreground image is then adjusted according to the parameters of the second virtual camera and the lighting information of the target background image. The adjusted foreground image is then composited into the target background image according to the mask, and the category of the target background image is recorded.
2. The training data generation method according to claim 1, characterized in that, The step of placing a target drone model in a three-dimensional virtual space according to the first target state parameters, and placing a virtual camera in the three-dimensional virtual space according to the first virtual camera parameters to obtain a drone deployment scene includes: Based on the position distribution, attitude angle information, and relative size of the target drone model, the target drone model is placed in the three-dimensional virtual space; and based on the position and orientation of the virtual camera, the virtual camera is set in the three-dimensional virtual space.
3. The training data generation method according to claim 1, characterized in that, The process involves invoking a rendering engine within the drone deployment scene to generate a foreground image based on the second target state parameters, second virtual camera parameters, and environmental parameters of the target drone model. This results in a foreground image containing only a solid-color background of the target drone model, including: Based on the material properties, lighting conditions, and internal parameters of the target drone model, ray tracing or rasterization rendering is performed to obtain the foreground image.
4. The training data generation method according to claim 1, characterized in that, The process of processing the foreground image to obtain a mask for the target drone model includes: The mask is obtained by extracting the drone image region from the foreground image using chroma keying based on the solid color background of the foreground image; or... Based on the depth map information of the foreground image, the image region of the UAV is extracted through depth testing to obtain the mask.
5. The training data generation method according to claim 1, characterized in that, The step of adjusting the foreground image based on the parameters of the second virtual camera and the lighting information of the target background image, and then compositing the adjusted foreground image into the target background image based on the mask, includes: The perspective transformation of the drone region in the foreground image is performed based on the internal parameters of the virtual camera, so that the perspective effect of the target drone model is consistent with that of the target background image; The brightness, contrast, highlights, and shadows of the foreground image are adjusted according to the lighting direction and color temperature of the background image. The adjusted foreground image is composited into the target background image according to the mask, and post-processing effects are added.
6. The training data generation method according to any one of claims 1 to 5, characterized in that, Also includes: Based on the location information, the parameters of the second virtual camera, and the parameters of the first virtual camera, the coordinate bounding box of the UAV in the two-dimensional image plane is determined, and the coordinate bounding box is associated with the category of the target background image and then stored.
7. A training data generation device, characterized in that, include: The scene layout module is used to place a target drone model in a three-dimensional virtual space according to the first target state parameters, and to place a virtual camera in the three-dimensional virtual space according to the first virtual camera parameters, so as to obtain a drone layout scene; The image generation module is used to call the rendering engine in the drone deployment scene, generate a foreground image based on the second target state parameters, the second virtual camera parameters and the environmental parameters of the target drone model, obtain a foreground image containing only a solid color background of the target drone model, and record the position information of the target drone model in the image coordinate system and the world coordinate system. as well as, An image compositing module is used to process the foreground image to obtain a mask of the target drone model, adjust the foreground image according to the parameters of the second virtual camera and the lighting information of the target background image, composite the adjusted foreground image into the target background image according to the mask, and record the category of the target background image.
8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the training data generation method as described in any one of claims 1 to 6.
9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the training data generation method as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the training data generation method as described in any one of claims 1 to 6.