Target detection data automatic synthesis method and system for unmanned aerial vehicle

Abstract

The invention provides an unmanned aerial vehicle-oriented target detection data automatic synthesis method and system, and the method comprises the steps: collecting multi-view two-dimensional image data of an unmanned aerial vehicle, calculating the pose information of a camera, and obtaining an image set with the pose information; performing volume rendering on the image set by using the customized neural radiation field model to obtain a multi-dimensional rendering graph of the unmanned aerial vehicle; extracting depth image data from the multi-dimensional rendering graph, and performing geometric fusion and weather effect simulation on the unmanned aerial vehicle and an environment background by combining an environment background image and utilizing a physical atmospheric scattering model to obtain synthetic image data; based on the synthesized image data, the multi-dimensional rendering image and the camera pose information, generating annotation information of the unmanned aerial vehicle, organizing the synthesized image data and the annotation information according to a preset format compatible with a target detection model, and generating a target detection synthesized data set; according to the method, unmanned aerial vehicle target detection data with geometric accuracy, scene diversity and labeling accuracy can be produced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to an automatic method and system for synthesizing target detection data for unmanned aerial vehicles (UAVs). Background Technology

[0002] The training and optimization of high-performance object detection algorithms heavily rely on large datasets with precise annotations (such as bounding boxes, poses, and categories) and diverse scenarios. However, for special targets like drones, data acquisition and annotation in real-world scenarios face multiple severe challenges: on the one hand, systematically acquiring complete data on drones under different weather conditions, lighting, backgrounds, and flight attitudes requires extremely high time and economic costs; on the other hand, drones are typically small in size and move rapidly, making manual two-dimensional or three-dimensional precise annotation not only extremely time-consuming, but also difficult to guarantee geometric consistency and accuracy of the annotation results when faced with occlusion, sparse shooting perspectives, and other situations.

[0003] Against this backdrop, generating synthetic datasets using computer graphics technology has become a core approach to solving the challenges of scarce and unmanned aerial vehicle (UAV) data annotation. However, traditional synthetic methods have revealed inherent and unavoidable flaws when dealing with high-difficulty targets like UAVs: traditional 3D reconstruction techniques, such as explicit modeling methods based on multi-view solid geometry (MVS), mesh, or voxel, generally lack the ability to accurately capture the core features of UAVs. For example, for typical thin structures like wings and propellers, MVS and voxel methods are limited by resolution and accuracy, failing to reproduce their slender and intricate geometric details, easily leading to incomplete reconstructed model structures or artifacts. Furthermore, for the non-Lambertian characteristics such as specular reflection and view-dependent appearance exhibited by the smooth reflective materials commonly found on UAV shells, traditional methods based on the Lambertian assumption and texture mapping strategies struggle to reproduce realistic optical phenomena, resulting in renderings lacking in lighting details and realism.

[0004] The emergence of Neural Radiation Field (NeRF) technology has brought breakthrough progress to scene 3D reconstruction and new perspective synthesis. By encoding the scene as a continuously differentiable volume density and color function, combined with volume rendering technology, it performs well in static scene new perspective synthesis. However, the original NeRF implementation has two major limitations, making it difficult to adapt to UAV data acquisition scenarios: First, scene independence, each new scene needs to be independently trained from random weights to a multilayer perceptron (MLP), which takes tens of hours; second, data dependence, requiring dozens to hundreds of images with precise attitude calibration as supervision signals, which contradicts the need for fast and flexible acquisition of UAV data, and cannot be used as an efficient and generalized reconstruction tool.

[0005] To address the scene independence issue of NeRF, Pixel NeRF was proposed. By introducing a fully convolutional image encoder E, it can extract pixel-aligned features from the input image and input them into the NeRF MLP, enabling the model to learn geometric and appearance priors across scenes, achieving scene NeRF prediction for single or a small number of sparse images. However, for the specific target of drones, standard Pixel NeRF still has key shortcomings and fails to solve the core pain points of drone data synthesis: on the one hand, its encoder E is trained on general datasets such as ShapeNet and lacks the ability to specifically perceive the thin structure and high reflectivity of drones, resulting in insufficient fidelity and geometric accuracy in the reconstruction of new perspectives of drones under sparse views; on the other hand, existing methods focus on geometric and appearance reconstruction under ideal lighting conditions, and the synthesized data lacks environmental perturbations in complex weather scenes such as fog, haze, rain, and snow, causing the detection model trained with it to degrade sharply in real complex environments.

[0006] In summary, the field of UAV target detection data acquisition and synthesis still faces three major core technical problems that urgently need to be solved: First, in real-world acquisition scenarios, the UAV's field of view is not fully covered, making it difficult to acquire diverse perspective data; second, existing synthesis methods lack sufficient geometric reconstruction accuracy for thin structures such as UAV wings and propellers, and cannot accurately reproduce their reflective properties; third, the synthesized data lacks diversity in complex weather scenarios such as rain, snow, fog, and haze, resulting in poor environmental adaptability of detection models trained based on this data. These are precisely the technical bottlenecks that this invention aims to overcome. Summary of the Invention

[0007] To address the problems in existing UAV target detection data acquisition, such as incomplete field of view coverage, difficulty in accurately reproducing the thin structure (e.g., wings, propellers) and reflective features of UAVs, and insufficient diversity of weather scenarios, this invention proposes an automatic target detection data synthesis method for UAVs, comprising:

[0008] Collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset flight path information to obtain an image set with pose information;

[0009] A pre-built, customized neural radiation field model is used to perform volume rendering on the image set containing attitude information to obtain a multi-dimensional rendering image of the UAV.

[0010] Depth image data of the UAV is extracted from the multi-dimensional rendering image. Combined with the pre-acquired environmental background image with depth information, the UAV and the environmental background are geometrically fused and the weather effect is simulated using a physical atmospheric scattering model to obtain synthetic image data.

[0011] Based on the synthetic image data, the multi-dimensional rendering map, and the camera pose information, the annotation information of the UAV is generated, and the synthetic image data and the annotation information are organized in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV.

[0012] The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

[0013] Optionally, the step of calculating the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset track information to obtain an image set with pose information includes:

[0014] Based on the two-dimensional image data, determine the acquisition method of the two-dimensional image data;

[0015] When the two-dimensional image data is acquired by a non-track-associated camera, the camera pose information of the camera is inversely calculated using a photogrammetry tool, and the camera pose information is associated with the corresponding two-dimensional image data to form an image set with pose information.

[0016] When the two-dimensional image data is collected by the UAV according to the preset flight path information, the camera pose information of the UAV is determined by the GPS positioning module and inertial measurement unit on the UAV, and the camera pose information is bound to the corresponding two-dimensional image data to form an image set with pose information.

[0017] The non-track-associated camera includes one or more of the following: handheld camera and fixed camera.

[0018] Optionally, the camera pose information includes: camera intrinsic parameter information and camera extrinsic parameter information;

[0019] The camera intrinsic parameters include one or more of the following: focal length information, image sensor size, and distortion coefficient information;

[0020] The camera extrinsic information includes one or more of the following: spatial position information, shooting angle information, rotation matrix and translation vector information;

[0021] The multi-dimensional rendering map includes one or more of the following: drone foreground color map, depth image data, and drone target mask;

[0022] The annotation information includes one or more of the following: two-dimensional bounding boxes, pixel-level semantic segmentation masks, and category labels.

[0023] Optionally, the step of using a pre-built customized neural radiation field model to perform volume rendering on the image set with pose information to obtain a multi-dimensional rendering image of the UAV includes:

[0024] Based on the image set with pose information, the core input elements of the model are extracted, and a new viewpoint is generated based on the core input elements of the model.

[0025] The target generated from the new perspective is input as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0026] Optionally, the step of extracting core input elements of the model based on the image set with pose information and determining the new viewpoint generation target based on the core input elements of the model includes:

[0027] Extract the core input elements of the model and the existing viewpoint distribution information from the image set containing pose information;

[0028] Based on the core input elements of the model and the existing view distribution information, determine the target for generating new viewpoints;

[0029] The core input elements of the model include one or more of the following: two-dimensional image data of the UAV, camera pose information, and geometric prior information of the UAV;

[0030] The geometric prior information includes one or more of the following: three-dimensional bounding box parameter information, thin structure thickness information, and geometric feature location information;

[0031] The existing view distribution information includes one or more of the following: existing view spatial coverage, existing view density distribution, and existing view correlation information;

[0032] The new perspective generation target includes one or more of the following: new perspective pose parameters, new perspective rendering output specifications, new perspective spatial distribution strategy, and new perspective image resolution;

[0033] The new perspective rendering output specifications include one or more of the following: new perspective image resolution, pixel format, and rendering quality requirements;

[0034] The new perspective spatial distribution strategy includes one or more of the following: the new perspective covers the target in space, increases the number of perspectives and new perspectives in low-density areas.

[0035] Optionally, the step of inputting the target generated from the new perspective as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV includes:

[0036] The target generated from the new perspective is used as input data, and the input data is standardized by a pre-built customized neural radiation field model to form standardized input data.

[0037] Based on the standardized input data, thin structure and reflective region features in the two-dimensional image data of the UAV are extracted through multi-scale dilated convolution in the customized neural radiation field model;

[0038] Based on the thin structure and the reflective area features, combined with the spatial coding features of the new perspective frustum, a joint feature is formed; wherein, the new perspective frustum is determined by the camera intrinsic parameter information and camera pose information of the target generated from the new perspective.

[0039] Based on the joint features, the spatial point core attribute information of the UAV is output through the conditional multilayer perceptron in the customized neural radiation field model;

[0040] Based on the core spatial attribute information of the UAV, volume rendering is performed on the pixel grid of the new perspective image to obtain a multi-dimensional rendering image of the UAV.

[0041] The pixel grid of the new perspective image is determined by the resolution of the new perspective image in the new perspective generation target.

[0042] Optionally, the step of extracting the depth image data of the UAV from the multi-dimensional rendered image, combining it with a pre-acquired environmental background image containing depth information, and using a physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data includes:

[0043] The depth image data and foreground image of the UAV are extracted from the multi-dimensional rendering image. Combined with the pre-acquired environmental background image with depth information, the UAV and the environmental background of the UAV are geometrically fused to obtain fused scene data.

[0044] Based on the fused scene data, a physical atmospheric scattering model is used to simulate the weather effects of the UAV and the environmental background in the fused scene data, resulting in a scene image with weather features.

[0045] Based on the scene images with weather features, consistency optimization and quality verification are performed to obtain synthetic image data;

[0046] The fused scene data includes: pixel-aligned original scene radiance information and depth values ​​corresponding to each pixel.

[0047] Optionally, the calculation formula for the physical atmospheric scattering model is as follows:

[0048] ;

[0049] In the formula,

[0050] ;

[0051] in, Indicates pixel position The pixel radiance of the scene image with weather characteristics output after weather effect simulation; Indicates pixel position Atmospheric transmittance at that location; Indicates pixel position The original radiance information of the scene itself; Indicates the brightness of atmospheric ambient light radiation; It is a natural constant; Represents the Rayleigh scattering coefficient; Represents the Mie scattering coefficient; Indicates pixel position The depth value at that location.

[0052] Based on the same inventive concept, this invention also provides an automatic target detection data synthesis system for unmanned aerial vehicles (UAVs), comprising:

[0053] The pose calculation module is used to collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data through photogrammetry tools or preset trajectory information to obtain an image set with pose information.

[0054] The image rendering module is used to perform volume rendering on the image set with attitude information using a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0055] The weather simulation module is used to extract the depth image data of the UAV from the multi-dimensional rendering image, combine it with the pre-acquired environmental background image with depth information, and use the physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data.

[0056] The data synthesis module is used to generate annotation information of the UAV based on the synthesized image data, the multi-dimensional rendering map and the camera pose information, and to organize the synthesized image data and the annotation information in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV.

[0057] The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

[0058] Optionally, the pose calculation module includes:

[0059] The data acquisition submodule is used to determine the acquisition method of the two-dimensional image data based on the two-dimensional image data.

[0060] The pose inversion submodule is used to inversely calculate the camera pose information of the camera by photogrammetry tools when the two-dimensional image data is acquired by a non-track-associated camera, and associate the camera pose information with the corresponding two-dimensional image data to form an image set with pose information.

[0061] The inertial measurement submodule is used to determine the camera pose information of the UAV by using the GPS positioning module and inertial measurement unit on the UAV when the two-dimensional image data is collected by the UAV according to the preset trajectory information, and bind the camera pose information with the corresponding two-dimensional image data to form an image set with attitude information.

[0062] The non-track-associated camera includes one or more of the following: handheld camera and fixed camera.

[0063] Optionally, the camera pose information includes: camera intrinsic parameter information and camera extrinsic parameter information;

[0064] The camera intrinsic parameters include one or more of the following: focal length information, image sensor size, and distortion coefficient information;

[0065] The camera extrinsic information includes one or more of the following: spatial position information, shooting angle information, rotation matrix and translation vector information;

[0066] The multi-dimensional rendering map includes one or more of the following: drone foreground color map, depth image data, and drone target mask;

[0067] The annotation information includes one or more of the following: two-dimensional bounding boxes, pixel-level semantic segmentation masks, and category labels.

[0068] Optionally, the image rendering module includes:

[0069] The feature extraction submodule is used to extract the core input features of the model based on the image set with pose information and determine the target for generating a new viewpoint based on the core input features of the model.

[0070] The multi-dimensional rendering submodule is used to input the target generated from the new perspective as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0071] Optionally, the feature extraction submodule includes:

[0072] The information extraction unit is used to extract the core input elements of the model and the existing viewpoint distribution information from the image set with pose information;

[0073] The new perspective generation unit is used to determine the new perspective generation target based on the core input elements of the model and the existing perspective distribution information;

[0074] The core input elements of the model include one or more of the following: two-dimensional image data of the UAV, camera pose information, and geometric prior information of the UAV;

[0075] The geometric prior information includes one or more of the following: three-dimensional bounding box parameter information, thin structure thickness information, and geometric feature location information;

[0076] The existing view distribution information includes one or more of the following: existing view spatial coverage, existing view density distribution, and existing view correlation information;

[0077] The new perspective generation target includes one or more of the following: new perspective pose parameters, new perspective rendering output specifications, new perspective spatial distribution strategy, and new perspective image resolution;

[0078] The new perspective rendering output specifications include one or more of the following: new perspective image resolution, pixel format, and rendering quality requirements;

[0079] The new perspective spatial distribution strategy includes one or more of the following: the new perspective covers the target in space, increases the number of perspectives and new perspectives in low-density areas.

[0080] Optionally, the multi-dimensional rendering submodule includes:

[0081] The standardization processing unit is used to take the target generated from the new perspective as input data, and to standardize the input data through a pre-built customized neural radiation field model to form standardized input data.

[0082] The feature extraction unit is used to extract thin structure and reflective region features from the two-dimensional image data of the UAV based on the standardized input data and through multi-scale dilated convolution in the customized neural radiation field model.

[0083] The feature combination unit is used to form a joint feature based on the thin structure and the reflective area features, combined with the spatial coding features of the new view frustum; wherein the new view frustum is determined by the camera intrinsic parameter information and camera pose information of the target generated by the new view.

[0084] The attribute generation unit is used to output the core spatial attribute information of the UAV based on the joint features and through the conditional multilayer perceptron in the customized neural radiation field model.

[0085] The spatial rendering unit is used to perform volume rendering on the pixel grid of the new perspective image based on the core attribute information of the UAV's spatial points, so as to obtain a multi-dimensional rendering image of the UAV.

[0086] The pixel grid of the new perspective image is determined by the resolution of the new perspective image in the new perspective generation target.

[0087] Optionally, the weather simulation module includes:

[0088] The geometric fusion submodule is used to extract the depth image data and foreground image of the UAV from the multi-dimensional rendering image, and combine them with the pre-acquired environmental background image with depth information to perform geometric fusion of the UAV and the environmental background of the UAV to obtain fused scene data.

[0089] The effect simulation submodule is used to simulate the weather effects of the UAV and the environmental background in the fused scene data using a physical atmospheric scattering model, based on the fused scene data, to obtain a scene image with weather features.

[0090] The image synthesis submodule is used to perform consistency optimization and quality verification on the scene image with weather features to obtain synthesized image data;

[0091] The fused scene data includes: pixel-aligned original scene radiance information and depth values ​​corresponding to each pixel.

[0092] Optionally, the calculation formula for the physical atmospheric scattering model is as follows:

[0093] ;

[0094] In the formula,

[0095] ;

[0096] in, Indicates pixel position The pixel radiance of the scene image with weather characteristics output after weather effect simulation; Indicates pixel position Atmospheric transmittance at that location; Indicates pixel position The original radiance information of the scene itself; Indicates the brightness of atmospheric ambient light radiation; It is a natural constant; Represents the Rayleigh scattering coefficient; Represents the Mie scattering coefficient; Indicates pixel position The depth value at that location.

[0097] In another aspect, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0098] The memory is used to store one or more programs;

[0099] When the one or more programs are executed by the at least one processor, an automatic synthesis method for target detection data for unmanned aerial vehicles (UAVs) is implemented as described above.

[0100] In another aspect, the present invention also provides a computer device readable storage medium having an executable program stored thereon, wherein when the executable program is executed, it implements the aforementioned method for automatic synthesis of target detection data for unmanned aerial vehicles.

[0101] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0102] This invention provides an automatic target detection data synthesis method and system for unmanned aerial vehicles (UAVs), comprising: acquiring multi-view two-dimensional image data of the UAV; calculating camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset flight path information to obtain an image set with pose information; performing volume rendering on the image set with pose information using a pre-constructed customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV; extracting depth image data of the UAV from the multi-dimensional rendering image; combining it with a pre-acquired environmental background image with depth information; and using a physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthesized image data; and generating annotation information of the UAV based on the synthesized image data, the multi-dimensional rendering image, and the camera pose information. The synthesized image data and the annotation information are organized in a preset format compatible with the target detection model to generate a synthetic target detection dataset for the UAV. The customized neural radiation field model includes at least an encoder optimized for the thin structure and reflective features of the UAV, and a conditional multilayer perceptron. This invention, through a complete technical solution that automatically completes multi-view data acquisition and pose calculation for the UAV, accurately reconstructs the thin structure and reflective features of the UAV using customized modeling, integrates environmental background and simulates diverse weather effects, and generates standardized annotations and datasets, can efficiently produce UAV target detection data with geometric accuracy, scene diversity, and precise annotation. This effectively solves the problems of high cost of real data acquisition, high annotation difficulty, and insufficient quality of synthetic data, providing high-quality data support for target detection model training to improve model detection performance. Attached Figure Description

[0103] Figure 1 This is a flowchart illustrating an automatic target detection data synthesis method for unmanned aerial vehicles (UAVs) provided by the present invention.

[0104] Figure 2 The NeRF MLP structure diagram of the customized neural radiation field model in the automatic synthesis method for target detection data of UAVs provided by this invention;

[0105] Figure 3 This is a schematic diagram of the overall framework of an automatic target detection data synthesis method for UAVs provided by the present invention;

[0106] Figure 4 This invention provides a schematic diagram of the structural composition of an automatic target detection data synthesis system for unmanned aerial vehicles (UAVs).

[0107] Figure 5 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0108] This invention proposes an automatic method, system, device, and medium for synthesizing target detection data for unmanned aerial vehicles (UAVs). The specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings.

[0109] Example 1:

[0110] This invention provides an automatic method for synthesizing target detection data for unmanned aerial vehicles (UAVs), the flowchart of which is shown below. Figure 1 As shown, it includes:

[0111] Step 1: Collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset flight path information to obtain an image set with pose information;

[0112] Step 2: Use a pre-built customized neural radiation field model to perform volume rendering on the image set with attitude information to obtain a multi-dimensional rendering image of the UAV;

[0113] Step 3: Extract the depth image data of the UAV from the multi-dimensional rendering image, combine it with the pre-acquired environmental background image with depth information, and use the physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data;

[0114] Step 4: Based on the synthesized image data, the multi-dimensional rendering map, and the camera pose information, generate the annotation information of the UAV, and organize the synthesized image data and the annotation information in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV;

[0115] The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

[0116] Generally, existing technologies for detecting drones suffer from incomplete viewpoint coverage and difficulty in acquiring diverse viewpoint data in real-world scenarios. Traditional synthesis methods cannot accurately reproduce the geometric details of thin structures such as drone wings and propellers, as well as the reflective properties of the outer shell. Furthermore, synthesized data lacks diversity in complex weather scenarios such as rain, snow, and fog. Manual annotation is not only time-consuming and labor-intensive but also struggles to guarantee geometric consistency and accuracy. These problems all contribute to poor training data quality for target detection models, thus affecting detection performance. To address these technical issues, this invention considers first accurately calculating the camera pose information corresponding to the two-dimensional image using photogrammetry tools or preset flight path information. This provides a precise viewpoint and spatial reference for the subsequent volume rendering of the customized neural radiation field model, ensuring the accuracy of multi-dimensional rendering and automatic annotation of the drone. Specifically:

[0117] In one implementation, the process of calculating the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset track information in step 1 above to obtain an image set with pose information may include:

[0118] Based on the two-dimensional image data, determine the acquisition method of the two-dimensional image data;

[0119] When the two-dimensional image data is acquired by a non-track-associated camera, the camera pose information of the camera is inversely calculated using a photogrammetry tool, and the camera pose information is associated with the corresponding two-dimensional image data to form an image set with pose information.

[0120] When the two-dimensional image data is collected by the UAV according to the preset flight path information, the camera pose information of the UAV is determined by the GPS positioning module and inertial measurement unit on the UAV, and the camera pose information is bound to the corresponding two-dimensional image data to form an image set with pose information.

[0121] The non-track-associated camera may include one or more of the following: handheld camera and fixed camera.

[0122] For example, the camera pose information may include: camera intrinsic parameters and camera extrinsic parameters;

[0123] The camera intrinsic parameters may include one or more of the following: focal length information, image sensor size, and distortion coefficient information;

[0124] The camera extrinsic information may include one or more of the following: spatial position information, shooting angle information, rotation matrix and translation vector information;

[0125] For example, the camera pose information can be shown in Table 1:

[0126] Table 1. Examples of Camera Pose Information

[0127]

[0128] For example, the multi-dimensional rendering map may include one or more of the following: drone foreground color map, depth image data, and drone target mask;

[0129] The annotation information may include one or more of the following: two-dimensional bounding boxes, pixel-level semantic segmentation masks, and category labels;

[0130] In this implementation, photogrammetry tools are used to accurately calculate the pose in reverse, and sensor fusion is used to ensure the real-time pose for the trajectory acquisition scenario. Both paths can achieve depth correlation between pose and image, which helps to avoid pose and image matching deviations caused by unified processing. Furthermore, by fully covering the camera's intrinsic parameters (such as focal length, distortion coefficients, etc.) and extrinsic parameters (such as rotation matrix, spatial position, etc.), the association logic between them and multi-dimensional rendering maps (such as depth maps, target masks) and annotation information (such as 2D bounding boxes, semantic masks) is clarified, which can provide a fully parametric spatial reference and avoid rendering geometric deviations caused by missing intrinsic parameters (such as ignoring distortion coefficients) or the impact of incomplete extrinsic parameters (such as missing translation vectors) on the accuracy of annotation coordinates.

[0131] Generally, NeRF encodes a scene as a function f, whose input is the three-dimensional spatial position. and observation direction The output is the color of that point. and volume density The specific relationships are as follows:

[0132] ;

[0133] To capture high-frequency geometric details, all input coordinates and direction All are encoded using high-dimensional positional encoding. Mapping, volume density Determines the light in The probability of termination at a certain point, and the color This depends on the direction of observation.

[0134] Final pixel color By along the light The expression obtained by volume rendering integral approximation is as follows:

[0135] ;

[0136] Among them, transmittance Indicates light from the near boundary Time The probability of traversing an unobstructed path is determined by... Integral calculation.

[0137] Standard Pixel NeRF achieves cross-scene generality by introducing a fully convolutional encoder (e.g., represented by E), which... Extracted pixel alignment features As a NeRF MLP Conditional input.

[0138] For any query position in space First, it is projected through the camera projection function. Mapping onto the input image plane to obtain features Then, in NeRF MLP The prediction is conditionalized by the feature using the following expression:

[0139] ;

[0140] By training jointly on multiple scenarios, the model can learn scenario priors, enabling fast forward inference from sparse views (such as a single image) to high-fidelity 3D representations.

[0141] Furthermore, due to the highly specific geometry and appearance of drone targets, standard general-purpose encoders struggle to efficiently extract features of sufficiently high quality. To guide accurate NeRF reconstruction, and because reconstructing thin-walled structures requires extremely high local feature resolution, while capturing reflective properties requires strong cross-view feature integration capabilities, this invention addresses these issues by introducing a customized neural radiation field model for volume rendering and structure recognition of images. Specifically:

[0142] In one implementation, step 2 above, which involves using a pre-built, customized neural radiation field model to perform volume rendering on the image set containing pose information to obtain a multi-dimensional rendering image of the UAV, may include:

[0143] Based on the image set with pose information, the core input elements of the model are extracted, and a new viewpoint is generated based on the core input elements of the model.

[0144] The target generated from the new perspective is input as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0145] In this implementation, the process of extracting core input elements of the model based on the image set with pose information and determining a new viewpoint to generate a target based on the core input elements of the model may include:

[0146] Extract the core input elements of the model and the existing viewpoint distribution information from the image set containing pose information;

[0147] Based on the core input elements of the model and the existing view distribution information, determine the target for generating new viewpoints;

[0148] The core input elements of the model may include one or more of the following: two-dimensional image data of the UAV, camera pose information, and geometric prior information of the UAV;

[0149] The geometric prior information may include one or more of the following: three-dimensional bounding box parameter information, thin structure thickness information, and geometric feature location information;

[0150] The existing view distribution information may include one or more of the following: existing view spatial coverage, existing view density distribution, and existing view correlation information;

[0151] The new perspective generation target may include one or more of the following: new perspective pose parameters, new perspective rendering output specifications, new perspective spatial distribution strategy, and new perspective image resolution;

[0152] The new perspective rendering output specifications may include one or more of the following: new perspective image resolution, pixel format, and rendering quality requirements;

[0153] The new perspective spatial distribution strategy may include one or more of the following: the new perspective covers the target in space, increases the number of perspectives and new perspectives in low-density areas.

[0154] In this implementation, the process of inputting the target generated from the new perspective as input data into a pre-constructed customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV may include:

[0155] The target generated from the new perspective is used as input data, and the input data is standardized by a pre-built customized neural radiation field model to form standardized input data.

[0156] Based on the standardized input data, thin structure and reflective region features in the two-dimensional image data of the UAV are extracted through multi-scale dilated convolution in the customized neural radiation field model;

[0157] Based on the thin structure and the reflective area features, combined with the spatial coding features of the new perspective frustum, a joint feature is formed; wherein, the new perspective frustum is determined by the camera intrinsic parameter information and camera pose information of the target generated from the new perspective.

[0158] Based on the joint features, the spatial point core attribute information of the UAV is output through the conditional multilayer perceptron in the customized neural radiation field model;

[0159] Based on the core spatial attribute information of the UAV, volume rendering is performed on the pixel grid of the new perspective image to obtain a multi-dimensional rendering image of the UAV.

[0160] The pixel grid of the new perspective image is determined by the resolution of the new perspective image in the new perspective generation target.

[0161] The customized neural radiation field model of this invention (also referred to as a customized encoder) It is built upon deep residual networks (such as variants of U-ResNet34) and incorporates several optimization modules within its fully convolutional architecture:

[0162] 1. Multi-scale Feature Extraction Module (MSFE):

[0163] To simultaneously capture the large-scale shape of the drone fuselage and the thin-walled details of the wings and propellers, this module employs dilated convolutions with different dilation rates. Dilated convolutions effectively expand the receptive field without increasing the number of parameters or losing spatial information. This fusion of multi-scale features helps ensure the encoder... It can extract geometrically more robust features, thereby significantly improving NeRF's ability to extract volume density data from slender UAV structures. The continuity and accuracy of the representation.

[0164] 2. Local Attention Awareness Mechanism (LAAM):

[0165] The reflective properties of drone shells result in low-quality feature information in certain areas of the image (highlight areas, weakly textured areas), which can easily lead to feature matching errors. The LAAM module of this invention embeds channel and spatial attention mechanisms at different scales of feature extraction. By adaptively learning the regions and channels in the feature map that contribute most to volume density and color prediction, this mechanism enhances the encoder's representation of key features, especially for view-dependent reflective surfaces and texture-poor areas, thus ensuring the efficiency and accuracy of feature extraction.

[0166] 3. Cross-view feature fusion mechanism (CVFF / Ray Transformer):

[0167] In the multi-view generalization scenario of Pixel NeRF, aggregating features from different input views is crucial. Traditional feature aggregation methods (such as simple average pooling) perform poorly when handling view inconsistencies or occlusion. This invention introduces a lightweight attention structure, modeled after Ray Transformer, as the CVFF module in the intermediate layer of the NeRF MLP. For query points on the light rays, the CVFF module uses a self-attention mechanism to dynamically calculate and assign contribution weights from features from different input views. This mechanism enables the model to prioritize the input features that are most consistent with the query view and have the most reliable information, thereby effectively solving view inconsistencies and greatly improving the reconstruction quality under sparse views.

[0168] pass Extracted pixel alignment features It is used as a conditional input (i.e., the input to the model) and injected into the NeRF MLP through residual connections or affine transformations similar to AdalN. In each or a specific hidden layer, this deep, multi-layered feature integration helps ensure that the density and color predictions of NeRF are strongly conditional throughout the training process, thereby achieving a high-fidelity representation of the drone model;

[0169] Figure 2 A detailed demonstration of the custom encoder The structure (where Encoder represents the encoder) involves downsampling the input image I through multiple convolutional blocks, embedding MSFE and LAAM modules in the intermediate layers to enhance feature extraction capabilities, and extracting feature maps. Query point in 3D space Projected onto a 2D image plane Then, the feature vector is obtained through bilinear interpolation, and this feature vector is compared with... Position encoding Combined, as input to f in NeRF MLP, in In the multi-layered structure, the CVFF / Ray Transformer architecture is used to aggregate multi-view features, ultimately outputting volume density. and color .

[0170] Model Training: Custom Encoder In NeRF MLP Perform cross-scenario end-to-end joint training on a large-scale dataset containing multiple types and multi-view drone targets;

[0171] Loss function: The main optimization objective is to minimize the rendered color. With target color Between To improve the accuracy of thin structures, an additional depth consistency loss can be introduced, utilizing multi-view geometric constraints on volume density. Regularization is crucial, especially for small structures that are difficult to reconstruct.

[0172] Optimizer: The Adam optimizer is used for training.

[0173] Generalization capability: By learning from large-scale UAV datasets, it can obtain powerful scene priors, thereby enabling fast, low-sample, high-fidelity reconstruction of new UAV models that have not appeared in the training set.

[0174] Physically Consistent Environment Simulation: The robustness enhancement synthesis module utilizes precise depth information provided by NeRF, combined with an atmospheric scattering model for physically-based rendering, ensuring that the weather effects between the foreground drone and the background are depth-dependent and physically consistent. This results in synthesized images exhibiting extremely high realism under extreme weather conditions.

[0175] Target detection models (such as YOLOv5 or YOLOv8) trained using datasets covering diverse weather conditions generated by the method of this invention exhibit significantly better performance than models trained using synthetic data from traditional ideal environments when tested in real-world complex environments (such as fog and haze). This addresses the problems of false detections and missed detections caused by concentrated image grayscale distribution and color distortion. Table 2 shows a performance comparison of this invention with existing technologies:

[0176] Table 2 Performance Comparison of the Customized Neural Radiation Field Model of the Present Invention with Existing Technologies

[0177] Performance indicators Traditional MVS / SfM Basic NeRF Standard Pixel NeRF This invention (customized UAV NeRF) Number of input views required More (>10) More (>50) Fewer (1-5) Fewer (1-5) Reconstruction speed quick Extremely slow (requires independent optimization) Quick (forward reasoning) Quick (forward reasoning) Thin structure detail fidelity Difference better good excellent Robustness of synthetic data environments Unable to generate Unable to generate Unable to generate Extremely strong (physically based rendering integration) Improved mAP in complex weather conditions (compared to baseline) - - - +4%～+6%

[0178] As shown in Table 2, in the detection task under simulated fog / snow scenes, the model trained with the dataset of this invention can improve the average accuracy (mAP) by 4% to 6% compared with the benchmark model using ideal lighting synthetic data. In the detection task targeting rare viewpoints in training, the improvement can even exceed 10%.

[0179] Furthermore, the method in this invention can also be based on explicit geometry reconstruction during 3D reconstruction. It can employ Multi-View Solid Geometry (MVS) technology combined with mesh or voxel representation to reconstruct the UAV model. It can also use other implicit neural representation methods, such as the Neural Symbolic Distance Function (SDF) to replace the volume density representation of NeRF, as SDF offers higher accuracy in geometric reconstruction. Moreover, the encoder in the customized neural radiation field model of this invention can use standard fully-convolutional networks, i.e., standard U-ResNet 14 or HRNet as the main body of the image encoder E, without including the MSFE or LAAM modules of this invention; or the encoder can use a pure Transformer encoder, i.e., a Vision Transformer or Swing Transformer structure as the encoder. The feature extractor is powerful in capturing global correlations, but it usually requires high computational overhead to maintain pixel-level alignment and local details. The CNN-Attention hybrid architecture adopted in this invention achieves a better balance between computational efficiency and feature accuracy.

[0180] In one implementation, step 3 above, which involves extracting the depth image data of the UAV from the multi-dimensional rendered image, combining it with a pre-acquired environmental background image containing depth information, and using a physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data, may include:

[0181] The depth image data and foreground image of the UAV are extracted from the multi-dimensional rendering image. Combined with the pre-acquired environmental background image with depth information, the UAV and the environmental background of the UAV are geometrically fused to obtain fused scene data.

[0182] Based on the fused scene data, a physical atmospheric scattering model is used to simulate the weather effects of the UAV and the environmental background in the fused scene data, resulting in a scene image with weather features.

[0183] Based on the scene images with weather features, consistency optimization and quality verification are performed to obtain synthetic image data;

[0184] The fused scene data includes: pixel-aligned original scene radiance information and depth values ​​corresponding to each pixel.

[0185] To overcome the lack of environmental robustness in training target detection models using existing synthetic datasets, this invention integrates a robust enhancement synthesis module. This module utilizes precise geometric information provided by NeRF to simulate realistic and complex weather effects through a physical model. Firstly, the synthesis process obtains the initial color of the drone target through NeRF rendering. Precise depth map and target mask Then, a variety of environmental background images were selected. and its corresponding depth map The key step is to utilize and Perform geometric blending of the foreground and background to generate a unified scene depth map. and the initial synthesized image .

[0186] For example, the physical atmospheric scattering model (e.g., a distance-based transmittance model) can be calculated as follows:

[0187] ;

[0188] In the formula,

[0189] ;

[0190] in, Indicates pixel position The pixel radiance of the scene image with weather characteristics output after weather effect simulation; Indicates pixel position Atmospheric transmittance at a point represents the light transmission from the scene point. The degree of attenuation during propagation to the camera; Indicates pixel position The original radiance information of the scene itself; The brightness of atmospheric ambient light radiation determines the color tendency and saturation of smog; It is a natural constant; Represents the Rayleigh scattering coefficient; The two scattering coefficients represent the Mie scattering coefficients, which are used to parameterize and control the concentration of smog. Indicates pixel position Depth value at (scene location) The precise distance to the camera comes directly from the fused depth map. In this example, the transmittance is analyzed using a distance-based transmittance model. Physical-level environmental disturbance simulations are performed, including the final color of pixels for weather effects such as fog and haze; atmospheric effects are distance-dependent. Distant objects (high...) Transmittance At lower levels, the color will be closer to atmospheric light A, manifesting as increased grayscale and blurriness. Nearby objects are less affected because this method utilizes precise 3D geometric information provided by NeRF. This ensures that the weather effects of the foreground drone and the background at different depths are physically consistent and believable, rather than simply a superposition of two-dimensional image filters. and The parameter values ​​can automatically generate large-scale training data covering a wide range of environmental conditions (from light fog to heavy haze and low light).

[0191] During the synthesis process, the 3D position and pose of the UAV target relative to the camera are precisely known. Therefore, the 2D bounding boxes, 3D bounding boxes, and pixel-level semantic segmentation masks (based on...) required for target detection are... All of these can be automatically and accurately exported from geometric information without any manual annotation.

[0192] In one implementation, step 4 above, which generates annotation information for the UAV based on the synthesized image data, the multi-dimensional rendered image, and the camera pose information, and organizes the synthesized image data and the annotation information in a preset format compatible with the target detection model to generate the target detection synthetic dataset for the UAV, may include:

[0193] (1) Extracting and labeling to generate core basic data

[0194] The drone target mask (pixel-level binary image, white for drone area and black for background), drone depth map, and new perspective camera pose parameters (including intrinsic and extrinsic homogeneous transformation matrices in a unified coordinate system) are separated from the multi-dimensional rendering image. The pixel array (RGB format) and camera pose information (camera intrinsic and extrinsic parameters corresponding to the original image set) of the synthetic image data are retrieved simultaneously to establish a pixel-level correlation mapping between the synthetic image, multi-dimensional rendering data, and camera pose.

[0195] (2) Generate annotation information for multiple types of UAVs

[0196] 2.1 Generate 2D bounding box annotation: Using the UAV target mask as input, obtain the minimum bounding rectangle of the UAV pixel region through a contour extraction algorithm (such as Canny edge detection). Combined with the resolution ratio of the synthesized image, convert the pixel coordinates (top left x / y, bottom right x / y) of the rectangle into normalized coordinates compatible with the target detection model (coordinate values ​​are mapped to the [0, 1] interval), label the category as "UAV" and assign a unique category ID (such as ID=1).

[0197] 2.2 Generate semantic segmentation labels: Align the target mask in the multi-dimensional rendering image with the synthetic image pixel by pixel, convert the binarized data (0 / 255) of the mask into a single-channel semantic label map, set the pixel value of the UAV region to the category ID (1), and set the pixel value of the background region to 0, to ensure that the resolution of the label map is completely consistent with that of the synthetic image;

[0198] 2.3 Generate depth annotations: Associate the depth values ​​(in meters) in the UAV depth map with the synthetic image according to the pixel position. After removing invalid depth values ​​in the background area, retain the depth data of the UAV area and store it in the form of key-value pairs of pixel coordinates and depth values ​​to help train the depth perception capability of the detection model.

[0199] 2.4 Generate pose association annotations: Bind the pose parameters of the new viewpoint camera to the corresponding synthetic image. The annotation content includes the camera intrinsic parameters (focal length f, principal point coordinates cx / cy), extrinsic parameters (rotation matrix R, translation vector t), and the viewpoint association relationship between the new viewpoint and the original image set (e.g., the new viewpoint is generated by interpolation of the original viewpoint by 30°).

[0200] (3) Organize data and annotations according to the preset format.

[0201] 3.1 Determine the compatible format of the target detection model: The preset format is either COCO or VOC standard format. If it is COCO format, a three-level JSON structure of "images, annotations, categories" is constructed. If it is VOC format, the corresponding XML annotation file is generated.

[0202] 3.2 Perform data association and organization: Store the synthesized image data into the dataset image directory according to the rule of "image ID-storage path", fill the two-dimensional bounding box, semantic segmentation and other annotation information into the corresponding fields according to the preset format (such as COCO's annotations field containing sub-items such as bbox, category_id, segmentation, etc.), and associate the camera pose information and depth annotation key-value pairs of the corresponding image in the annotation file.

[0203] 3.3 Supplement dataset metadata: Generate a dataset description file (README) that records the data scale (number of images, number of annotations), format specifications, category definitions (only UAVs), camera parameter range, and the type of detection model applicable to the data (such as YOLO, Faster R-CNN).

[0204] (4) Dataset validation and output

[0205] Verify the consistency between the synthesized image and the annotation information: check whether the bounding box completely surrounds the UAV region, whether the semantic label matches the target mask contour, and whether the association between the depth value and the pixels of the synthesized image is accurate; after removing data with annotation errors (such as bounding box out of bounds or missing labels), divide the training set, validation set and test set in a ratio of 8:1:1, and output the final UAV target detection synthetic dataset.

[0206] The core objective of automatic annotation output is to automatically transform the previously generated synthetic images and precise geometric information into an image + label combination that can be directly used by the object detection model, completely eliminating the high cost (time and manpower) of manual annotation, while ensuring 100% accuracy of the annotation. In addition, this implementation method can also perform data synthesis based on learned image style transfer or domain adaptation (GAN / Style Transfer). For example, a generative adversarial network (GAN) can be trained or style transfer technology can be used to transform rendered images in ideal environments into images with styles such as haze and low light.

[0207] In summary, this invention addresses the problems in existing UAV target detection data acquisition, such as incomplete field of view coverage, difficulty in accurately reproducing the thin structure (wings, propellers) and reflective features of UAVs, and insufficient diversity of weather scenarios. It proposes an automatic target detection data synthesis method for UAVs, the complete process of which is as follows: Figure 3 As shown, a comprehensive framework covering input data processing, high-fidelity 3D reconstruction, robust enhancement data synthesis, and annotation was constructed: First, in the input data preprocessing stage, based on a small number of multi-view UAV 2D images, the camera intrinsic and extrinsic parameters of each image were accurately calibrated using photogrammetry tools or preset flight path information, outputting an image set with attitude information to provide accurate pre-processing data; then, in the customized NeRF reconstruction stage, the attitude-infused image set was fed into a customized multi-scale attention-aware encoder E... UAV The reconstruction module, composed of conditional NeRF MLP, is made up of E UAVBy extracting pixel-aligned high-level features through multi-scale dilated convolution and attention mechanisms, NeRF MLP combines this feature constraint with the volumetric rendering process to output a multi-dimensional rendered image containing UAV color, depth, and target mask. This enables high-fidelity and high-generalization reconstruction of complex features such as thin UAV structures and reflective surfaces with minimal input views. Next, the process moves to a robust enhancement compositing stage, fusing the foreground rendered by NeRF with a pre-prepared environmental background containing precise depth information. Using this depth information combined with a parameterized physical atmospheric scattering model, physical-level weather effects such as fog and haze are simulated for the foreground and background, outputting high-quality synthetic image data. This automatically generates scene data with high environmental diversity and strong robustness. Finally, the automatic annotation output stage integrates the synthetic image data, multi-dimensional rendered image, and camera pose information. The method automatically generates 2D bounding boxes, pixel-level semantic segmentation masks, and category labels, and finally organizes and generates a synthetic dataset for UAV target detection with precise annotations. The entire process is automated from data acquisition and preprocessing to high-quality labeled dataset output. Therefore, the method of this invention, through a customized neural radiation field model encoder and robustness enhancement synthesis module, can efficiently and faithfully reconstruct 3D UAV models from a small number of 2D images and automatically synthesize highly robust datasets. In addition, the method of this invention can also be applied to scenarios requiring high-precision, all-weather UAV target recognition and detection, such as national defense and military, urban security, traffic monitoring, and automatic cruise systems. It is especially suitable for training target detection models that maintain high robustness under extreme weather or complex lighting conditions.

[0208] Example 2:

[0209] Based on the same inventive concept, this invention also provides an automatic target detection data synthesis system for unmanned aerial vehicles (UAVs), the structural composition of which is shown in the schematic diagram below. Figure 4 As shown, it includes:

[0210] The pose calculation module is used to collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data through photogrammetry tools or preset trajectory information to obtain an image set with pose information.

[0211] The image rendering module is used to perform volume rendering on the image set with attitude information using a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0212] The weather simulation module is used to extract the depth image data of the UAV from the multi-dimensional rendering image, combine it with the pre-acquired environmental background image with depth information, and use the physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data.

[0213] The data synthesis module is used to generate annotation information of the UAV based on the synthesized image data, the multi-dimensional rendering map and the camera pose information, and to organize the synthesized image data and the annotation information in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV.

[0214] The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

[0215] In one implementation, the pose calculation module may include:

[0216] The data acquisition submodule is used to determine the acquisition method of the two-dimensional image data based on the two-dimensional image data.

[0217] The pose inversion submodule is used to inversely calculate the camera pose information of the camera by photogrammetry tools when the two-dimensional image data is acquired by a non-track-associated camera, and associate the camera pose information with the corresponding two-dimensional image data to form an image set with pose information.

[0218] The inertial measurement submodule is used to determine the camera pose information of the UAV by using the GPS positioning module and inertial measurement unit on the UAV when the two-dimensional image data is collected by the UAV according to the preset trajectory information, and bind the camera pose information with the corresponding two-dimensional image data to form an image set with attitude information.

[0219] The non-track-associated camera includes one or more of the following: handheld camera and fixed camera.

[0220] For example, the camera pose information may include: camera intrinsic parameters and camera extrinsic parameters;

[0221] The camera intrinsic parameters may include one or more of the following: focal length information, image sensor size, and distortion coefficient information;

[0222] The camera extrinsic information may include one or more of the following: spatial position information, shooting angle information, rotation matrix and translation vector information;

[0223] The multi-dimensional rendering map may include one or more of the following: drone foreground color map, depth image data, and drone target mask;

[0224] The annotation information may include one or more of the following: two-dimensional bounding boxes, pixel-level semantic segmentation masks, and category labels.

[0225] In one implementation, the image rendering module may include:

[0226] The feature extraction submodule is used to extract the core input features of the model based on the image set with pose information and determine the target for generating a new viewpoint based on the core input features of the model.

[0227] The multi-dimensional rendering submodule is used to input the target generated from the new perspective as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

[0228] In this implementation, the feature extraction submodule may include:

[0229] The information extraction unit is used to extract the core input elements of the model and the existing viewpoint distribution information from the image set with pose information;

[0230] The new perspective generation unit is used to determine the new perspective generation target based on the core input elements of the model and the existing perspective distribution information;

[0231] The core input elements of the model include one or more of the following: two-dimensional image data of the UAV, camera pose information, and geometric prior information of the UAV;

[0232] The geometric prior information includes one or more of the following: three-dimensional bounding box parameter information, thin structure thickness information, and geometric feature location information;

[0233] The existing view distribution information includes one or more of the following: existing view spatial coverage, existing view density distribution, and existing view correlation information;

[0234] The new perspective generation target includes one or more of the following: new perspective pose parameters, new perspective rendering output specifications, new perspective spatial distribution strategy, and new perspective image resolution;

[0235] The new perspective rendering output specifications include one or more of the following: new perspective image resolution, pixel format, and rendering quality requirements;

[0236] The new perspective spatial distribution strategy includes one or more of the following: the new perspective covers the target in space, increases the number of perspectives and new perspectives in low-density areas.

[0237] In this implementation, the multi-dimensional rendering submodule may include:

[0238] The standardization processing unit is used to take the target generated from the new perspective as input data, and to standardize the input data through a pre-built customized neural radiation field model to form standardized input data.

[0239] The feature extraction unit is used to extract thin structure and reflective region features from the two-dimensional image data of the UAV based on the standardized input data and through multi-scale dilated convolution in the customized neural radiation field model.

[0240] The feature combination unit is used to form a joint feature based on the thin structure and the reflective area features, combined with the spatial coding features of the new view frustum; wherein the new view frustum is determined by the camera intrinsic parameter information and camera pose information of the target generated by the new view.

[0241] The attribute generation unit is used to output the core spatial attribute information of the UAV based on the joint features and through the conditional multilayer perceptron in the customized neural radiation field model.

[0242] The spatial rendering unit is used to perform volume rendering on the pixel grid of the new perspective image based on the core attribute information of the UAV's spatial points, so as to obtain a multi-dimensional rendering image of the UAV.

[0243] The pixel grid of the new perspective image is determined by the resolution of the new perspective image in the new perspective generation target.

[0244] In one implementation, the weather simulation module may include:

[0245] The geometric fusion submodule is used to extract the depth image data and foreground image of the UAV from the multi-dimensional rendering image, and combine them with the pre-acquired environmental background image with depth information to perform geometric fusion of the UAV and the environmental background of the UAV to obtain fused scene data.

[0246] The effect simulation submodule is used to simulate the weather effects of the UAV and the environmental background in the fused scene data using a physical atmospheric scattering model, based on the fused scene data, to obtain a scene image with weather features.

[0247] The image synthesis submodule is used to perform consistency optimization and quality verification on the scene image with weather features to obtain synthesized image data;

[0248] The fused scene data includes: pixel-aligned original scene radiance information and depth values ​​corresponding to each pixel.

[0249] For example, the calculation formula for the physical atmospheric scattering model can be as follows:

[0250] ;

[0251] In the formula,

[0252] ;

[0253] in, Indicates pixel position The pixel radiance of the scene image with weather characteristics output after weather effect simulation; Indicates pixel position Atmospheric transmittance at that location; Indicates pixel position The original radiance information of the scene itself; Indicates the brightness of atmospheric ambient light radiation; It is a natural constant; Represents the Rayleigh scattering coefficient; Represents the Mie scattering coefficient; Indicates pixel position The depth value at that location.

[0254] Example 3:

[0255] like Figure 5 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0256] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to implement the corresponding method flow or corresponding function, so as to realize the steps of the automatic synthesis method for target detection data of UAVs in the above embodiments.

[0257] Example 4:

[0258] Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the automatic target detection data synthesis method for UAVs described in the above embodiments.

[0259] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0260] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0261] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0262] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0263] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. An automatic target detection data synthesis method for unmanned aerial vehicles (UAVs), characterized in that, include: Collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset flight path information to obtain an image set with pose information; A pre-built, customized neural radiation field model is used to perform volume rendering on the image set containing attitude information to obtain a multi-dimensional rendering image of the UAV. Depth image data of the UAV is extracted from the multi-dimensional rendering image. Combined with the pre-acquired environmental background image with depth information, the UAV and the environmental background are geometrically fused and the weather effect is simulated using a physical atmospheric scattering model to obtain synthetic image data. Based on the synthetic image data, the multi-dimensional rendering map, and the camera pose information, the annotation information of the UAV is generated, and the synthetic image data and the annotation information are organized in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV. The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

2. The method as described in claim 1, characterized in that, The step of calculating the camera pose information corresponding to the two-dimensional image data using photogrammetry tools or preset track information to obtain an image set with pose information includes: Based on the two-dimensional image data, determine the acquisition method of the two-dimensional image data; When the two-dimensional image data is acquired by a non-track-associated camera, the camera pose information of the camera is inversely calculated using a photogrammetry tool, and the camera pose information is associated with the corresponding two-dimensional image data to form an image set with pose information. When the two-dimensional image data is collected by the UAV according to the preset flight path information, the camera pose information of the UAV is determined by the GPS positioning module and inertial measurement unit on the UAV, and the camera pose information is bound to the corresponding two-dimensional image data to form an image set with pose information. The non-track-associated camera includes one or more of the following: handheld camera and fixed camera.

3. The method as described in claim 1 or 2, characterized in that, The camera pose information includes: camera intrinsic parameters and camera extrinsic parameters; The camera intrinsic parameters include one or more of the following: focal length information, image sensor size, and distortion coefficient information; The camera extrinsic information includes one or more of the following: spatial position information, shooting angle information, rotation matrix and translation vector information; The multi-dimensional rendering map includes one or more of the following: drone foreground color map, depth image data, and drone target mask; The annotation information includes one or more of the following: two-dimensional bounding boxes, pixel-level semantic segmentation masks, and category labels.

4. The method as described in claim 1, characterized in that, The process of using a pre-built, customized neural radiation field model to perform volume rendering on the image set containing attitude information to obtain a multi-dimensional rendering image of the UAV includes: Based on the image set with pose information, the core input elements of the model are extracted, and a new viewpoint is generated based on the core input elements of the model. The target generated from the new perspective is input as input data into a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV.

5. The method as described in claim 4, characterized in that, The step of extracting core input elements of the model based on the image set with pose information and determining the target for generating a new viewpoint based on the core input elements of the model includes: Extract the core input elements of the model and the existing viewpoint distribution information from the image set containing pose information; Based on the core input elements of the model and the existing view distribution information, determine the target for generating new viewpoints; The core input elements of the model include one or more of the following: two-dimensional image data of the UAV, camera pose information, and geometric prior information of the UAV; The geometric prior information includes one or more of the following: three-dimensional bounding box parameter information, thin structure thickness information, and geometric feature location information; The existing view distribution information includes one or more of the following: existing view spatial coverage, existing view density distribution, and existing view correlation information; The new perspective generation target includes one or more of the following: new perspective pose parameters, new perspective rendering output specifications, new perspective spatial distribution strategy, and new perspective image resolution; The new perspective rendering output specifications include one or more of the following: new perspective image resolution, pixel format, and rendering quality requirements; The new perspective spatial distribution strategy includes one or more of the following: the new perspective covers the target in space, increases the number of perspectives and new perspectives in low-density areas.

6. The method as described in claim 4, characterized in that, The process of inputting the target generated from the new perspective as input data into a pre-constructed customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV includes: The target generated from the new perspective is used as input data, and the input data is standardized by a pre-built customized neural radiation field model to form standardized input data. Based on the standardized input data, thin structure and reflective region features in the two-dimensional image data of the UAV are extracted through multi-scale dilated convolution in the customized neural radiation field model; Based on the thin structure and the reflective area features, combined with the spatial coding features of the new perspective frustum, a joint feature is formed; wherein, the new perspective frustum is determined by the camera intrinsic parameter information and camera pose information of the target generated from the new perspective. Based on the joint features, the spatial point core attribute information of the UAV is output through the conditional multilayer perceptron in the customized neural radiation field model; Based on the core spatial attribute information of the UAV, volume rendering is performed on the pixel grid of the new perspective image to obtain a multi-dimensional rendering image of the UAV. The pixel grid of the new perspective image is determined by the resolution of the new perspective image in the new perspective generation target.

7. The method as described in claim 1, characterized in that, The process involves extracting depth image data of the UAV from the multi-dimensional rendered image, combining it with a pre-acquired environmental background image containing depth information, and using a physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data, including: The depth image data and foreground image of the UAV are extracted from the multi-dimensional rendering image. Combined with the pre-acquired environmental background image with depth information, the UAV and the environmental background of the UAV are geometrically fused to obtain fused scene data. Based on the fused scene data, a physical atmospheric scattering model is used to simulate the weather effects of the UAV and the environmental background in the fused scene data, resulting in a scene image with weather features. Based on the scene images with weather features, consistency optimization and quality verification are performed to obtain synthetic image data; The fused scene data includes: pixel-aligned original scene radiance information and depth values ​​corresponding to each pixel.

8. The method as described in claim 1 or 7, characterized in that, The calculation formula for the physical atmospheric scattering model is as follows: ； In the formula, ； in, Indicates pixel position The pixel radiance of the scene image with weather characteristics output after weather effect simulation; Indicates pixel position Atmospheric transmittance at that location; Indicates pixel position The original radiance information of the scene itself; Indicates the brightness of atmospheric ambient light radiation; It is a natural constant; Represents the Rayleigh scattering coefficient; Represents the Mie scattering coefficient; Indicates pixel position The depth value at that location.

9. An automatic target detection data synthesis system for unmanned aerial vehicles (UAVs), characterized in that, include: The pose calculation module is used to collect multi-view two-dimensional image data of the UAV, and calculate the camera pose information corresponding to the two-dimensional image data through photogrammetry tools or preset trajectory information to obtain an image set with pose information. The image rendering module is used to perform volume rendering on the image set with attitude information using a pre-built customized neural radiation field model to obtain a multi-dimensional rendering image of the UAV. The weather simulation module is used to extract the depth image data of the UAV from the multi-dimensional rendering image, combine it with the pre-acquired environmental background image with depth information, and use the physical atmospheric scattering model to perform geometric fusion and weather effect simulation on the UAV and the environmental background to obtain synthetic image data. The data synthesis module is used to generate annotation information of the UAV based on the synthesized image data, the multi-dimensional rendering map and the camera pose information, and to organize the synthesized image data and the annotation information in a preset format compatible with the target detection model to generate the target detection synthetic dataset of the UAV. The customized neural radiation field model includes at least an encoder and a conditional multilayer perceptron optimized for the thin structure and reflective characteristics of the UAV.

10. The system as described in claim 9, characterized in that, The pose calculation module includes: The data acquisition submodule is used to determine the acquisition method of the two-dimensional image data based on the two-dimensional image data. The pose inversion submodule is used to inversely calculate the camera pose information of the camera by photogrammetry tools when the two-dimensional image data is acquired by a non-track-associated camera, and associate the camera pose information with the corresponding two-dimensional image data to form an image set with pose information. The inertial measurement submodule is used to determine the camera pose information of the UAV by using the GPS positioning module and inertial measurement unit on the UAV when the two-dimensional image data is collected by the UAV according to the preset trajectory information, and bind the camera pose information with the corresponding two-dimensional image data to form an image set with attitude information. The non-track-associated camera includes one or more of the following: handheld camera and fixed camera.

Images