A spacecraft-oriented multi-task data set construction method

Abstract

The application provides a spacecraft-oriented multi-task data set construction method, and creates a high-precision satellite geometric model. The image based on PBR rendering not only has a bright visual effect, but also conforms to the physical optical principle, especially the simulation of the star body wrinkle effect by the normal map. Unreal Engine 4 is used to build a space simulation scene, and a realistic rendering effect is provided. Combined with the scene capture component and the rendering target function of the unreal engine, a high-quality satellite image is generated. The 2D image automatic labeling method based on the 3D model automatically generates semantic segmentation labels, instance segmentation labels, depth labels, normal labels and pose labels. The frame-by-frame rendering and automatic labeling method can quickly generate a large-scale data set. The generated image is subjected to semantic segmentation by a deep learning model DeepLab V3+, so that the accuracy and efficiency of model prediction are further improved, and the reality of rendering is greatly improved through careful analysis and optimization.

Description

[Technical Field]

[0001] This invention belongs to the field of intelligent space target recognition technology, and in particular relates to the simulation dataset required for intelligent recognition. Specifically, it is a method for constructing a multi-mission image dataset for spacecraft based on Unreal Engine. [Background Technology]

[0002] With increasing global emphasis on the space environment, the importance of space situational awareness is rising. Attacks, protection, and on-orbit servicing of critical space targets, primarily satellites, have become key development directions for aerospace technology worldwide. Furthermore, the number of space debris and defunct spacecraft is constantly increasing, making space debris removal and on-orbit servicing missions crucial development areas in aerospace. Satellite intelligent identification technology is a key component, with features such as solar panels and antennas requiring recognition, and position and attitude estimation technologies being extremely important. Since non-cooperative target spacecraft lack identifiable markers, accurately obtaining their position and attitude information relative to servicing satellites is critical. In recent years, artificial intelligence algorithms, represented by deep learning, have flourished. Key technologies such as target detection, semantic segmentation, and pose estimation have been successfully applied in multiple fields, bringing new technological breakthroughs to space target identification.

[0003] However, due to the lack of space target datasets for training, research on applying deep learning to intelligent spacecraft recognition is still relatively limited. Some scholars have used software such as STK to simulate satellites operating in Earth orbit and extracted images to create satellite datasets. However, due to limitations in the modeling quality of satellites and the Earth, the constructed satellite images differ significantly from real images. Simulated images often lack an Earth background, have relatively simple lighting conditions, lack the inherent visual features and lighting variations of space-based camera images, and have a limited number of models. Furthermore, simulated images lack detailed annotations, and manual annotation is costly and cannot be applied to large-scale training. Deep learning algorithms trained using such simulated images are difficult to apply to real-world, in-orbit satellite recognition. Therefore, how to construct high-quality, high-fidelity space target datasets suitable for various visual tasks and automatic annotation methods for simulated images have become research priorities.

[0004] Unreal Engine is a powerful game development engine with highly realistic scene development capabilities. Therefore, space scenes can be built using Unreal Engine software, satellite models can be imported, and automatically labeled satellite datasets can be generated in batches, providing data support for subsequent identification of spacecraft feature components based on deep learning methods. [Summary of the Invention]

[0005] (a) Purpose of the invention

[0006] This invention overcomes the shortcomings of existing satellite datasets, such as unrealistic lighting and limited quantity, and provides a method for constructing multi-mission datasets for spacecraft. It provides large-scale, multi-type labeled automatically generated satellite images with high similarity to real on-orbit images for intelligent identification of spacecraft feature components.

[0007] (II) Technical Solution of the Invention

[0008] Step 1: Establish a high-quality satellite model

[0009] A 3D model of the space target was constructed using the modeling software 3DS MAX. Polygonal modeling was employed, and a detailed satellite geometric model was established by referring to satellite structure diagrams, dimension diagrams, and schematic diagrams. First, basic modeling of the satellite was performed, creating the approximate shape and structure of the object. Satellite components, including the satellite body, antennas, solar panels, high-resolution lenses, star sensors, thrusters, and docking rings, were meticulously constructed according to dimensions and assembled according to the satellite structure diagram. For satellites of general shapes, the coordinate origin was adjusted to the center of the satellite body, the y-axis was aligned with the deployment direction of the satellite solar panels, and the z-axis was aligned as closely as possible with the orientation of the satellite cameras or antennas. For satellites with special shapes, the coordinate axes were aligned with the working directions of the satellite's critical payloads. This ensured the accuracy and consistency of the pose information generated in subsequent steps.

[0010] The model is UV unwrapped to ensure textures are correctly mapped onto its surface. This is then wrapped back onto the 3D model. Physically based rendering (PBR) material maps are created based on the model's UVs, including base color maps, normal maps, roughness maps, and metallicity maps. The PBR rendering pipeline makes lighting rendering more realistic. The model and material maps are imported into Unreal Engine, material spheres are created, and the material maps are wrapped back onto the 3D model. This invention creates 300 different types of satellites, converting the model into Unreal Engine assets for later use.

[0011] Step 2: Annotate the 3D model

[0012] This paper presents an automatic 2D image annotation method based on 3D models. It automatically generates original images of spatial targets and semantically annotated images of feature components, constructing a semantic segmentation dataset for key spatial target components. This method requires annotation of the 3D model, specifically using different colored materials to mark feature components. This allows for the determination of the type of feature component belonging to each pixel in the image during 2D rendering based on the characteristics exhibited by different materials.

[0013] Step 3: Construct the rendering scene

[0014] In this invention, a spatial simulation scene is built based on Unreal Engine 4, including the construction of spatial lighting and background. Spatial lighting construction involves the simulation of sunlight, ambient lighting, and stray light; sunlight is simulated using Unreal Engine's directional light sources. Spatial background construction includes the simulation of the Earth and starry sky backgrounds. A blueprint of the Earth is created using high-resolution images from NASA, featuring customizable selection, day / night cycles, and atmospheric flow. A rendering scene is set up to support the satellite model. A cube environment map is created from High Dynamic Range (HDRI) images and attached to a sky light source to construct ambient lighting. An HDRI starry sky image is attached to a celestial sphere object to construct the starry sky background.

[0015] Step 4: Generate spatial target image

[0016] The 3D satellite model (Obj or FBX format) created in Step 1 is imported into the pre-built space scene using Unreal Engine 4's built-in model import tool. Next, a camera is created using the Scene Capture Component 2D, capable of capturing the 3D scene and outputting it as a 2D image for generating satellite images. These images are then processed using Unreal Engine's Render Target feature, which allows the rendered results to be converted into textures at runtime, storing various image information. These settings include adjustments to the color gamut of the capture source, hidden actors, gamma values, motion blur, and post-processing volume, as well as manual settings of exposure parameters to simulate the imaging effects caused by ambient light and satellite motion. To enhance the realism of the images, an OpenCV plugin is introduced for post-processing. OpenCV is integrated into the UE4 project as a plugin, and post-processing of color arrays is implemented through C++ functions and Blueprint interfaces, thereby improving image generation efficiency. Gaussian blur and random noise are used to further enhance the realism of the images.

[0017] Furthermore, the user interface was created using the Unreal Graphics Interface Designer (UMG) within Unreal Engine to control and display the simulated space environment. This interface includes a real-time running interface, raw and annotated image display windows from the camera viewport, and multiple function buttons such as screenshot, model replacement, attitude adjustment, and lighting adjustment. The screenshot function is implemented using a custom C++ function, converting the rendered target into a PNG file for saving. The model replacement function allows users to select different satellite models and automatically replace them to improve the automation of batch image generation. Finally, the on-orbit attitude of the satellite was simulated, including setting the pose, linear velocity, and angular velocity of the satellite model to achieve motion simulation and motion blur effects.

[0018] Step 5: Generate multi-type annotations

[0019] Multiple annotation types include image classification labels, semantic segmentation masks, instance segmentation masks, depth maps, surface normal maps, and position and pose labels. These annotations cover all annotation types in computer vision tasks. This invention proposes an automatic 2D image annotation method based on 3D models. In Unreal Engine, we achieve simultaneous export of simulated and annotated images of space targets by conjugating the original satellite model and the annotated satellite model. This process involves several key steps. First, it involves importing the original satellite model and its annotated models of key components into UE4, and creating two static mesh scene components in the simulation environment to hold these models. These models are placed on a designed satellite orbit, and the position and pose of the annotated models completely overlap with the original satellite models in the scene.

[0020] To capture images of both models simultaneously, two identical cameras were used, differing only in their rendering settings. One camera captured the original satellite image, set to sRGB final colors as the capture source, displaying all scene components except for the labeled satellite model; the other captured the labeled image, set to RGB basic colors as the capture source, masking all scene components except for the labeled satellite model. Two rendering targets were used to combine the satellite images from both cameras.

[0021] In addition, the system generates depth maps and normal maps. The depth map is obtained by calculating the distance between the spatial point corresponding to each pixel and the camera's optical center, with the capture source set to "scene depth in R"; the normal map captures the surface normal direction information of each pixel in the scene, with the capture source set to "RGB normal". These images are generated by setting specific camera capture sources and rendering targets.

[0022] The generation of pose information is based on UE4 blueprints and C++ classes, and is ultimately saved as a JSON file. The pose information of the satellite and its key components is obtained by defining a coordinate system and using blueprint functions, and then stored in JSON format using C++ functions. In the JSON file, the pose information is presented in a nested structure, clearly listing the pose information of different components, and also includes additional information such as illumination angle, Earth's position, image size, and field of view.

[0023] Overall, this process, through UE4's advanced features and visualization tools, automatically generates 2D images containing rich spatial information, and effectively annotates and stores the information for subsequent use and analysis.

[0024] Step Six: Optimization and Enhancement

[0025] In on-orbit equivalent simulations of space imaging, the surface material texture of space targets has the greatest impact on rendering effects. Satellite surface materials can be summarized as follows: thermal insulation layer, solar panels, back of the solar panels, metallic surfaces, and non-metallic paint surfaces. By meticulously analyzing and optimizing these materials, the realism of the rendering is improved. The main optimizations are as follows: Optimizing the rendered materials of space targets using photo blending techniques. First, photographs are taken of the main materials of the space targets (thermal insulation layer, solar panels, etc.) using a camera, and texture maps are created using Reality Capture. These texture maps are then blended with the original textures using photo blending techniques, thereby optimizing the rendered materials and enhancing the realism of the space imaging simulation. Material properties are analyzed, and reference photographs and real materials are carefully observed to adjust surface roughness, color variations, and reflectivity. Deep learning model evaluation: The semantic segmentation model is trained using generated sample data, and its performance on the target domain test set is analyzed to measure the overlap between the model's predictions and the real labels. Improvements and optimizations are made based on the model's recognition performance on the aforementioned constructed feature component database.

[0026] (III) Advantages and benefits of the present invention

[0027] This invention mainly involves the construction, annotation, scene construction, image generation, and optimization of high-quality satellite models to improve the simulation realism and effect of space imaging. The main advantages and effects of this invention are as follows: A high-precision satellite geometric model is created. A unified model coordinate axis and origin are used, and a PBR material system is employed to create realistic material textures. Images rendered based on PBR not only have striking visual effects but also conform to the principles of physical optics. In particular, normal maps simulate the wrinkling effect of the satellite body. Unreal Engine 4 is used to construct a space simulation scene, including space lighting, background, and global illumination, providing realistic rendering effects. High-quality satellite images are generated by combining Unreal Engine's scene capture component and rendering target function. A 2D image automatic annotation method based on 3D models is used to automatically generate semantic segmentation labels, instance segmentation labels, depth labels, normal labels, and pose labels. Frame-by-frame rendering and automatic annotation can quickly generate large-scale datasets. The generated images are semantically segmented using the DeepLab V3+ deep learning model, further improving the accuracy and efficiency of model prediction. Through meticulous analysis and optimization, the realism of the rendering is greatly improved.

[0028] Overall, this invention effectively improves the realism and effect of space imaging simulation by comprehensively utilizing advanced modeling technology, rendering technology, and deep learning methods, providing strong technical support for the research and application of intelligent recognition of space targets. [Attached Image Description]

[0029] Figure 1 This is a flowchart of the present invention.

[0030] Figure 2a , Figure 2b , Figure 2c , Figure 2d , Figure 2e , Figure 2f , Figure 2g , Figure 2h , Figure 2i Examples of partial satellite models established for this invention.

[0031] Figure 3a , Figure 3b This is a schematic diagram comparing the satellite model and the labeled model established in this invention.

[0032] Figure 4 The rendering effect of the spatial simulation scene established for this invention.

[0033] Figure 5 The user interface established for this invention.

[0034] Figure 6 The flowchart of the batch drawing output program established for this invention is shown.

[0035] Figure 7 This is a schematic diagram illustrating the motion blur effect of the spatial target image constructed according to the present invention.

[0036] Figure 8a , Figure 8b This is a diagram illustrating the automatic annotation effect of 2D images based on 3D models according to the present invention.

[0037] Figure 9 A flowchart for optimizing and enhancing the image rendering effect of this invention.

[0038] Figure 10 This is a promotional image for the EAGLE satellite.

[0039] Figure 11 This is a 3D model of the EAGLE satellite.

[0040] Figure 12a , Figure 12b The spatial target image and semantic segmentation annotation generated by this invention.

Detailed Implementation Methods

[0041] The following is in conjunction with the appendix Figure 1 - Appendix Figure 12b The technical solution of the present invention will be further described in detail below.

[0042] A flowchart of a multi-mission dataset construction method for spacecraft proposed in this invention is shown below. Figure 1 As shown.

[0043] Step 1: Establish satellite geometric model

[0044] Building a high-quality satellite model involves creating a 3D satellite model and physically based rendering (PBR) texture mapping. The model building process includes steps such as 3D modeling, UV mapping, texture painting, texture mapping, and rendering testing, which will be described in detail below. Figures 2a-2i Examples of partial satellite models established for this invention. Figure 2a It is an Orion electronic reconnaissance satellite, which has a large dish-shaped antenna for signal reception. Figure 2b This is the Gaofen-13 remote sensing satellite. Figure 2c This is the Phobos-2 remote sensing satellite, whose main body is a hexagonal prism and equipped with four solar panels. Figure 2d It is a GSSAP reconnaissance satellite. Figure 2e It is the EAGLE satellite, which is cylindrical in shape, has a single solar array, and contains multiple payloads to form a parent-child satellite. Figures 2f-2i These are TDRS13, Telstar12V, Tianlian-1, and Sirius XM8. All of these are communication satellites, employing a dual-solar-wing design and featuring circular communication antennas.

[0045] High-quality 3D modeling: A 3D model of the space target is constructed using the 3DS MAX modeling software. Polygonal modeling is employed, and a detailed satellite geometric model is established by referring to satellite structure diagrams, dimension diagrams, and schematic diagrams. Before modeling, relevant materials for satellite modeling are collected and organized by consulting official websites or authoritative websites, including information on appearance, dimensions, surface materials, and payload types. For the same satellite, we cross-verify and supplement relevant information by consulting different websites, maximizing the authenticity and accuracy of the satellite material sources. The satellite is then reconstructed using polygonal modeling in 3ds Max. First, basic satellite modeling is performed, creating the approximate shape and structure of the object. Satellite components such as the satellite body, antennas, solar panels, high-resolution lenses, star sensors, thrusters, and docking rings are established based on the satellite's envelope dimensions (length, width, and height) and component proportions, and these components are assembled according to the satellite structure diagram. The initial shapes of satellite components are created, typically starting with basic geometric shapes (such as cubes, spheres, and cylinders). In polygonal modeling, the model consists of vertices, edges, and faces. A vertex is a corner point of a polygon, and edges connect vertices to form the outline of the polygon. A face is composed of multiple edges and represents the surface of an object.

[0046] Secondly, subdivision modeling is used to add more detail, refining the shape and features of the model. Extrusion is a commonly used operation, generating new geometry on existing surfaces to create complex shapes. Subdivision adds more vertices and faces, making the model more detailed. The shape of the model can be adjusted by manipulating vertices, edges, and faces individually through movement, rotation, and scaling. The smoothing tool can remove abrupt changes to the surface, making the model surface smoother.

[0047] Furthermore, for repetitive components such as solar panels, the array function can be used to copy polyhedra to avoid repeated modeling. For irregularly shaped components such as antennas, spline curves can be used to assist in modeling and improve the model's precision. Attention should also be paid to modeling the connecting mechanisms to ensure the basic mechanical functions of the satellite model. Small components such as screws can be simplified without affecting the appearance. Model layers should be set up and grouped with named names for easy selection.

[0048] Different models often use different coordinate axes and origins during construction, which is detrimental to batch control of model movement and sample generation. Therefore, it is necessary to align the model origin and coordinate axes to be consistent. Specifically, for satellites of a general shape (square satellite body, dual solar panels), the coordinate origin should be adjusted to the center of the satellite body, the y-axis should be aligned with the deployment direction of the satellite's solar panels, and the z-axis should be aligned as closely as possible with the orientation of the satellite's camera or antenna. For satellites with special shapes (such as...), the coordinate origin should be adjusted to the center of the satellite body, the y-axis should be aligned with the deployment direction of the satellite's solar panels, and the z-axis should be aligned with the orientation of the satellite's camera or antenna. Figure 2a To ensure the coordinate axes are aligned with the working direction of the satellite's critical payloads, we must strive to ensure the accuracy and consistency of the pose information generated in subsequent steps.

[0049] UV Unwrapping: UV unwrapping prepares the model for texture mapping. UV processing of a 3D model is the process of mapping 2D textures onto the 3D model. First, seams are marked and cut into the model. These seams define how the model is cut when converting from 3D to 2D, determining how the model unfolds on the 2D plane. Second, UVs are unwrapped using the UV unwrapping tool in 3D software. The position and size of UV islands are adjusted to maximize the use of texture space, ensuring that important detail areas have sufficient space on the UV map. All parts of the model must have a consistent texture density, avoiding areas that are too large or too small. Finally, the completed texture is mapped back onto the 3D model, and the effect of the texture is checked in the 3D software to ensure that the texture is correctly mapped onto the model's surface. Final adjustments and optimizations are made to the UV map and texture as needed. The UV layout is exported as an image file for use in texture creation software.

[0050] Physically Based Rendering (PBR) Texture Painting: Creating PBR Material Maps Based on Model-Based UVs. PBR materials refer to physically based rendering material systems, widely used for their more realistic rendering and compatibility and interpretability, which facilitate industrial production. Using Substance Painter software and reference materials, texture maps were created for each part of the satellite, including the heat shield and solar panels. PBR maps include a base color map, normal map, roughness map, and metallicity map. Different maps serve different functions, ultimately combining to create the material for each part. The base color map defines the material's color and some basic details, creating the color and texture of real-world objects. The metallicity map defines the material's metallic properties. It distinguishes the refraction and reflection between metallic and non-metallic materials (also known as dielectrics); materials with high metallicity have stronger specular highlights. The roughness map defines the surface roughness of the material, affecting light scattering. Low roughness values ​​(smoother surfaces) produce sharp specular highlights, while high roughness values ​​(rougher surfaces) produce blurred highlights. Normal maps are used to simulate small details and textures on a material's surface without adding geometric details. These maps change the direction of light reflection, thus simulating an uneven effect.

[0051] UV texture mapping: Use UV coordinates to ensure textures are correctly mapped to the model surface. Set appropriate lighting and rendering parameters, perform test renders, and check the performance of the model and materials under different lighting conditions. Export the drawn textures in either TGA or JPG format, ensuring each texture is the same size for easy rendering by other software. Keep the exported textures for later use.

[0052] Importing to Unreal Engine for rendering test: In 3DS MAX software, the satellite model was exported as FBX format. Compared to the widely used .obj format, FBX format can automatically import material maps after importing the model, making it more efficient for batch processing. FBX import follows these settings: Set the imported model as a static mesh instead of a skeletal mesh, automatically generate collisions; check "Build Adjacent Buffers" to use PNT tessellation; uncheck "Remove Degeneracy" to retain found degenerate triangle meshes; check "Build Inverse Index Buffers"; check "Generate Lightmap UVs"; check "Each Vertex"; check "Merge Mesh" to prevent the imported model from being saved as separate parts for easier management; check "Convert Vertices to Absolute"; in the normal import options, select "Import Normals" instead of "Calculate Normals," and specify the model's normal storage path to calculate weighted normals; in the material import options, select "Search Locally for Model Materials and Create New Material Spheres" to automatically import the model and its materials, saving them as static meshes and corresponding material spheres respectively for easy management. All material maps (base color map, roughness map, specular map, metallic map, and normal map) will be automatically imported into the material sphere. After import, the satellite model is converted into a Unreal Engine asset (UESET), which can be used within Blueprints. The base color map defines the material's color and some basic details, creating the color and texture of real-world objects.

[0053] Step 2: Annotate the 3D model

[0054] An automatic annotation method for 2D images based on 3D models is proposed to automatically generate original images of spatial targets and semantically annotated images of feature components, thereby constructing a semantic segmentation dataset of key components of spatial targets.

[0055] The automatic annotation method for 2D images based on 3D models first requires annotating the 3D model by using different materials to mark feature parts. This allows the type of feature part to be identified for each pixel in the image during planar rendering based on the characteristics exhibited by different materials. For 3D models built using 3DS MAX software, the software's layer function allows for easy filtering and categorization of the meshes represented by each feature part, assigning them different solid-color materials. Specifically, in the Layer Explorer, the "Group" function is used to assemble the components into feature parts. Then, in the Simplified Material Editor, solid-color materials are created by modifying the self-illumination color and opacity, and different materials are assigned to each feature part according to the corresponding color table. Taking the GSSAP satellite model as an example, the rendering effects of the original model and the annotated model in 3DS MAX software are as follows: Figure 3a , Figure 3b As shown.

[0056] After the model annotation is completed, import the original spatial target model and the annotated model into Unreal Engine using the model import method in Step 1, converting them into Unreal Assets for later use. Create a new material sphere in the folder, set its color according to the color label mapping table, and name it the same as the material in the 3DS MAX material editor, establishing a one-to-one correspondence with the label. After importing into Unreal Engine, the model will automatically search for materials, enabling material sphere reuse.

[0057] Step 3: Construct the UE4 scene

[0058] Unreal Engine 4 is a powerful game development engine; therefore, this invention uses Unreal Engine 4 to build a spatial simulation scene. This mainly includes: spatial lighting construction: simulation of sunlight, ambient lighting, and stray light; and spatial background construction: simulation of the Earth background and the starry sky background.

[0059] Space Lighting Construction: Unreal Engine provides various light sources, including directional lights, point lights, spotlights, and rectangular lights. Sunlight is high-intensity, uniform, parallel light. Directional lights are the primary outdoor light sources simulating light emanating from extremely far or near-infinite distances. Because orbital space lacks an atmosphere, intense sunlight shines directly on the spacecraft. Directional lights use directional normal illuminance, expressed in lux, equal to one lumen per square meter. It represents the illuminance produced on Earth by the visible portion of direct solar radiation on a surface perpendicular (or orthogonal to) sunlight. It represents the total amount of illumination falling on a surface. The intensity of the directional light source is set to 150,000 lux.

[0060] Ambient light refers to indirect light formed by the reflection, scattering, and refraction of light sources in a scene. In spatial scenes, it mainly includes reflected light from the Earth and Moon, and starlight. Applying HDRI images to ambient lighting models in computer graphics rendering involves several steps. First, obtaining the HDRI image used to construct the ambient lighting is necessary. These images are typically taken by professional photographers using special photographic techniques, or they can be obtained from an HDRI image library. Applying the HDRI image to the scene's ambient lighting model is then done. In UE4, the HDRI image is mapped to a cube (called an environment map), and this map provides the scene with ambient light illumination information. This cube environment map is then applied to the sky light source. Finally, the intensity, color, and other properties of the ambient light can be adjusted. Using HDRI to construct ambient lighting allows for a more accurate simulation of real-world lighting effects, providing more realistic rendering results.

[0061] Satellite rendering scene: Define a custom "Static Mesh" class to hold the satellite model. Multiple instances can be added to the scene to hold multiple satellite models. Create two instances of this class in the scene, named "SAT" and "SAT_label" respectively. Set collision to pure physics, check the simulated physics option, and disable gravity and impulse. Under the lighting tab, check cast shadows.

[0062] Add a "Lightmass Critical Volume" module to the satellite model. This module is a box that ensures higher lighting quality within the box, improving lighting performance. Global Illumination performance depends on the scene size, and lighting build time will also increase. The Global Illumination Critical Volume controls the area where global illumination emits photons, concentrating them on areas requiring sharp indirect lighting. Areas outside the critical volume will only receive a single indirect lighting reflection at lower quality.

[0063] Earth Background Construction. First, the planetary body, a sphere, is created and its size is adjusted to its actual proportions. Then, textures are collected and created, including surface terrain, surface environment, nighttime lights, atmosphere, and auroras. These textures are based on high-resolution image data from NASA, with colors and tones adjusted according to true-color photographs. Next, material functions are established and material spheres are created, utilizing some textures to construct relationships. An Earth blueprint is created, setting the static mesh as a sphere, configuring various material spheres, and adding events. Features such as sunlight direction, Earth's spin speed, day-night cycle, and atmospheric flow are designed within the blueprint.

[0064] Considering that space target mission simulations mainly focus on Earth satellite orbits, there are higher requirements for Earth modeling. To ensure that Earth images acquired from various perspectives and distances have high resolution and fidelity, we adopted more refined materials and combined them with physical information to construct an Earth blueprint.

[0065] Creating the Starry Sky Background. First, collect and create a High Dynamic Range (HDRI) image of the starry sky. This is a 360-degree panoramic image, providing a full view of the surrounding environment. Next, create a material for the HDRI image. In the Material Editor, connect the HDRI image to an Emissive color node. Ensure the material's smoothness and metallicity are set to 0 to better simulate the background's appearance. Add a Sky Sphere object to the scene. Apply the created HDRI material to the Sky Sphere. Ensure the material is correctly mapped onto the Sky Sphere to display the full starry sky background. Adjust the Sky Sphere's size, position, and rotation as needed to ensure the background looks natural and fits the scene's perspective. The rendering result is as follows. Figure 4 As shown.

[0066] Step 4: Generate spatial target image

[0067] Import the space satellite model created in step one into the space scene created in step three. You can import Obj or Fbx format files using the model import tool that comes with Unreal Engine 4.

[0068] A camera is created using the "Scene Capture Component 2D" class to capture satellite images. It can capture 3D scenes and output them as 2D images. Unreal Engine's "Render Target" feature is used to handle the satellite images generated by the camera for subsequent processing. A render target is a texture in Unreal Engine that can be written to at runtime. It can store information such as base color, normals, and ambient occlusion, allowing users to convert the engine's rendering results into material textures for further editing, rendering, or storage. Specifically, a render target is created to store the satellite images generated by the camera, and the render target is edited to set appropriate resolution and format. In the UE4 editor, the "Scene Capture Component 2D" is added to the scene, thus obtaining a camera instance. The created render target is set as a "Texture Target," the capture source is set to the final color of the linear sRGB color gamut, the "Hidden Actor" is set to the annotation model, the gamma value is adjusted to simulate ambient light, motion blur is set to simulate image degradation caused by satellite motion, post-processing volume is set, and exposure parameters are manually set. With these settings, the satellite image rendered with spatial lighting environment can be presented in the render target.

[0069] To achieve more realistic rendering effects, we introduced an OpenCV plugin into UE4 to enhance the generated images through post-processing. OpenCV was integrated as a plugin into the UE4 project, using its header files to call its functions and classes. C++ functions were written and their interfaces exposed to Blueprints to call OpenCV functions for post-processing the generated color arrays. This method effectively avoids repeated read / write operations on the hard drive, improving image generation efficiency. Specifically, the conversion functions ConvertTArrayToCvMat and ConvertCvMatToTArray were first written to convert the color array obtained from the rendering target into an OpenCV format array. Then, the GaussianBlur function from the OpenCV library was used to add Gaussian and random noise to the image.

[0070] The batch rendering program and user interface are designed based on the Unreal Engine's Unreal Graphics UI Designer (UMG). UMG is a visual UI creation tool used to create UI elements such as HUDs, menus, or other interface-related graphics in a project. At the core of UMG are controls, which are pre-made functions used to build interfaces (such as buttons, checkboxes, sliders, progress bars, etc.). These controls are edited in a dedicated control blueprint, which is constructed using two tabs: the Designer tab allows for a visual layout of the interface and basic functions, while the Chart tab provides the functionality behind the controls used. The constructed user interface is as follows: Figure 5 As shown, the right side of the interface displays the real-time running interface of the simulated spatial environment. The two windows at the top of the interface are the original image and the labeled image of the spatial target from the camera's viewport, respectively. The left side of the interface contains function buttons for screenshotting, replacing models, adjusting pose, changing backgrounds, adjusting lenses, outputting pose, and batch generating images. The structure of the simulated image batch generation module is as follows: Figure 6 As shown, the process begins with selecting a satellite model from a pre-defined list. Next, satellite attitude simulation is performed, images are captured by the camera, and post-processing and image saving occur. Then, the satellite attitude is iterated through according to rules, generating images in a loop. Finally, the model list is iterated through until all images have been generated. The implementation methods for these functions are described below.

[0071] The screenshot function is used to capture images taken from a specified camera viewpoint and export and save them. Since Unreal Engine does not directly provide an export interface for camera render targets, we designed our own screenshot saving function using C++ functions provided by Unreal Engine. This function converts the specified render target into a .png image file and saves it to the local directory. Specifically, the program imports the render target texture material input by calling the render target source class FTextureRenderTargetResource, uses the class method ReadPixels() to transfer the texture pixel by pixel into a color array to form a bitmap, and then converts the bitmap into a PNG file format for local saving.

[0072] The model replacement function allows users to select satellite models from the space target model database in the user interface to replace static meshes in the simulated space scene. It also provides an interface for automatic model replacement, improving the automation level of the batch mapping function.

[0073] The function "Get All Actors of Labeled Classes" searches for a specified Actor among all objects in the project based on Actor type and label. The label is set in the Actor details options; for the original satellite model, it's set to "sat," and for the labeled satellite model, it's set to "sat_label." Two pipelines are set up in the program to simultaneously replace the original and labeled models. Here, the Actor class option for the "Get All Actors of Labeled Classes" function is set to "Static Mesh," and the labels are "sat" and "sat_label," respectively. The "Set Static Mesh" function is then used to specify the satellite model for the selected Actor. References to all satellite models are added to an array, and the "get" function is used to retrieve the static mesh references of the satellite models and connect them to the "Set Static Mesh" function. Users can select and replace satellite models by entering their model numbers from the keyboard.

[0074] Simulate the satellite's on-orbit attitude. For motion simulation tasks, Unreal Engine can provide the 6D pose of the satellite's 3D model center position. By "Get All Actors of Labeled Classes," the target model in the simulation space environment is located. Combined with "Set Actor Spatial Position" and "Set Actor Rotation," the 6D pose of the target model's center point can be set. In batch rendering, the three Euler angles of the satellite model are traversed, each axis is divided into 15 parts, and randomization is performed within each interval using "Random Floating-Point Numbers within the Range." In step three, the physics simulation of the satellite model is enabled. Combined with "Set Physical Linear Velocity" and "Set Physical Angular Velocity," the linear and angular velocities of the target model can be set from the blueprint. When there is a relative velocity between satellites, motion blur can be simulated. The motion blur effect is as follows. Figure 7 As shown.

[0075] Step 5: Automatically generate multiple types of annotations

[0076] Based on the automatic annotation method for 2D images using 3D models, in Unreal Engine, we achieve simultaneous export of simulated and annotated images of space targets by conjugating the original satellite model and the annotated satellite model. First, all original satellite models and their annotated key components are imported into Unreal Engine for later use. Two new static mesh scene components are created to hold the models. Two models of a specific satellite type are embedded into the mesh components, and the models are placed on the designed simulated satellite orbit. The annotated model is then attached to the satellite model, and the pose transformation of the annotated model relative to the satellite model is set to zero, ensuring that the poses of the two models in the scene are completely overlapped. Figure 8a , Figure 8b As shown.

[0077] To facilitate simultaneous imaging and image saving of both models, two configured cameras are used to capture the images of the two models respectively. The poses and parameter configurations of the two cameras are kept identical; the difference lies in the rendering settings. The cameras are constructed using the method described in step four. Specifically, two rendering targets are used to capture the satellite images generated by the two cameras. The rendering targets are edited: for the original satellite image, the capture source is set to sRGB final colors, displaying all scene components except for the labeled satellite model; for the labeled satellite image, the capture source is set to RGB basic colors, masking all scene components except for the labeled satellite model, turning off all lighting, and keeping other settings consistent with the first rendering target. The display effect is the key component annotation result of the original satellite image. With these settings, the original satellite image with normal lighting and shadow rendering and the key component annotation image using RGB primary colors can be presented in the two rendering targets. The two rendering targets are then packaged into a rendering target texture for later use.

[0078] A depth map is an image mapped from the distance of each pixel in the generated image to the camera's optical center, containing spatial information. A normal map captures the surface normal direction information of each pixel in the scene, typically represented as an RGB image. We create depth maps based on Unreal Engine's realistic physics system, calculated using the absolute distance between the satellite model's center and the camera's optical center, and the relative spatial distribution of the satellite model's components. Surface normal information is usually obtained by capturing the normal vector of each pixel and mapping it to the RGB color space. Specifically, we construct two cameras similar to those in step four, setting the capture source to "RGB normal" and "scene depth in R". We set the rendering target to receive the images from both cameras. We use the "ReadPixels" function to read pixel data from the rendering target; this data can be used to generate the normal map and depth map. The normal map is output as an RGB image. Raw depth data is typically non-linear, and its range and accuracy depend on the depth buffer configuration. We normalize the depth values, adjust their range, and output the processed depth data as a grayscale image.

[0079] Pose information generation is implemented using UE4 blueprints and C++ classes, and saved to a JSON file. For the pose information of key components, such as optical payloads, a coordinate system is defined, and the pose information of the satellite and its key components in the space scene is obtained using the blueprint function "Get Actor Transform". The pose information of the satellite relative to the camera is calculated by "Reverse Transform Rotation". The "json.h" library is included in the C++ function, and the "jsonWriter" class is used to store the pose information in JSON format. A nested structure is built in the JSON file to clearly list the pose information of different components. In addition, this function reads and stores information such as illumination angle, Earth's position, image size, and field of view. This function interface is exposed in the blueprint, referenced in the drawing program, and parameters such as relative pose are passed to this function to generate the pose information file.

[0080] Step Six: Optimization and Enhancement

[0081] In on-orbit equivalent simulations of space imaging, the surface texture of space targets has the greatest impact on rendering results. Satellite surface materials can be summarized into the following parts: thermal insulation layer, solar panels, back of the solar panels, metallic surface, and non-metallic paint surface. By meticulously analyzing and optimizing these materials, the realism of the rendering can be improved. The optimization process is as follows: Figure 10 As shown, the main optimization steps are as follows:

[0082] (1) Optimize the rendering materials of spatial targets using photo blending technology. First, take photos of the main materials of the spatial targets (insulation layer, solar panels, etc.) using a camera, and create texture maps using Reality Capture. Then, use photo blending technology to blend the textures with the original textures, thereby optimizing the rendering materials and enhancing the realism of the spatial imaging simulation.

[0083] (2) Analyze material properties: Carefully observe reference photos and real materials, paying attention to surface roughness, color variations, reflectivity, and any other unique characteristics or defects. Consider lighting conditions and viewing angles to understand material behavior under different conditions. Make subtle adjustments and optimizations by observing and comparing the rendered results with reference photos and real materials to achieve a more realistic effect.

[0084] (3) Deep learning model evaluation: The semantic segmentation model is trained with generated sample data, and its performance on the target domain test set is analyzed. The 3D model, material mapping and rendering lighting environment of the feature components are optimized and adjusted to make the features of the source domain and the target domain similar.

[0085] This paper employs DeepLab V3+, the state-of-the-art deep learning model in semantic segmentation, as the basic algorithm for intelligent feature information recognition. A randomly initialized neural network is trained on a simulated dataset generated by this invention, and evaluated on a validation set. Specifically, the training set consists of simulated images, and the validation set consists of real images. Since real images of space targets in orbit are extremely rare, images obtained from full physical simulation on the ground are used as an extension of the target domain. Various evaluation metrics are calculated based on the model's predicted segmentation results and the actual segmentation labels to measure model performance. Common evaluation metrics include: Mean Pixel Accuracy (MPI) and Mean Intersection over Union (IoU), which measure the degree of overlap between the model's predictions and the actual labels. The model is then improved and optimized based on its recognition performance on the aforementioned feature component database.

[0086] Implementation Cases

[0087] The method for constructing a complete dataset is illustrated using the EAGLE (ESPA Augmented Geostationary Laboratory Experiment) satellite as an example.

[0088] Step 1: First, after consulting the information, I learned that the EAGLE satellite is the first spacecraft based on Orbital ATK's ESPAStar platform. (The satellite promotional image is shown below.) Figure 10 As shown. This satellite can carry any combination of up to 6 fixed or 12 separable flight payloads in low Earth orbit and geostationary orbit. The satellite has a net weight of 430-470 kg, and the main body (cylindrical part) dimensions are: diameter 157.5 cm and height 61 cm. EAGLE carries four fixed experimental payloads and one deployable subsatellite. The subsatellite Mycroft is a microsatellite (70 kg) with a payload mass of approximately 30 kg, designed to test spacecraft self-testing technology. The satellite's dimensions are (56.6 × 56.6 × 70 cm). Based on the above dimensional information, the basic shape of the EAGLE satellite is created. Details are added according to the schematic diagram, and the satellite is assembled. The model is shown below. Figure 11 As shown. Unwrap the UVs of the model to prepare for texture mapping. Set the PBR material for each face, render and export the texture. Save the model as an "EAGLE.fbx" file and import it into UE4.

[0089] Step 2: Label the EAGLE satellite. The satellite consists of a main body, solar panels, solar panel supports, payload, rod-shaped antenna, thrusters, and docking ring. Semantic label each component. For example, select the four solar panels and assign them a 2#default material in 3ds Max; repeat this process for the remaining components. Export the file as 'EAGLE_label.fbx'. Create a material 02_-_default in Unreal Engine and edit it to a solid color (255, 0, 0). Import the 'EAGLE_label.fbx' file into the same folder; the material will be automatically matched. As a result, the solar panels of the labeled model will be red.

[0090] Step 3: Construct the UE4 scene, including spatial lighting construction: sunlight simulation, ambient lighting simulation, stray light simulation; spatial background construction: Earth background simulation and starry sky background simulation.

[0091] Step 4: Generate spatial target images. Assign the imported EAGLE model and EAGLE_label model to the "SAT" and "SAT_label" actors respectively. Run the UE4 program, and click "Single Star Image Generation" on the UMG interface to generate the spatial target images.

[0092] Step 5: Automatically Generate Multi-Type Annotations. In the generated spatial target image directory, a semantic segmentation mask image and a JSON file containing pose information, target detection boxes, and other annotation information will be automatically generated. The generated spatial target image and semantic segmentation labels are as follows: Figure 12a , Figure 12b As shown.

[0093] Step Six: Optimization and Enhancement. Observing the generated EAGLE image, an overly bright surface on the front of the sailboard was observed. The base color map and roughness map in the PBR material of the sailboard section were modified, reducing the material's reflectivity and increasing surface smoothness. The reflectivity of the components was adjusted based on the deep learning model's recognition results for different parts. New image data was then generated iteratively.

[0094] The above description is merely a specific embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements can be made or some technical features can be equivalently replaced without departing from the method of the present invention. These improvements and replacements should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a multi-mission dataset for spacecraft, characterized in that, Includes the following steps: Step 1: Establish a high-quality satellite model A three-dimensional model of the space target was constructed using the modeling software 3DS MAX. Polygon modeling was used to establish the satellite geometric model by referring to the satellite structure diagram, dimension diagram and schematic diagram. Step 2: Annotate the 3D model A method for automatic annotation of 2D images based on 3D models is used to automatically generate original images of spatial targets and semantically annotated images of feature components, and to construct a semantic segmentation dataset of key components of spatial targets. Step 3: Construct the rendering scene Create a spatial simulation scene based on Unreal Engine 4, including the construction of spatial lighting and background; Step 4: Generate spatial target image Import the satellite 3D model created in step one into the pre-built space scene using the model import tool built into Unreal Engine 4; then, use the scene capture component 2D to create a camera, which can capture the 3D scene and output it as a 2D image to generate satellite images; the images are then processed by Unreal Engine 4's render target function, which allows the rendering results to be converted into textures and stored at runtime. Step 5: Generate multi-type annotations Multiple annotation types include image classification labels, semantic segmentation masks, instance segmentation masks, depth maps, surface normal maps, and position and pose labels; these annotations cover all annotation types in computer vision tasks. The automatic annotation method for 2D images based on 3D models, in Unreal Engine 4, achieves the synchronous export of space target simulation images and annotated images by conjugating the original satellite model and the annotated satellite model. Step Six: Optimization and Enhancement In on-orbit simulation of space imaging, the surface material texture of space targets has the greatest impact on the rendering effect. The satellite surface material consists of the following parts: heat insulation layer, solar panels, back of solar panels, metal surface, and non-metallic paint surface. By conducting detailed analysis and optimization of the above materials, the realism of the rendering is improved. In step one, the satellite is first modeled to create its approximate shape and structure. The satellite components—including the main body, antenna, solar panels, high-resolution lens, star sensor, thrusters, and docking ring—are meticulously constructed according to dimensions, and then assembled according to the satellite structure diagram. For satellites of general shape, the origin of the coordinate system is adjusted to the center of the main body, the y-axis is aligned with the deployment direction of the solar panels, and the z-axis is aligned as closely as possible with the orientation of the satellite camera or antenna. For satellites of special shape, the coordinate axes must be aligned with the working direction of the satellite's critical payloads. This ensures that the pose information generated in subsequent steps is accurate and consistent. In step one, the model is UV unwrapped to ensure that the textures are correctly mapped onto the surface of the model; then it is wrapped back onto the 3D model; a physically based rendering (PBR) material map is created based on the model's UVs, including a base color map, normal map, roughness map, and metallicity map; the PBR rendering pipeline makes the lighting rendering more realistic; the model and material map are imported into Unreal Engine 4, a material sphere is created, and the material map is wrapped back onto the 3D model; In step two, the 3D model needs to be labeled, that is, the feature parts are marked with different colored materials, so that the type of feature part to which each pixel in the image belongs can be determined by the characteristics exhibited by different materials during planar rendering.

2. The method for constructing a multi-mission dataset for spacecraft according to claim 1, characterized in that: In step three, the spatial lighting construction involves the simulation of sunlight, ambient light, and stray light. Sunlight is simulated using directional light sources from Unreal Engine 4. The spatial background construction includes the simulation of the Earth and starry sky backgrounds. A blueprint of the Earth is created using high-resolution images from NASA, featuring customizable day / night cycles and atmospheric flow. A rendering scene is set up to support the satellite model. A cube environment map is created using high dynamic range (HDRI) images and attached to the sky light source to construct ambient light. An HDRI starry sky image is attached to the celestial sphere object to construct the starry sky background.

3. The method for constructing a multi-mission dataset for spacecraft according to claim 1, characterized in that: In step four, the settings include adjusting the color gamut of the capture source, hidden actors, gamma values, motion blur, and post-processing volume, as well as manually setting exposure parameters to simulate the imaging effects caused by ambient light and satellite motion. To enhance the realism of the image, an OpenCV plugin is introduced to post-process the generated image. OpenCV is integrated into the UE4 project as a plugin, and post-processing of the color array is implemented by writing C++ functions and Blueprint interfaces to improve image generation efficiency. Gaussian blur and random noise are used to enhance the realism of the image.

4. A method for constructing a multi-mission dataset for spacecraft according to claim 1 or 3, characterized in that: In step four, the Unreal Engine 4's Unreal Graphics Interface Designer (UMG) is used to create a user interface to control and display the simulated space environment. This interface includes a real-time running interface, raw and annotated image display windows under the camera viewport, and multiple function buttons, including screenshot, model replacement, attitude adjustment, and lighting adjustment. The screenshot function is implemented through a custom C++ function, which converts the rendered target into a PNG file for saving. The model replacement function allows users to select different satellite models and automatically replace them to improve the automation of batch image generation. Finally, the on-orbit attitude of the satellite is simulated, including setting the pose, linear velocity, and angular velocity of the satellite model to achieve motion simulation and motion blur effects.

5. The method for constructing a multi-mission dataset for spacecraft according to claim 1, characterized in that: In step five, the first step involves importing the original satellite model and its key component annotation models into UE4, and creating two static mesh scene components in the simulation environment to support these models; these models are placed on the designed satellite orbit, and the position and attitude of the annotation models completely overlap with the original satellite model in the scene; To capture images of both models simultaneously, two cameras with identical configurations were used, differing only in their rendering settings. One camera was used to capture the original satellite image, with the capture source set to sRGB final colors, displaying all scene components except for the labeled satellite model. The other camera was used to capture the labeled image, with the capture source set to RGB basic colors, masking all scene components except for the labeled satellite model. Two rendering targets were used to combine the satellite images generated by the two cameras. In addition, a depth map and a normal map are generated; the depth map is obtained by calculating the distance between the spatial point corresponding to each pixel and the camera optical center, with the capture source set to the scene depth in R; the normal map captures the surface normal direction information of each pixel in the scene, with the capture source set to RGB normals; these images are generated by setting the camera capture source and rendering target.

6. A method for constructing a multi-mission dataset for spacecraft according to claim 1 or 5, characterized in that: In step five, the generation of pose information is implemented based on UE4 blueprints and C++ classes, and is finally saved as a JSON file. The pose information of the satellite and its key components is obtained by defining a coordinate system and using blueprint functions, and then the C++ functions are used to store this information in JSON format. In the JSON file, the pose information is presented in a nested structure, clearly listing the pose information of different components, and also including additional information such as illumination angle, Earth position, image size, and field of view.

7. The method for constructing a multi-mission dataset for spacecraft according to claim 1, characterized in that: In step six, the rendering materials of the spatial targets are optimized using photo blending technology. First, the materials of the spatial targets are photographed using a camera, and texture maps are created using Reality Capture. Then, photo blending technology is used to blend the textures with the original textures to optimize the rendering materials and enhance the realism of the spatial imaging simulation. The material properties are analyzed, and the reference photos and real materials are carefully observed to adjust the surface roughness, color changes, and reflectivity of the materials. Deep learning model evaluation: The semantic segmentation model is trained using generated sample data, and its performance on the target domain test set is analyzed to measure the degree of overlap between the model's predictions and the real labels; improvements and optimizations are made based on the recognition performance of the model on the feature component database.

Images