Spacecraft-oriented multi-task data set construction method

Abstract

The invention provides a spacecraft-oriented multi-task data set construction method. A high-precision satellite geometric model is created. The image rendered based on the PBR not only has a bright-eye visual effect, but also conforms to a physical optics principle, and particularly, a normal mapping simulates a star body wrinkle effect. A space simulation scene is constructed by using an unreal engine 4, and a vivid rendering effect is provided. A high-quality satellite image is generated by combining a scene capture component and a target rendering function of an unreal engine, and a semantic segmentation tag, an instance segmentation tag, a depth tag, a normal tag and a pose tag are automatically generated based on a 3D model 2D image automatic labeling method. And a large-scale data set can be quickly generated in a frame-by-frame rendering and automatic labeling mode. Semantic segmentation is carried out on the generated image through the deep learning model DeepLab V3 +, the accuracy and efficiency of model prediction are further improved, and the rendering trueness is greatly improved through detailed analysis and optimization.

Description

[Technical field]

[0001] The present invention belongs to the technical field of intelligent recognition of space targets, and in particular to a simulation data set required for intelligent recognition, and specifically is a method for constructing a multi-task image data set for spacecraft based on an Unreal Engine. [Background technology]

[0002] As countries around the world continue to pay attention to the space environment, the importance of space situational awareness is increasing. Attacking, protecting and on-orbit servicing important space targets, mainly satellites, has become an important development direction of aerospace technology in countries around the world. In addition, the number of space debris and abandoned spacecraft is also increasing, and space missions such as space debris removal and on-orbit servicing have become important development directions for aerospace. Satellite intelligent recognition technology is a key link, and feature recognition technology such as sails and antennas and position and attitude estimation technology are very important. Non-cooperative target spacecraft are not equipped with identifiable markers, so it is very important to accurately obtain the position and attitude information of the target spacecraft relative to the service satellite. In recent years, artificial intelligence algorithms represented by deep learning have flourished. Key technologies such as target detection, semantic segmentation, and pose estimation have been successfully applied to many fields, which has brought new technological breakthroughs to space target recognition.

[0003] However, due to the lack of space target datasets for training, there are relatively few studies on applying deep learning to intelligent recognition of spacecraft. Some scholars simulate satellites orbiting the Earth based on software such as STK, and capture pictures to establish satellite datasets. However, due to the limited modeling quality of satellites and the Earth, the constructed satellite pictures are quite different from real pictures. Simulated images often do not have the background of the Earth, and the lighting conditions are relatively simple. They lack the inherent visual features and lighting changes of space-based camera images, and the number of models is small. In addition, simulated images lack fine annotations, and the cost of manual annotation is high, which cannot be applied to large-scale training. The deep learning algorithm trained with such simulated pictures is difficult to apply to the recognition of real satellites in orbit. Therefore, how to construct a space target dataset with high-quality and high-fidelity images suitable for a variety of visual tasks and an automatic annotation method for simulated images has become a research focus.

[0004] Unreal Engine is a powerful game development engine with very realistic scene development capabilities. Therefore, space scenes can be built based on Unreal Engine software, satellite models can be imported, and automatically annotated satellite data sets can be generated in batches, providing data support for the subsequent recognition of spacecraft feature components based on deep learning methods. [Summary of the invention]

[0005] (I) Purpose of the invention

[0006] The present invention overcomes the shortcomings of existing satellite data sets, such as unrealistic lighting and small number, and provides a method for constructing a multi-task data set for spacecraft, providing large-scale, multi-type annotation to automatically generate satellite images with a high similarity to actual in-orbit photography for intelligent identification of spacecraft feature components.

[0007] (II) Technical solution of the present invention

[0008] Step 1: Build high-quality satellite modeling

[0009] The three-dimensional model of the space target is constructed based on the modeling software 3DS MAX. The polygonal modeling method is used to establish a detailed satellite geometric model with reference to the satellite structure diagram, dimension diagram and schematic diagram. First, the basic modeling of the satellite is carried out to create the general shape and structure of the object. Satellite components such as the satellite body, antenna, solar sail, high-resolution lens, star sensor, thruster, docking ring, etc. are established strictly according to the size, and the components are spliced ​​together according to the satellite structure diagram. For satellites of general shape, the coordinate origin is adjusted to the center of the satellite body, the coordinate y-axis is aligned with the direction of the satellite sail deployment, and the coordinate z-axis is consistent with the direction of the satellite camera or antenna as much as possible. For satellites with special shapes, try to ensure that the coordinate axis is aligned with the working direction of the satellite's key payload. Ensure that the pose information generated in subsequent steps is accurate and unified.

[0010] UV unwrap the model to ensure that the texture is correctly mapped on the surface of the model. Then wrap it back to the three-dimensional model. Based on the UV of the model, physically based rendering (PBR) material maps are produced, including base color maps, normal maps, roughness maps, and metalness maps. The PBR rendering pipeline makes lighting rendering more realistic. The model and material maps are imported into the Unreal Engine, a material ball is produced, and the material map is wrapped back to the three-dimensional model. The present invention has established a total of 300 satellites of different types and converted the models into Unreal assets for backup.

[0011] Step 2: Label the 3D model

[0012] The automatic annotation method of 2D images based on 3D models automatically generates the original image of space targets and the images annotated with semantic information of characteristic components, and constructs a semantic segmentation dataset of key components of space targets. This method requires annotating the 3D model, that is, marking characteristic components with materials of different colors, so that the type of characteristic component to which each pixel in the image belongs can be determined by the characteristics of different materials during plane rendering.

[0013] Step 3: Build the rendering scene

[0014] In the present invention, a space simulation scene is established based on Unreal Engine 4, including the construction of space illumination and background. The construction of space illumination involves the simulation of sunlight, ambient illumination and stray light, and the directional light source of Unreal Engine simulates sunlight. The construction of space background includes the simulation of the earth and the starry sky background. The earth blueprint is made from NASA's high-definition images, with functions such as self-selection, day and night change, and atmospheric flow. A rendering scene is set to carry a satellite model. A cube environment map is made from a high dynamic range imaging (HDRI) image and attached to a sky light source to construct ambient light. The HDRI starry sky image is attached to a celestial sphere object to construct a starry sky background.

[0015] Step 4: Generate space target image

[0016] The satellite 3D model (Obj or Fbx format) produced in step 1 is imported into the pre-established space scene through the model import tool provided by Unreal Engine 4. Next, a camera is created using the "Scene Capture Component 2D", which can capture the 3D scene and output it as a 2D image to generate satellite images. These images are then processed through the "render target" function of Unreal Engine, which allows the rendering results to be converted into textures and store various image information when the program is running. These settings include the adjustment of the color gamut of the capture source, hidden actors, gamma values, dynamic blur and post-processing volumes, as well as the manual setting of exposure parameters to simulate the imaging effects caused by space ambient light and satellite movement. In order to enhance the realism of the image, the OpenCV plug-in is introduced to post-process the generated image. OpenCV is integrated into the UE4 project as a plug-in, and the post-processing of the color array is realized by writing C++ functions and blueprint interfaces, thereby improving the efficiency of image generation. Gaussian blur and random noise are used to enhance the realism of the image.

[0017] In addition, the Unreal Graph Designer (UMG) in the Unreal Engine is used to create a user interface to control and display the simulated space environment. The interface includes a real-time running interface, a display window for original and annotated images under the camera viewport, and multiple function buttons, such as screenshots, model replacement, attitude adjustment, and lighting adjustment. The screenshot function is implemented through a custom C++ function, and the rendering target is transcoded into a png file for saving. The model replacement function allows users to select different satellite models and automatically replace them to improve the automation of batch output. Finally, the satellite's in-orbit attitude is simulated, including setting the satellite model's position, linear velocity, and angular velocity to achieve motion simulation and motion blur effects.

[0018] Step 5: Generate multi-type annotations

[0019] Multi-type annotations include image classification labels, semantic segmentation masks, instance segmentation masks, depth maps, surface normal maps, and position and posture labels. These annotations cover all annotation types in computer vision tasks. The present invention proposes an automatic annotation method for 2D images based on 3D models. In Unreal Engine, we achieve the simultaneous export of simulated images and annotated images of space targets by conjugating the original satellite model and the annotated satellite model. The process is divided into several key steps. First, it involves importing the annotated models of the original satellite model and its key components into UE4, and creating two static mesh scene components in the simulation environment to carry these models. These models are placed on the designed satellite orbit, and the position and posture of the annotated model and the original satellite model in the scene completely overlap.

[0020] In order to capture images of the two models at the same time, two cameras with the same configuration are used, the difference lies in their rendering settings. One is used to capture the original satellite image, set the capture source to sRGB final color, and display all scene components except the marked satellite model; the other is used to capture the annotated image, set the capture source to RGB basic color, and mask all scene components except the marked satellite model. Two rendering targets are used to receive the satellite images generated by the two cameras.

[0021] In addition, the system also generates depth maps and normal maps. The depth map is obtained by calculating the distance between the spatial point corresponding to each pixel and the camera optical center, and the capture source is set to "scene depth in R"; the normal map captures the surface normal direction information of each pixel in the scene, and the capture source is set to "RGB normal". These images are generated by setting specific camera capture sources and rendering targets.

[0022] The generation of pose information is based on UE4's blueprint and C++ class, and is ultimately saved as a JSON file. The pose information of the satellite and its key components is obtained by defining the coordinate system and using the blueprint function, and then the C++ function is used to store this information in JSON format. In the JSON file, the pose information is presented in a nested structure, which clearly lists the pose information of different components, and also includes additional information such as illumination angle, earth position, image size, and field of view.

[0023] In general, this process automatically generates 2D images containing rich spatial information through UE4's advanced functions and visualization tools, and effectively annotates and stores information for subsequent use and analysis.

[0024] Step 6: Optimization and enhancement

[0025] In the on-orbit equivalent simulation of space imaging, the surface material texture of the space target has the greatest impact on the rendering effect. The surface material of the satellite can be summarized as follows: thermal insulation layer, solar panels, back of the panels, metal surface, and non-metallic paint surface. The realism of the rendering is improved by carefully analyzing and optimizing the above materials. The following aspects are mainly optimized: Use photo blending technology to optimize the rendering material of the space target. First, take pictures of the main materials of the space target (thermal insulation layer, solar panels, etc.) with a camera, and use Reality Capture to make texture maps. Use photo blending technology to merge with the original texture to optimize the rendering material and enhance the realism of the space imaging simulation. Analyze the material properties, carefully observe the reference photos and real materials, and adjust the surface roughness, color change, and reflectivity of the material. Deep learning model evaluation: Use the generated sample data to train the semantic segmentation model, analyze its performance on the target domain test set, and measure the overlap between the model prediction and the real label. Improve and optimize the model based on the recognition effect of the model on the feature component database constructed above.

[0026] (III) Advantages and effects of the present invention

[0027] The present invention mainly involves the steps of building, labeling, scene construction, image generation and optimization and enhancement of high-quality satellite models to improve the simulation realism and effect of space imaging. The following are the main advantages and effects of the present invention: a high-precision satellite geometric model is created. The unified model coordinate axis and origin adopt the PBR material system to produce realistic material maps. The image rendered based on PBR not only has a dazzling visual effect but also conforms to the principle of physical optics. In particular, the star body wrinkle effect is simulated by the normal map. The space simulation scene is constructed using Unreal Engine 4, including space illumination, background and global illumination, etc., to provide realistic rendering effects. Combined with the scene capture component and rendering target function of Unreal Engine, high-quality satellite images are generated. The 2D image automatic labeling method based on the 3D model automatically generates semantic segmentation labels, instance segmentation labels, depth labels, normal labels and pose labels. The frame-by-frame rendering and automatic labeling method can quickly generate large-scale data sets. The generated images are semantically segmented by the deep learning model DeepLab V3+, which further improves the accuracy and efficiency of model prediction, and the rendering realism is greatly improved through careful analysis and optimization.

[0028] In general, the present invention effectively improves the authenticity and effect of space imaging simulation by comprehensively using advanced modeling technology, rendering technology and deep learning methods, and provides strong technical support for the research and application of intelligent recognition of space targets.

Brief Description of the Drawings

[0029] Figure 1 It is a flow chart of the present invention.

[0030] Figure 2a , Figure 2b , Figure 2c , Figure 2d , Figure 2e , Figure 2f , Figure 2g , Figure 2h , Figure 2i Examples of some satellite models built for the present invention.

[0031] Figure 3a , Figure 3b A schematic diagram showing the comparison between the satellite model and the annotation model established in the present invention.

[0032] Figure 4 This is the rendering effect of the space simulation scene established by the present invention.

[0033] Figure 5 A user interface is established for the present invention.

[0034] Figure 6 The flowchart of the batch drawing program established for the present invention.

[0035] Figure 7 A schematic diagram of the motion blur effect of a space target image constructed by the present invention.

[0036] Figure 8a , Figure 8b This is an automatic annotation effect diagram of a 2D image based on a 3D model of the present invention.

[0037] Fig. 9 The present invention is a flowchart for optimizing and enhancing the image rendering effect.

[0038] Fig.10 This is a promotional image for the EAGLE satellite.

[0039] Fig.11 This is the three-dimensional model of the EAGLE satellite.

[0040] Fig.12a , Figure 12b The spatial target images and semantic segmentation annotations generated by the present invention. [Specific implementation method]

[0041] The following is combined with Figure 1 -Attached Figure 12b The technical solution of the present invention is further described in detail.

[0042] The present invention proposes a flow chart of a method for constructing a multi-task data set for spacecraft, such as Figure 1 shown.

[0043] Step 1: Create satellite geometry model

[0044] The construction of high-quality satellite models includes satellite 3D models and physically based rendering (PBR) material maps. The model construction process includes 3D modeling, UV mapping, texture drawing, texture mapping, rendering testing and other steps, which will be introduced below. Figure 2a-2i Examples of some satellite models built for the present invention. Figure 2a It is an Orion electronic reconnaissance satellite with a large dish antenna for signal reception. Figure 2b It is the Gaofen-13 remote sensing satellite. Figure 2c It is the Phobos 2 remote sensing satellite, the main body of which is a hexagonal prism and is equipped with four solar panels. Figure 2d It is a GSSAP reconnaissance satellite. Figure 2e It is the EAGLE satellite, which has a cylindrical main body, a single solar wing, and contains multiple payloads to form a mother-daughter satellite. Figure 2f-Figure 2i They are TDRS13, Telstar12V, Tianlian-1, and Sirius XM8. These satellites are all communication satellites, all of which adopt a dual solar wing design and have circular communication antennas.

[0045] High-quality 3D modeling: Based on the modeling software 3DS MAX, the 3D model of the space target is constructed. The polygonal modeling method is used to establish a fine satellite geometric model with reference to the satellite structure diagram, dimension diagram and schematic diagram. Before modeling, the relevant material information required for satellite modeling is collected and sorted out by consulting the official website or authoritative website, such as appearance, size, surface material, payload type and other related information. For the material collection of the same satellite, we consult the introduction of the satellite on different websites, verify and supplement the satellite-related material information, and try to improve the authenticity and accuracy of the source of satellite material information. The 3D model of the satellite in the data is restored by polygonal modeling of 3dmax software. First, the satellite is modeled in the basic way to create the approximate shape and structure of the object. Satellite components such as satellite body, antenna, solar sail, high-resolution lens, star sensor, thruster, docking ring, etc. are established with reference to the satellite envelope size (length, width and height) and the proportional relationship of the components, and the components are spliced ​​together according to the satellite structure diagram. To create the initial shape of satellite components, it is usually started with basic geometric bodies (such as cubes, spheres, cylinders, etc.). In polygonal modeling, the model is composed of vertices, edges and faces. Vertices are the corner points of a polygon, edges connect vertices to form the outline of a polygon, and faces are composed of multiple edges, representing the surface of an object.

[0046] Next, use subdivision modeling to add more details and refine the shape and features of the model. Extrusion is one of the common operations that generates new geometry on existing faces to create complex shapes. Subdivision adds more vertices and faces to make the model more refined. The shape of the model can be adjusted by moving, rotating, and scaling the vertices, edges, and faces one by one. The smoothing tool can remove the abrupt parts of the surface to make the surface of the model smoother.

[0047] In addition, for repeated components such as solar panels, you can use the array function to copy polyhedrons to avoid repeated modeling. For special-shaped components such as antennas, use spline curves to assist in modeling to improve the precision of the model. At the same time, you need to pay attention to the modeling of the connection mechanism to ensure the basic mechanical function of the satellite model. For small components such as screws, you can simplify the processing without affecting the appearance. Set up model layers and manage grouping and naming for easy selection.

[0048] Different models often have different coordinate axes and origins when they are constructed, which is not conducive to batch control of model movement and sample generation. Therefore, the model origin and coordinate axes need to be adjusted to be consistent. Specifically, for satellites of general shape (square satellite body, dual solar wings), the coordinate origin is adjusted to the center of the satellite body, the coordinate y-axis is aligned with the direction of the satellite sailboard, and the coordinate z-axis is aligned with the direction of the satellite camera or antenna as much as possible. For satellites with special shapes (such as Figure 2a ), try to ensure that the coordinate axis is aligned with the working direction of the satellite's key payload. Ensure that the pose information generated in subsequent steps is accurate and unified.

[0049] UV Unwrapping: UV unwrap the model to prepare for texture mapping. UV processing of a 3D model is the process of mapping a 2D texture onto a 3D model. First, mark the gaps and cut the gaps on the model. These gaps define how the model is cut when converted from 3D to 2D to determine how the model is unfolded on a 2D plane. Secondly, unfold the UVs and use the UV unwrapping tool of the 3D software to unfold the model into a 2D plane. Adjust the position and size of the UV islands to maximize the efficiency of the texture space and ensure that important detail areas have enough space on the UV map. Make sure that all parts of the model have consistent texture density and avoid some areas being too large or too small. Finally, map the completed texture back to the 3D model and check the effect of the texture in the 3D software to ensure that the texture is correctly mapped on the surface of the model. Make final adjustments and optimizations to the UV map and texture as needed. Export the UV layout as an image file for use in the texture making software.

[0050] Physically Based Rendering (PBR) Texture Drawing: PBR material mapping is made based on the UV of the model. PBR material refers to a material system based on physical information rendering. It is widely used for its more realistic rendering images and compatibility and interpretability that helps industrial production. Based on the information of the reference materials, the software Substance Painter is used to draw the texture materials of each part of the satellite, including the thermal insulation layer, solar sails and other parts. PBR maps include base color maps, normal maps, roughness maps and metal maps. Different maps play different roles, and finally combine to make the material of each part. The base color map defines the color and some basic details of the material, and the color and texture of real-world objects are made. The metal map defines the metal properties of the material. It is used to distinguish the refraction and reflection methods between metal materials and non-metal materials (also known as dielectrics). Materials with high metalness will have stronger specular highlights. The roughness map defines the surface roughness of the material and affects the scattering of light. Low roughness values ​​(smoother surfaces) produce sharp specular highlights, while high roughness values ​​(rougher surfaces) produce blurry highlights. Normal maps are used to simulate small details and textures on the surface of a material without adding geometric detail. These maps change the direction in which light is reflected, thus simulating a bumpy effect.

[0051] UV texture mapping: Use UV coordinates to ensure that the texture is correctly mapped to the model surface. Set appropriate lighting and rendering parameters, perform test renderings, and check how the model and material perform under different lighting conditions. Export the drawn textures in the format of tga or jpg files. When exporting the drawn textures, keep the size of each texture consistent to facilitate rendering in other software. Keep the exported textures for future use.

[0052] Import Unreal Engine Rendering Test: Export the satellite model to FBX format in 3DS MAX software. Compared with the widely used .obj format, FBX format can automatically import material maps after importing the model, which is more efficient in batch processing. FBX import follows the following settings: Set the incoming model as a static mesh instead of a skeletal mesh, automatically generate collisions; check Build adjacent buffers to use PNT tessellation; uncheck Remove degeneration to retain the degenerate triangle mesh found; check Build reverse index buffers; check Generate lightmap UV; check each vertex; check Merge meshes so that the imported model will not be saved as separate parts for easy management; check Transform vertices to absolute; in the Normal Import option, select Import normals instead of Calculated normals, and guide the model normal storage path to calculate weighted normals; in the Material Import option, select Local Search Model Material and Create a New Material Ball, so as to automatically import the model and its material, and store them as static meshes and corresponding material balls respectively, for easy management. Each material map (base color map, roughness map, specular map, metalness map, normal map) will be automatically imported into the material ball. After importing, the satellite model is converted into an Unreal asset ueset and can be called in the blueprint. The base color map defines the color and some basic details of the material, and the color and texture of real-world objects are produced.

[0053] Step 2: Annotate the 3D model

[0054] An automatic annotation method for two-dimensional images based on three-dimensional models automatically generates original images of space targets and images annotated with semantic information of characteristic components, and constructs a semantic segmentation dataset of key components of space targets.

[0055] The automatic annotation method of two-dimensional images based on three-dimensional models first needs to annotate the three-dimensional model and use different materials to mark the characteristic components, so that when rendering the plane, the type of characteristic component to which each pixel in the image belongs can be determined by the characteristics of different materials. For three-dimensional models constructed using 3DS MAX software, relying on the layer function of the software, we can easily filter out and classify the mesh bodies represented by each characteristic component, and assign different solid color materials. Specifically, first use the "Group" function in the layer explorer to group the components into characteristic components, and then create a solid color material in the streamlined material editor by modifying the self-luminous color and opacity, and assign different materials to each characteristic component according to the corresponding annotation color table of the characteristic component. Taking the GSSAP satellite model as an example, the rendering effects of the original model and the annotated model in 3DS MAX software are as follows Figure 3a , Figure 3b shown.

[0056] After the model is labeled, import the original model of the space target and the labeled model into the Unreal Engine according to the model import method in step 1, and convert them into Unreal assets for use. Create a new material ball in the folder, set its color according to the color label correspondence table, name it the same as the material in the material editor of 3DS MAX, and form a one-to-one correspondence with the label. After importing into the Unreal Engine, the model will automatically search for materials to achieve material ball reuse.

[0057] Step 3: Build the UE4 scene

[0058] Unreal Engine 4 is a powerful game development engine, so the present invention builds a space simulation scene based on Unreal Engine 4. It mainly includes: space lighting construction: sunlight simulation, ambient lighting simulation, stray light simulation; space background construction: earth background simulation and starry sky background simulation.

[0059] Space lighting construction: Unreal Engine provides a variety of light sources including directional light sources, point light sources, spot light sources, and rectangular light sources. Sunlight is high-intensity uniform parallel light. Directional Lights are the main outdoor light sources that simulate light sources that emit light from extremely far or nearly infinite distances. Since there is no atmosphere in orbital space, high-intensity sunlight shines directly on the spacecraft. Directional light sources use directional normal illumination, expressed as lux, which is equal to one lumen per square meter. It represents the illumination on the earth produced by the visible part of direct solar radiation on a surface perpendicular to (or orthogonal to) the sun's rays. It represents the total amount of illumination falling on a surface. Set the directional light intensity to 150,000 lux.

[0060] Ambient light refers to indirect light formed by reflection, scattering and refraction of light sources in the scene. In space scenes, it mainly includes reflected light from the earth and the moon, and starlight. Apply HDRI images to the ambient lighting model in computer graphics rendering. First, you need to obtain an HDRI image for building ambient lighting. These images are usually taken by professional photographers using special photography techniques, or they can be obtained from an HDRI image library. Apply the HDRI image to the ambient lighting model of the scene. In UE4, HDRI images are mapped to a cube (called an environment map), and the map provides lighting information of ambient light for the scene. Apply the cube environment map to the sky light source. Finally, you can adjust the intensity, color, and other properties of the ambient light. Using HDRI to build ambient lighting can more accurately simulate the lighting effects in the real world and provide more realistic rendering results.

[0061] Satellite rendering scene: Customize a "static mesh" class to hold satellite models. Multiple instances can be added to the scene to hold multiple satellite models. Create two instances of this class in the scene, named "SAT" and "SAT_label". Set the collision to pure physics, check the simulation physics option, and disable gravity and impulse. Check the cast shadows under the lighting tab.

[0062] Add a "Lightmass Importance Volume" module to the satellite model. This module is a box that will ensure that the area inside the box gets higher lighting quality, improving lighting performance. The performance of global illumination depends on the size of the scene, and lighting build time will also increase. The global illumination importance volume controls the area where global illumination emits photons, focusing it on areas that need clear indirect lighting. Areas outside the importance volume will only get one bounce of indirect lighting at a lower quality.

[0063] Earth background construction. First, create the planet body, that is, the sphere, and adjust the size of the sphere according to its actual size ratio. Then collect and make textures, including surface terrain, surface environment, night lights, atmosphere, and aurora. The material texture is based on NASA's high-definition image data, and the color and tone of the texture are adjusted according to the true color photos taken. Secondly, establish material functions and make material balls, using some texture function structure relationships; establish the earth blueprint, set the static mesh as a sphere, set each material ball and add events. Design functions such as sunlight direction, earth rotation speed, day and night changes, and atmospheric flow in the blueprint.

[0064] Considering that space target mission simulation is mainly concentrated in the Earth satellite orbit, higher requirements are placed on Earth modeling. To ensure that the Earth images collected at various perspectives and distances have high resolution and restoration, we use more refined materials and combine physical information to build a blueprint of the Earth.

[0065] Construction of the starry sky background. First, collect and make a high dynamic range imaging (HDRI) starry sky image. It is a 360-degree panoramic image that provides an all-round view of the surrounding environment. Then, create a material for the HDRI image. In the Material Editor, connect the HDRI image to the Emissive color node. Make sure the smoothness and metalness of the material are set to 0 to better simulate the look of the background. Add a Sky Sphere object to the scene. Apply the created HDRI material to the celestial sphere. Make sure the material is correctly mapped to the celestial sphere to show the complete starry sky background. Adjust the size, position, and rotation of the celestial sphere as needed to ensure that the background looks natural and fits the perspective of the scene. The rendering effect is as follows Figure 4 shown.

[0066] Step 4: Generate space target image

[0067] Import the space satellite model created in step one into the space scene created in step three. You can import Obj or Fbx format files through the model import tool provided by Unreal 4.

[0068] The camera created by the "Scene Capture Component 2D" class is used to take satellite images. It can capture 3D scenes and output them as 2D images. The "render target" function of the Unreal Engine is used to receive the satellite images generated by the camera for subsequent processing. The render target is a texture in the Unreal Engine that can be written when the program is running. It can store information such as basic color, normal, and ambient occlusion, allowing users to convert the engine's rendering results into material textures for further editing, rendering or storage operations. Specifically, create a render target to store satellite images generated by the camera, and edit the render target to set the appropriate resolution and format. In the UE4 editor, add the "Scene Capture Component 2D" to the scene to obtain a camera instance. Set the created render target to "Texture Target", set the capture source to the linear sRGB color space final color, set the "Hidden Actor" to the annotation model, adjust the gamma value to simulate the space environment light, set the dynamic blur to simulate the imaging degradation caused by the satellite movement, set the post-processing volume, and manually set the exposure parameters. With the above settings, satellite images rendered in the space lighting environment can be presented in the render target.

[0069] In order to achieve a more realistic rendering effect, we introduced the OpenCV plug-in in UE4 to perform post-processing enhancement on the generated image. Integrate OpenCV as a plug-in for the UE4 project, and use the header file of OpenCV to call its functions and classes. Write a C++ function and expose the interface to the blueprint, and call the OpenCV function to post-process the generated color array. This method effectively avoids repeated reading and writing of image files on the hard disk and improves the efficiency of image generation. The specific method is: first write the conversion functions ConvertTArrayToCvMat and ConvertCvMatToTArray to convert the color array obtained by the rendering target into an array in OpenCV format. Use the GaussianBlur function in the OpenCV library to add Gaussian noise and random noise to the image.

[0070] The batch drawing program and user interface are designed based on the Unreal Motion Graphics Interface Designer in Unreal Engine. Unreal Motion Graphics UIDesigner (UMG) is a visual UI authoring tool that can be used to create UI elements such as HUD, menus or other interface-related graphics in a project. The core of UMG is controls, which are a series of pre-made functions that can be used to build interfaces (such as buttons, check boxes, sliders, progress bars, etc.). These controls are edited in a dedicated control blueprint, which is constructed using two tabs: the designer tab allows for the visual layout of the interface and basic functions, while the diagram tab provides the functionality behind the controls used. The constructed user interface is like Figure 5 As shown in the figure, the right part of the interface is the real-time operation interface of the simulation space environment. The two windows on the upper part of the interface are the original image of the space target under the camera viewport and the annotated image. On the left part of the interface, there are function buttons such as screenshot, replace model, adjust posture, change background, adjust lens, output posture, batch output, etc. The structure of the simulation image batch generation module is shown in the figure. Figure 6 As shown. First, select the satellite model according to the pre-set list. Second, perform satellite attitude simulation, capture images with the camera, post-process and save the images. Then, traverse the satellite attitude according to the rules and generate images in a loop. Finally, traverse the model list until all images are generated. The following describes the implementation methods of the functions one by one.

[0071] The screenshot function is used to capture the image captured from the specified camera perspective and export it for saving. Since the Unreal Engine does not directly provide an export interface for the camera render target, we independently designed the screenshot saving function through the C++ function provided by the Unreal Engine, and transcoded the specified render target into a .png image file and saved it in a local directory. Among them, the program imports the render target texture material input by the program by calling the render target source class FTextureRenderTargetResource, and uses the class method ReadPixels() to transfer the texture to the color array by pixel to form a bitmap, and then transcodes the bitmap into a png file format and outputs it to the local for saving.

[0072] The replacement model function enables users to independently select satellite models in the space target model database in the user interface and replace static meshes in the simulated space scene. It also provides an interface for automatic model replacement to improve the degree of automation in the batch output function.

[0073] The "Get All Actors of Classes with Labels" function is used to search for a specified Actor in all objects of the project according to the Actor type and label. The label is set in the Actor Details option. It is set to "sat" for the original satellite model and "sat_label" for the labeled satellite model. Two pipelines are set up in the program to replace the original model and the labeled model synchronously. Here, the Actor Class option of the "Get All Actors of Classes with Labels" function selects "Static Mesh", and the labels are entered as "sat" and "sat_label" respectively. Next, use the "Set Static Mesh" function to specify the satellite model for the selected Actor. Add references to all satellite models in an array, use the "get" function to get the static mesh reference of the satellite model and connect it to the "Set Static Mesh" function. The user can select the satellite model to replace by entering the model number on the keyboard.

[0074] Simulate the satellite's in-orbit posture. For motion simulation tasks, Unreal Engine can provide the 6D pose of the center position of the satellite's 3D model. By "getting all Actors of the labeled class", the target model in the simulation space environment is located. In conjunction with "setting the Actor's spatial position" and "setting the Actor's rotation", the 6D pose of the center point of the target model can be set. In batch output, the three Euler angles of the satellite model are traversed, and each axis is divided into 15 parts. In each interval, "random floating point numbers within the range" are used for randomization. In step three, the physical simulation of the satellite model has been turned on. In conjunction with "setting physical linear speed" and "setting physical angular speed", the linear speed and angular speed of the target model can be set by the blueprint. When there is a relative speed between satellites, motion blur can be simulated. The motion blur effect is as follows Figure 7 shown.

[0075] Step 5: Automatically generate multiple types of annotations

[0076] According to the automatic annotation method of 2D images based on 3D models, in Unreal Engine, we achieve the simultaneous export of simulated images and annotated images of space targets by conjugating the original satellite model and the annotated satellite model. First, import all the original satellite models and the models of their annotated key components into Unreal Engine for standby use, create two static mesh scene components to receive the models, take two models of a certain type of satellite and embed them into the mesh components, and place the models on the designed simulated satellite orbit. Attach the annotated model to the satellite model, set the pose transformation of the annotated model relative to the satellite model to zero, and the poses of the two models in the scene completely overlap. Figure 8a , Figure 8b shown.

[0077] In order to facilitate the simultaneous imaging of the two models and save the pictures, two configured cameras are used to capture the two models respectively. The poses of the two cameras are also kept overlapping, and the camera parameter configurations are also the same. The difference between the two is the rendering settings. The camera is constructed using the method in step 4. Specifically, two rendering targets are used to receive the satellite images generated by the two cameras, and the rendering targets are edited. For the original satellite image, the capture source is set to sRGB final color, and all scene components except the marked satellite model are displayed; for the marked satellite image, the capture source is set to RGB basic color, all scene components except the marked satellite model are shielded, all lighting is turned off, and other settings are consistent with the first rendering target. The display effect is the key component annotation result of the original satellite image. With the above settings, the original satellite image rendered with normal light and shadow and the key component annotation image annotated with RGB primary colors can be presented in the two rendering targets, and the two rendering targets can be packaged into the rendering target texture for standby.

[0078] The depth map refers to the image mapped by the distance from the camera optical center to the spatial point corresponding to each pixel in the generated image, which contains spatial information. The normal map captures the surface normal direction information of each pixel in the scene, usually represented in the form of an RGB image. We make depth maps based on the real physical system of Unreal Engine, and calculate them by the absolute distance between the center of the satellite model and the optical center of the camera, as well as the relative spatial position distribution of each component of the satellite model. Surface normal information is usually achieved by capturing the normal vector of each pixel and mapping it to the RGB color space. Specifically, build two cameras similar to step 4, set the capture source to "RGB normal" and "scene depth in R". Set the rendering target to receive the images obtained by the two cameras. Use the "ReadPixels" function to read pixel data from the rendering target, which can be used to generate normal maps and depth maps. The normal map is output as an RBG image. The original depth data is usually nonlinear, and the range and accuracy depend on the configuration of the depth buffer. Normalize the depth value, adjust the range of the depth value, and output the processed depth data in the form of a grayscale image.

[0079] The pose information generation is based on UE4 blueprint and C++ class implementation, and saved in JSON file. For the pose information of key components, the coordinate system is defined for key components such as optical payloads, and the pose information of the satellite and its key components in the space scene is obtained by using the blueprint function "Get Actor Transformation". The pose information of the satellite relative to the camera is calculated by "Invert Transformation Rotation". The "json.h" library is introduced in the C++ function, and the pose information is stored in JSON format using the "jsonWriter" class. A nested structure is constructed in the JSON file to clearly list the pose information of different components. In addition, the function reads and stores information such as illumination angle, earth position, image size, field of view, etc. The pose information file can be generated by exposing the function interface in the blueprint, referencing it in the output program, and passing parameters such as relative pose to the function.

[0080] Step 6: Optimization and enhancement

[0081] In the on-orbit equivalent simulation of space imaging, the surface material texture of the space target has the greatest impact on the rendering effect. The surface material of the satellite can be summarized as follows: thermal insulation layer, solar panels, back of the panels, metal surface, and non-metallic paint surface. By carefully analyzing and optimizing the above materials, the realism of the rendering is improved. The optimization process is as follows: Fig.10 As shown, the main optimization steps are as follows:

[0082] (1) Use photo blending technology to optimize the rendering materials of space targets. First, take photos of the main materials of space targets (thermal insulation, solar panels, etc.) with a camera and use Reality Capture to create texture maps. Then use photo blending technology to merge with the original textures to optimize the rendering materials and enhance the realism of space imaging simulation.

[0083] (2) Analyze material properties: Carefully observe reference photos and real materials, paying attention to the material's surface roughness, color variations, reflectivity, and any other unique features or defects. Pay attention to lighting conditions and viewing angles to understand how the material behaves under different conditions. By observing and comparing the rendering results with reference photos and real materials, make subtle adjustments and optimizations to achieve more realistic results.

[0084] (3) Deep learning model evaluation: Use the generated sample data to train the semantic segmentation model, analyze its performance on the target domain test set, and optimize the 3D model, material map, and rendering lighting environment of the feature components to make the source domain and target domain features similar.

[0085] DeepLab V3+, the deep learning model with the best performance in the current semantic segmentation field, is used as the basic algorithm for intelligent recognition of feature information. A randomly initialized neural network is trained on the simulated data set generated by the present invention, and evaluated on the validation set. That is, the training set is a simulated image, and the validation set is a real image. Since the real images of space targets on orbit are extremely rare, the images obtained by full physical simulation on the ground are used as an expansion of the target domain. According to the segmentation results predicted by the model and the real segmentation labels, various evaluation indicators are calculated to measure the performance of the model. Common evaluation indicators include: Mean Pixel Accuracy (Mean Pixel Accuracy), Mean IoU (Intersection over Union). Measure the degree of overlap between the model prediction and the real label. Improve and optimize the model based on the recognition effect of the model on the feature component database constructed above.

[0086] Implementation Cases

[0087] The EAGLE (ESPA Augmented Geostationary Laboratory Experiment) satellite is used as an example to illustrate the complete construction method of the dataset.

[0088] Step 1: First, check the information and find out that the EAGLE satellite is the first spacecraft based on the ESPAStar platform of Orbital ATK. The satellite promotional picture is as follows Fig.10 As shown. The satellite can accommodate any combination of up to 6 fixed or 12 detachable flight payloads in low orbit and geosynchronous orbit. The net weight of the satellite is 430-470kg, and the dimensions of the main body (cylindrical part) are: 157.5cm in diameter and 61cm in height. EAGLE carries four fixed experimental payloads and one deployable sub-satellite. The sub-satellite Mycroft is a microsatellite (70kg) with a payload mass of approximately 30kg, designed to test spacecraft self-inspection technology. The dimensions of the satellite are (56.6×56.6×70cm). The basic shape of the EAGLE satellite is made from the above size information, details are added according to the schematic diagram and the satellite is assembled. The model made is as follows Fig.11 As shown. UV unwrap the model to prepare for texture mapping. Set the PBR material for each face separately, render and export the map. Save the model as an "EAGLE.fbx" file and import it into UE4.

[0089] Step 2: Label the EAGLE satellite. The satellite consists of a main body, sails, sail brackets, payloads, rod antennas, thrusters, and docking rings. Semantically label each component separately. For example, select four sails and assign them the 2#default material ball in 3dMAX, and do the same for the remaining components. Export as the 'EAGLE_label.fbx' file. Create the material ball 02_-_default in Unreal Engine and edit it to a solid color (255, 0, 0). Import the 'EAGLE_label.fbx' file in the same folder and automatically match the material ball. As a result, the sails of the labeled model are red.

[0090] Step 3: Build the UE4 scene, including space lighting construction: sunlight simulation, ambient lighting simulation, stray light simulation; space background construction: earth background simulation and starry sky background simulation.

[0091] Step 4: Generate space target image. Set the imported EAGLE model and EAGLE_label model to the two Actors "SAT" and "SAT_label" respectively. Run the UE4 program and click "Single Star Image" on the UMG interface to generate the space target image.

[0092] Step 5: Automatically generate multiple types of annotations. In the generated spatial target image directory, a semantic segmentation mask image and a JSON file containing pose information, target detection box and other annotation information will be automatically generated. The generated spatial target image and semantic segmentation label are as follows: Fig.12a , Figure 12b shown.

[0093] Step 6: Optimization and enhancement. Observe the generated EAGLE image and find that the front of the sailboard is too bright. Modify the base color map and roughness map in the PBR material of the sailboard part, lower the reflectivity of the material, and increase the surface smoothness. Adjust the reflectivity of the components based on the recognition results of different components by the deep learning model. Generate new image data iteratively.

[0094] The above description is only a specific implementation mode of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements can be made, or some of the technical features can be replaced by equivalents without departing from the method of the present invention. These improvements and replacements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for constructing a multi-task dataset for spacecraft, characterized in that: The steps include: Step 1: Build high-quality satellite modeling The three-dimensional model of the space target is constructed based on the modeling software 3DS MAX. The satellite geometric model is established by using polygonal modeling and referring to the satellite structure diagram, dimension diagram and schematic diagram. Step 2: Label the 3D model The automatic annotation method of 2D images based on 3D models can automatically generate the original image of space targets and the images annotated with semantic information of characteristic components, and build a semantic segmentation dataset of key components of space targets; Step 3: Build the rendering scene Build a space simulation scene based on Unreal Engine 4, including the construction of space lighting and background; Step 4: Generate space target image The satellite 3D model produced in step 1 is imported into the pre-built space scene through the model import tool provided by Unreal Engine 4. Next, a camera is created using the Scene Capture Component 2D. The camera can capture the 3D scene and output it as a 2D image to generate satellite images. The image is then processed through the Render Target function of Unreal Engine 4, which allows the rendering results to be converted into textures to store various image information while the program is running. Step 5: Generate multi-type annotations Multi-type annotations include image classification labels, semantic segmentation masks, instance segmentation masks, depth maps, surface normal maps, and position and posture labels; these annotations cover all annotation types in computer vision tasks; The automatic annotation method of 2D images based on 3D models, in Unreal Engine 4, realizes the simultaneous export of space target simulation images and annotated images by conjugating the original satellite model and the annotated satellite model; Step 6: Optimization and Enhancement In the on-orbit simulation of space imaging, the surface material texture of the space target has the greatest impact on the rendering effect; the satellite surface material consists of the following parts: thermal insulation layer, solar panels, back of panels, metal surface, and non-metallic paint surface; by conducting detailed analysis and optimization of the above materials, the realism of the rendering is improved.

2. The method for constructing a multi-task dataset for a spacecraft according to claim 1, characterized in that: In step 1, first, perform basic modeling of the satellite to create the approximate shape and structure of the object; strictly build satellite components such as the satellite body, antenna, solar panels, high-resolution lens, star sensor, thruster, and docking ring according to the size, and splice the components according to the satellite structure diagram; for satellites of general form, adjust the coordinate origin to the center of the satellite body, align the coordinate y-axis with the direction of the satellite panel deployment, and try to align the coordinate z-axis with the direction of the satellite camera or antenna. For satellites of special form, ensure that the coordinate axis is aligned with the working direction of the satellite's key payload; Ensure that the pose information generated in subsequent steps is accurate and unified.

3. A method for constructing a multi-task dataset for a spacecraft according to claim 1 or 2, characterized in that: In step one, the model is UV unfolded to ensure that the texture is correctly mapped on the surface of the model; then it is wrapped back onto the 3D model; based on the model's UV, physically based rendering PBR material maps are made, including base color maps, normal maps, roughness maps, and metalness maps; the PBR rendering pipeline makes lighting rendering more realistic; the model and material maps are imported into Unreal Engine 4, a material ball is made, and the material map is wrapped back onto the 3D model.

4. The method for constructing a multi-task data set for a spacecraft according to claim 1, characterized in that: In step two, the three-dimensional model needs to be annotated, that is, the characteristic components are marked with materials of different colors, so that when the plane is rendered, the type of characteristic component to which each pixel in the image belongs can be determined by the characteristics of different materials.

5. The method for constructing a multi-task data set for a spacecraft according to claim 1, characterized in that: In step three, the construction of space lighting involves the simulation of sunlight, ambient lighting and stray light. The directional light source of Unreal Engine 4 simulates sunlight; the construction of space background includes the simulation of the earth and the starry sky background; the earth blueprint is made from NASA's high-definition images, with the functions of self-selection, day and night change, and atmospheric flow; the rendering scene is set up to carry the satellite model; the cube environment map is made from the high dynamic range imaging HDRI image, and attached to the sky light source to construct the ambient light; the HDRI starry sky image is attached to the celestial sphere object to construct the starry sky background.

6. The method for constructing a multi-task data set for a spacecraft according to claim 1, characterized in that: In step 4, the settings include adjusting the color gamut of the capture source, hidden Actors, gamma values, dynamic blur, and post-processing volumes, as well as manually setting exposure parameters to simulate the imaging effects caused by space ambient light and satellite motion. In order to enhance the realism of the image, an OpenCV plug-in is introduced to post-process the generated image. OpenCV is integrated into the UE4 project as a plug-in, and by writing C++ functions and blueprint interfaces, post-processing of the color array is implemented to improve image generation efficiency. Gaussian blur and random noise are used to enhance the realism of the image.

7. A method for constructing a multi-task dataset for a spacecraft according to claim 1 or 6, characterized in that: In step 4, the Unreal Graph Graphics Interface Designer UMG in Unreal Engine 4 is used to create a user interface to control and display the simulated space environment; the interface includes a real-time running interface, original and annotated image display windows under the camera viewport, and multiple function buttons, including screenshot, model replacement, posture adjustment, and lighting adjustment; the screenshot function is implemented through a custom C++ function, and the rendering target is transcoded into a png file for saving; the model replacement function allows users to select different satellite models and automatically replace them to improve the automation of batch output; finally, the satellite's on-orbit posture is simulated, including setting the satellite model's posture, linear velocity, and angular velocity to achieve motion simulation and motion blur effects.

8. The method for constructing a multi-task data set for a spacecraft according to claim 1, characterized in that: In step five, the original satellite model and the annotated models of its key components are first imported into UE4, and two static mesh scene components are created in the simulation environment to carry these models; these models are placed on the designed satellite orbit, and the position and posture of the annotated model and the original satellite model in the scene are completely overlapped; In order to capture images of the two models at the same time, two cameras with the same configuration were used, the difference being their rendering settings; one was used to capture the original satellite image, with the capture source set to sRGB final color, showing all scene components except the marked satellite model; the other was used to capture the annotated image, with the capture source set to RGB basic color, shielding all scene components except the marked satellite model; two rendering targets were used to receive the satellite images generated by the two cameras; In addition, depth maps and normal maps are also generated; the depth map is obtained by calculating the distance between the spatial point corresponding to each pixel and the camera optical center, and the capture source is set to the scene depth in R; the normal map captures the surface normal direction information of each pixel in the scene, and the capture source is set to RGB normal; these images are generated by setting the camera capture source and rendering target.

9. A method for constructing a multi-task dataset for a spacecraft according to claim 1 or 8, characterized in that: In step five, the generation of pose information is based on UE4's blueprint and C++ class, and finally saved as a JSON file; the pose information of the satellite and its key components is obtained by defining the coordinate system and using the blueprint function, and then the C++ function is used to store this information in JSON format; in the JSON file, the pose information is presented in a nested structure, which clearly lists the pose information of different components, and also includes additional information such as illumination angle, earth position, image size, and field of view.

10. The method for constructing a multi-task data set for a spacecraft according to claim 1, characterized in that: In step six, the rendering material of the space target is optimized using photo blending technology. First, the material of the space target is photographed with a camera, and texture maps are made using Reality Capture. The photo blending technology is then used to blend with the original texture to optimize the rendering material and enhance the realism of the space imaging simulation. The material properties are analyzed, and the reference photos and real materials are carefully observed to adjust the surface roughness, color change, and reflectivity of the material. Deep learning model evaluation: Use generated sample data to train the semantic segmentation model, analyze its performance on the target domain test set, and measure the overlap between the model prediction and the true label; improve and optimize the model based on its recognition effect on the feature component database.

Images