A method and system for constructing a spacecraft simulation data set in a space environment

Abstract

The application discloses a kind of space environment under spacecraft simulation dataset construction method and system, the application is based on Unity3D's space environment under spacecraft simulation dataset SpaceSim-Dataset, propose a kind of space environment under Unity3D construction in the spacecraft simulation dataset SpaceSim-Dataset in the fly-around observation, approach inspection task scene.It can be in high simulation space scene, continuously output the color picture (RGB) and depth map (Depth) sequence that virtual camera is photographed, and recorded gyroscope and accelerometer IMU (Inertial Measurement Unit) sequence data, for realizing the SLAM (Simultaneous Localization and Mapping) task of space robot autonomous perception lay foundation.

Description

Technical Field

[0001] This invention belongs to the field of spacecraft simulation technology, and relates to a method and system for constructing a spacecraft simulation dataset in a space environment. Background Technology

[0002] Spacecraft image datasets in the space environment are a prerequisite for on-orbit service perception tasks. The performance of machine learning and computer vision methods also highly depends on the availability and quality of the data used to train them. However, in the space domain, acquiring space images is expensive and time-consuming, and such data is extremely scarce and often considered sensitive. Although existing spacecraft image datasets have addressed various space scenario tasks, current technologies still have the following shortcomings and limitations:

[0003] 1) The use of optical facilities and image synthesis technology requires a high cost and long cycle to build a physical simulation environment;

[0004] 2) It has significant limitations for perception tasks such as pose estimation and part recognition based only on two-dimensional RGB images;

[0005] 3) Single-frame, discontinuous, and incomplete observations also cannot establish an effective non-cooperative target model perception.

[0006] In conclusion, it remains necessary to study how to build simple and efficient 3D spatial environments on simulators to simulate different aircraft and scenarios, test algorithm outputs, and create diverse datasets. This will facilitate the testing and evaluation of subsequent perception tasks such as pose estimation and model reconstruction, and reduce the time and economic costs of real hardware processes.

[0007] It is understood that, to date, there is no dataset suitable for SLAM (Simultaneous Localization and Mapping) tasks for establishing non-cooperative target model perception under continuous observation in three-dimensional space scenes. However, geometric model perception of non-cooperative targets in space is extremely important for complex on-orbit service tasks such as operation and capture. Summary of the Invention

[0008] The purpose of this invention is to solve the problems in the prior art and provide a method and system for constructing a spacecraft simulation dataset in a space environment.

[0009] To achieve the above objectives, the present invention employs the following technical solution:

[0010] In a first aspect, the present invention provides a method for constructing a spacecraft simulation dataset in a space environment, comprising the following steps:

[0011] Configure environmental simulations of Earth, the Sun, and space;

[0012] Build importable spacecraft models;

[0013] Control the motion of the service spacecraft to generate RGBD image data around the observed target;

[0014] Plot the true pose trajectory of the image data sequence and calculate its acceleration and angular velocity data;

[0015] Post-processing of RGBD and IMU data, along with the acquisition of sequence data from different spacecraft using various typical observation trajectories, was performed to construct a simulation dataset.

[0016] Secondly, the present invention provides a system for constructing a spacecraft simulation dataset in a space environment, comprising:

[0017] The simulation configuration module is used to configure environmental simulations of Earth, the Sun, and space.

[0018] The model building module is used to build importable spacecraft models;

[0019] The image creation module is used to control the motion of the service spacecraft and create RGBD image data around the observation target;

[0020] The first calculation module is used to draw the true pose trajectory of the image data sequence and solve its acceleration and angular velocity data.

[0021] The second calculation module is used to post-process RGBD and IMU data, and to collect sequence data of different spacecraft through various typical observation trajectories to construct a simulation dataset.

[0022] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0023] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] To obtain spatial simulation scenes and construct datasets simply and efficiently, this invention proposes a novel method and system for constructing the SpaceSim-Dataset dataset, generated using the Unity3D simulation environment. Unity3D's excellent graphics rendering capabilities include real-time lighting with optical lights and ray tracing, high-resolution texture model shaders, and physically based material reflections and shadows. Therefore, it can simulate some extreme lighting conditions, darkness, and dynamic Earth backgrounds in real-world spatial environments. SpaceSim also incorporates other features already present in previous datasets, such as high-texture, high-resolution Earth satellite images and solar light sources, providing a more realistic spatial environment for simulated visual sensors.

[0026] This invention uses an Earth model, scene lighting, and over 120 types of models with complex textures, including satellites, space shuttles, space stations, free-flying robots, rovers, and extravehicular activity astronauts, in Unity3D to build a space simulation environment for both target and service spacecraft. The invention also allows for customizing the pose of observation cameras for fly-around monitoring and approach checks. Finally, the invention includes continuously acquiring sequenced color and depth images of the observed object; providing the pose ground truth of the RGBD sequence; and designing a trajectory generation IMU simulation data tool, a TUM format conversion tool, and a ROS simulation time synchronization tool. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the method of the present invention.

[0029] Figure 2 This is a schematic diagram of the system of the present invention.

[0030] Figure 3 This is a schematic diagram of a high-resolution Earth model in Unity3D; where (a) is a nighttime illumination map of the Earth, and (b) is a base color texture map of the Earth.

[0031] Figure 4This is a schematic diagram of a spacecraft model for the SpaceSim-Dataset dataset; where (a) is a satellite: NASA-Aqua, (b) is a space shuttle: STS-135 Atlantis, (c) is a space station and module: International Space Station, (d) is a manned spacecraft: Boeing CST-100, (e) is a rover: Perseverance Mars rover, and (f) is an extravehicular activity astronaut.

[0032] Figure 5 This is a schematic diagram of the SpaceSim-Dataset simulation acquisition environment.

[0033] Figure 6 This is a schematic diagram of the SpaceSim-Dataset dataset; where (a) is the RGB image of the observed Federatsiya target spacecraft, and (b) is the Depth image of the observed Federatsiya target spacecraft. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0037] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0039] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0040] The present invention will now be described in further detail with reference to the accompanying drawings:

[0041] See Figure 1 This invention discloses a method for constructing a spacecraft simulation dataset in a space environment, comprising the following steps:

[0042] S1 provides environmental simulations of Earth, the Sun, and space;

[0043] S2 builds importable spacecraft models;

[0044] S3 controls the movement of the spacecraft and establishes RGBD image data around the observed target;

[0045] S4 plots the true pose trajectory of the image data sequence and calculates its acceleration and angular velocity data;

[0046] S5 performs post-processing on RGBD and IMU data, collects sequence data from different spacecraft through various typical observation trajectories, and constructs a simulation dataset.

[0047] like Figure 2 As shown in the figure, this invention discloses a system for constructing a spacecraft simulation dataset in a space environment, comprising:

[0048] The simulation configuration module is used to configure environmental simulations of Earth, the Sun, and space.

[0049] The model building module is used to build importable spacecraft models;

[0050] The image creation module is used to control the motion of the service spacecraft and create RGBD image data around the observation target;

[0051] The first calculation module is used to draw the true pose trajectory of the image data sequence and solve its acceleration and angular velocity data.

[0052] The second calculation module is used to post-process RGBD and IMU data, and to collect sequence data of different spacecraft through various typical observation trajectories to construct a simulation dataset.

[0053] Example

[0054] This embodiment discloses a method for constructing a spacecraft simulation dataset in a space environment, including the following steps:

[0055] Step 1, Configure simulations of Earth, Sun, and space environments:

[0056] Earth Model: Based on NASA's Visible-Earth and obtained from Unity-Assert, this model features a high-resolution, realistic Earth model comprised of an atmosphere renderer, planetary shaders, and cloud shaders. Its textures consist of a 64K base color map, a 32K normal map, a 32K specular mask map, a 32K cloud base color map, a 32K cloud normal map, and a 64K nighttime illumination map to simulate the effects of clouds, shadows, and external atmospheric scattering. A space background from the European Southern Observatory has also been added, such as... Figure 3 As shown.

[0057] Scene lighting: To give the scene the characteristics of a planetary surface, we first construct the sunlight of a point light source as the position of the world space light; then we construct the sunlight and nighttime Fresnel properties of the Earth's atmosphere so that the materials exhibit different reflection effects at different distances of reflection from the ocean and clouds; finally, we set the glow and Fresnel properties of the cloud shader to represent different lighting conditions and contrast.

[0058] Step 2: Create and convert a spacecraft model into FBX format for import using SolidWorks or Blender.

[0059] Target spacecraft: Represents non-cooperative (spacecraft without cooperative markers and unable to communicate or transmit attitude information) observable objects with complex textures. It includes 128 different 3D models of various spacecraft, such as satellites, space shuttles, space station modules, free-flying robots, manned spacecraft, rovers, and astronauts, collected and organized, including NASA-3D. Figure 4 As shown.

[0060] Step 3: By controlling the motion of the service spacecraft, RGBD image data is established around the observed target, such as... Figure 4 As shown, the image names generated by Unity3D are formatted as "Year Month Day Hour Minute Second + Number of Images + World Coordinate System Position (x, y, z) and Pose Quaternion True Value (q)". x ,q y ,q z,q w The dataset is ").

[0061] Service spacecraft: These represent observation objects equipped with visual cameras, which can be moved away from or closer to the target and have their orbital radius changed to acquire image data of the observed target. Both the observation object and the observed object can be placed at any location near the Earth in low Earth orbit to extend the observation range and angle.

[0062] Virtual Camera: Includes a pinhole camera and a depth camera with VGA (640×480) resolution, outputting RGBD images in PNG / JPG format at 1-30fps. It can also be set to expand to other resolutions such as HD (1280×720), WXGA (1366×768), FULL-HD (1920×1080), QHD (2560×1440), and 4K-UHD (3840×2160), or the encoding format of the color map and depth map.

[0063] Optical sensor noise: Due to the small sensor size, reflections, absorptions, and holes occur when the deep infrared array illuminates the spacecraft's surface material. Furthermore, spaceborne images are susceptible to significant noise during high dynamic range imaging. Therefore, Gaussian noise was added to the generated simulation images to simulate the noise effects found in real spaceborne images.

[0064] Step 4: By plotting the trajectory of the pose ground truth of the image sequence, interpolation and differentiation are performed to obtain 200Hz IMU acceleration and angular velocity data; then, RGBD+IMU data is read through ROS (Robot Operating System) and recorded into ROS-bag using ROS simulation time; Matlab is used to read and convert the dataset sequence into TUM format "timestamp + position + attitude quaternion"; various spacecraft are photographed and collected through other typical observation trajectories such as fly-around, fly-along, fly-by, and close approach, and the data is recorded as the Ground truth of the sequence.

[0065] like Figure 5 As shown, Figure 5 The observation spacecraft is the American Boeing CST-100 Starliner manned spacecraft, and the target spacecraft being observed is the Russian Energiya manned spacecraft Federatsiya "Eagle" model. The middle relative motion circular trajectory represents the observation path of the observation body with the observed object as the reference coordinate system, the upper straight line represents the absolute motion trajectory of the observation body with the Earth as the reference coordinate system, and the lower straight line represents the absolute motion trajectory of the observed object with the Earth as the reference coordinate system.

[0066] This invention provides a flexible data acquisition interface for SpaceSim, which can be applied to a wide range of sensing tasks and platforms to promote the development of space environment datasets. A detailed comparison of the SpaceSim-Dataset dataset with existing datasets is shown in Table 1.

[0067] Table 1 Comparison of spacecraft datasets in space environments

[0068]

[0069] This invention features 128 different 3D models of various spacecraft, including satellites, space shuttles, space station modules, free-flying robots, manned spacecraft, rovers, and astronauts.

[0070] like Figure 6 As shown, this invention features the ability to continuously acquire sequenced color images, depth images, IMU data, and pose ground truth values ​​of RGBD sequences of the observed object. Figure 6 The target spacecraft observed was the Russian Energiya manned spacecraft Federatsiya "Eagle" model.

[0071] A computer device is provided according to an embodiment of the present invention. This computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0072] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0073] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0074] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0075] The memory can be used to store the computer program and / or module, and the processor implements various functions of the computer device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0076] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0077] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a spacecraft simulation dataset in a space environment, characterized in that, Includes the following steps: Environment simulations of Earth, the Sun, and space are configured: The Earth model is based on NASA's Visible-Earth and includes an atmosphere renderer, planetary shader, and cloud shader. It features a 64K base color map, a 32K normal map, a 32K specular mask map, a 32K cloud base color map, a 32K cloud normal map, and a 64K nighttime illumination map. Simulations of cloud, shadow, and atmospheric scattering effects are provided, along with a space background from the European Southern Observatory. Point light sources are used to simulate sunlight as the world's space light source, and the illumination and nighttime Fresnel properties of Earth's atmosphere, as well as the glow and Fresnel properties of the cloud shader, are set. Build importable 3D models of spacecraft; The control service spacecraft moves within the low Earth orbit range, orbits and observes the target spacecraft, and acquires and generates RGBD image data; Gaussian noise is added to the generated simulation images to simulate the reflection, absorption, and hole effects generated when the deep infrared dot array illuminates the surface material of the spacecraft, as well as the noise impact of the onboard images under high dynamic range. Plot the true pose trajectory of the image data sequence, and obtain the 200Hz IMU acceleration and angular velocity data by interpolation and differentiation; After time synchronization and post-processing of RGBD and IMU data, the simulation time is recorded in ROS-bag format through the robot operating system ROS, and then converted into TUM format pose ground truth values ​​of "timestamp + position + attitude quaternion" through Matlab. Sequence data of different spacecraft are collected through on-orbit observation trajectories of fly-around, fly-along, fly-by, and close approach, and a simulation dataset for vision-inertial SLAM mission for non-cooperative targets in space is constructed.

2. The method for constructing a spacecraft simulation dataset in a space environment according to claim 1, characterized in that, The construction of an importable 3D spacecraft model specifically includes: creating a 3D spacecraft model using SolidWorks or Blender, and converting it to FBX format for import into the simulation environment.

3. The method for constructing a spacecraft simulation dataset in a space environment according to claim 1, characterized in that, The control service spacecraft operates within a low Earth orbit, orbiting and observing the target spacecraft and acquiring RGBD image data, specifically including: Set the image names generated by Unity3D to the format of "Year Month Day Hour Minute Second + Number of Images + World Coordinate System Position (x, y, z) and Pose Quaternion Values". , , , ”; Service spacecraft are observation objects equipped with visual cameras that can be moved away from or closer to the target and have their orbital radius changed in order to acquire image data of the observed target. The virtual camera includes a pinhole camera and a depth camera with VGA resolution, outputting RGBD images in PNG / JPG format at 1-30fps; or can be set to expand to HD, WXGA, FULL-HD, QHD, and 4K-UHD resolutions.

4. The method for constructing a spacecraft simulation dataset in a space environment according to claim 2, characterized in that, The constructed importable 3D spacecraft models include various types such as satellites, space shuttles, space station modules, free-flying robots, manned spacecraft, rovers, and extravehicular astronauts.

5. A system for constructing a spacecraft simulation dataset in a space environment, characterized in that, include: The simulation configuration module is used to configure the environmental simulation of Earth, the Sun, and space: The Earth model is based on NASA's Visible-Earth and includes an atmosphere renderer, planetary shader, and cloud shader. It configures a 64K base color map, a 32K normal map, a 32K specular mask map, a 32K cloud base color map, a 32K cloud normal map, and a 64K nighttime illumination map to simulate cloud, shadow, and atmospheric scattering effects, and adds a space background from the European Southern Observatory. Point light sources are used to simulate sunlight as the world's space light source, and the illumination and nighttime Fresnel properties of Earth's atmosphere, as well as the glow and Fresnel properties of the cloud shader, are set. The model building module is used to build 3D models of spacecraft that can be imported. The image creation module is used to control the movement of the service spacecraft in the low Earth orbit range, orbit and observe the target spacecraft and collect and generate RGBD image data; Gaussian noise is added to the generated simulation images to simulate the reflection, absorption and hole effects generated when the deep infrared array illuminates the surface material of the spacecraft, as well as the noise impact of the spaceborne images under high dynamic range. The first calculation module is used to draw the true pose trajectory of the image data sequence and obtain the 200Hz IMU acceleration and angular velocity data through interpolation and differentiation. The second calculation module is used to perform time synchronization post-processing on RGBD and IMU data. The simulation time is recorded in ROS-bag format through the robot operating system ROS, and then converted into TUM format pose ground truth values ​​with "timestamp + position + attitude quaternion" through Matlab. Sequence data of different spacecraft are collected through on-orbit observation trajectories of fly-around, fly-along, fly-by, and close approach to construct a simulation dataset for visual-inertial SLAM missions for non-cooperative targets in space.

6. The system for constructing spacecraft simulation datasets in a space environment according to claim 5, characterized in that, In the model building module, a three-dimensional model of the spacecraft is created using SolidWorks or Blender and converted into FBX format for import into the simulation environment.

7. The system for constructing spacecraft simulation datasets in a space environment according to claim 5, characterized in that, In the image creation module: Set the image names generated by Unity3D to the format of "Year Month Day Hour Minute Second + Number of Images + World Coordinate System Position (x, y, z) and Pose Quaternion Values". , , , ”; Service spacecraft are observation objects equipped with visual cameras that can be moved away from or closer to the target and have their orbital radius changed in order to acquire image data of the observed target. The virtual camera includes a pinhole camera and a depth camera with VGA resolution, outputting RGBD images in PNG / JPG format at 1-30fps; or can be set to expand to HD, WXGA, FULL-HD, QHD, and 4K-UHD resolutions.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images