A spacecraft attitude interaction control method and device based on multi-dimensional perception

Abstract

The application provides a spacecraft attitude interaction control method based on multi-dimensional perception, wherein the method comprises the following steps: collecting user operation input signals and head posture data; mapping the operation input signals into control moments and thrust vector, and performing real-time calculation based on the dynamic parameters of the spacecraft to generate six-degree-of-freedom state quantities; mapping the head posture data to the camera control parameters of a visual rendering engine after filtering processing, so as to drive the simulation visual angle to change synchronously with the user head movement; returning the six-degree-of-freedom state quantities to the rendering engine, updating the visual picture, and realizing real-time interaction control based on the closed-loop feedback between the user input and the state quantities. The application can realize the spacecraft attitude interaction control with millisecond-level closed-loop feedback, significantly improve the intuitiveness and immersive fidelity of the operation and control, effectively relieve the motion sickness through head posture perception, and enhance the situation awareness ability of the user.

Description

Technical Field

[0001] This invention relates to the field of simulation interaction technology, and in particular to a spacecraft attitude interaction control method and device based on multi-dimensional perception. Background Technology

[0002] Spacecraft simulation, as an important technology for aerospace science popularization and professional training, is widely used in immersive experiences and operational skills training. Among related technologies, an interactive simulation environment is constructed through the collaborative operation of control devices, physics engines, and visual systems. Specifically, this system covers the entire process from user input acquisition and dynamics calculation to visual updates, including the coupling of input devices such as joysticks and touchscreens with underlying mechanical algorithms, as well as the linkage between attitude control and visual feedback.

[0003] However, existing simulation interaction methods directly use input signals to drive screen displacement without deeply coupling real-time mechanical mapping with a physics engine. This may result in a lack of physical realism in the operation feedback, or inconsistencies between visual and tactile feedback, thus affecting the user's immersion and the seriousness of the operation. In addition, traditional fixed field of view limits situational awareness capabilities, easily causing dizziness, and lacks real-time linkage between head posture and perspective, making it difficult to meet the natural interaction requirements of complex tasks. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] The main objective of this invention is to provide a spacecraft attitude interactive control method based on multi-dimensional perception.

[0006] Another objective of this invention is to propose a spacecraft attitude interaction control device based on multidimensional perception.

[0007] The third objective of this invention is to provide an electronic device.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a spacecraft attitude interactive control method based on multi-dimensional perception, comprising:

[0009] S1, collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust. The head posture data includes pitch angle, yaw angle and roll angle. S2, the operation input signal is mapped into control torque and thrust vector, and a six-degree-of-freedom state variable is generated based on the dynamic parameters of the spacecraft in real time. S3, the head posture data is filtered and then mapped to the camera control parameters of the visual rendering engine to drive the simulation viewpoint to change synchronously with the user's head movement; S4, the six-degree-of-freedom state variables are sent back to the rendering engine to update the visual scene, and real-time interactive control is achieved based on the closed-loop feedback between user input and state variables.

[0010] Optionally, the acquisition of user operation input signals and head posture data includes: By reading the USB digital interface signal of the Logitech X56 HOTAS flight stick through the PyGame library, the stick displacement and throttle input are resolved into four types of operation input signals: pitch, roll, yaw and throttle. The RetinaFace face detection algorithm is used to perform high-precision face localization and cropping on the video stream. The cropped face image is then input into the 6DRepNet neural network for head pose regression, and the output is raw head pose data including pitch angle, yaw angle and roll angle.

[0011] Optionally, it also includes: smoothing the raw head attitude data containing pitch, yaw and roll angles using an adaptive Kalman filter, and generating filtered head attitude Euler angle variables after filtering out high-frequency jitter noise.

[0012] Optionally, the step of mapping the operation input signal into control torque and thrust vector, and performing real-time calculations based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables includes: The operation input signal is injected into the underlying dynamic link library (DLL). The DLL converts the operation input signal into control torque and thrust vector based on the spacecraft's current mass, moment of inertia matrix, atmospheric density, and gravitational gradient. The DLL calculates the angular and linear acceleration of the spacecraft in the next integration step based on the orbital dynamics model, generating a six-degree-of-freedom state variable containing spatial position and attitude information.

[0013] Optionally, the step of converting the operation input signal into control torque and thrust vector includes: The user inputs the desired angular velocity via a joystick into the attitude kinematics PID control system based on the flywheel assembly. The Lyapunov control law calculates the error between the desired attitude quaternion and the actual attitude quaternion in real time, and automatically allocates the rotational speed of the virtual reaction flywheel based on the error to generate an auxiliary stabilizing torque.

[0014] Optionally, the step of mapping the head pose data to the camera control parameters of the visual rendering engine after filtering includes: directly mapping the filtered head pose Euler angle variables to the first-person camera control blueprint of the UE5 engine, so that the pitch, yaw and roll motions of the simulated viewpoint change synchronously with the pitch, yaw and roll motions of the user's head.

[0015] Optionally, the step of sending the six-degree-of-freedom state variables back to the rendering engine to update the view and achieving real-time interactive control based on the closed-loop feedback between user input and state variables includes: The six-degree-of-freedom state variables are fed back to the UE5 rendering engine to drive the synchronous update of the visual scene. The updated visual scene is presented to the user as visual feedback. Based on the visual feedback, the user adjusts the operation input signal, forming a millisecond-level latency closed-loop feedback of user input, signal reading, dynamic calculation, result feedback, and visual scene update.

[0016] To achieve the above objectives, a second aspect of the present invention provides a spacecraft attitude interaction control device based on multi-dimensional perception, comprising: The first module is used to collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust, and the head posture data includes pitch angle, yaw angle and roll angle. The second module is used to map the operation input signal into control torque and thrust vector, and perform real-time calculation based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables; The third module is used to map the head posture data to the camera control parameters of the visual rendering engine after filtering, so as to drive the simulation viewpoint to change synchronously with the user's head movement. The fourth module is used to send the six-degree-of-freedom state variables back to the rendering engine, update the visual scene, and realize real-time interactive control based on the closed-loop feedback between user input and state variables.

[0017] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory to implement the method described in the first aspect.

[0018] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0019] The embodiments of the present invention have the following beneficial effects: they can realize spacecraft attitude interactive control with millisecond-level closed-loop feedback, significantly improve the intuitiveness and immersive realism of operation, and effectively alleviate motion sickness through head attitude perception, thereby enhancing the user's situational awareness. Attached Figure Description

[0020] The above-described and additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a spacecraft attitude interactive control method based on multi-dimensional perception provided for an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] The following description, with reference to the accompanying drawings, describes a spacecraft attitude interactive control method and apparatus based on multidimensional perception, according to an embodiment of the present invention.

[0024] Example 1 Figure 1 This is a flowchart of a spacecraft attitude interactive control method based on multi-dimensional perception, according to an embodiment of the present invention.

[0025] like Figure 1 As shown, the spacecraft attitude interaction control method based on multi-dimensional perception includes the following steps: S1, collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust. The head posture data includes pitch angle, yaw angle and roll angle.

[0026] This method first senses and collects user actions on interactive devices in real time to obtain raw input signals that characterize the user's control intentions. These input signals specifically include displacement signals generated by joystick-like devices and propulsion signals generated by throttle-like devices; these signals reflect the user's direct control commands to the spacecraft's attitude and propulsion systems.

[0027] Simultaneously, this method also collects the user's head posture data. This data uses visual perception technology to detect the spatial rotation state of the user's head in real time, specifically including angle information representing head orientation such as pitch angle, yaw angle, and roll angle. The above-mentioned operation input signals and head posture data together constitute a multi-dimensional source of user input information, providing basic data for subsequent mechanical mapping and visual synchronization.

[0028] As one implementation method, the operation input signal can be collected from the flight joystick and throttle controller through a high-precision Hall sensor, while the head posture data can be obtained in real time from the video stream by means of face detection and head posture regression neural network, thereby realizing a comprehensive perception of the user's operation intention and visual attention.

[0029] By synchronously acquiring operation input signals and head posture data, this method can integrate the user's physical operations and visual behavior into the interactive control system, providing complete and real-time input data for subsequent mechanical calculations and visual rendering, thereby significantly improving the comprehensiveness and immersion of interactive control.

[0030] S2, the operation input signal is mapped into control torque and thrust vector, and a six-degree-of-freedom state variable is generated based on the spacecraft's dynamic parameters in real time.

[0031] In the interactive control method, the acquired operation input signals are mapped into mechanical forces acting on the spacecraft model, and real-time calculations are performed based on the spacecraft's own dynamic parameters to generate six-degree-of-freedom state variables characterizing its motion state.

[0032] Specifically, this step first converts the electrical signals input by the user through the control device, such as the displacement of a joystick or the thrust of the throttle, into control torques and thrust vectors that can drive changes in the spacecraft's dynamic model. This conversion process is not a simple linear mapping, but rather based on a preset mechanical mapping algorithm that correlates the input signals with the force characteristics of the spacecraft under specific environmental conditions.

[0033] Subsequently, the system acquires the spacecraft's dynamic parameters, which include at least the spacecraft's mass, moment of inertia matrix, and key physical quantities affecting its motion response, such as atmospheric density and gravitational gradient of its environment. Based on the aforementioned control torque, thrust vector, and dynamic parameters, the system calls the underlying dynamics calculation module, such as an algorithm encapsulated in a dynamic link library, to perform real-time iterative calculations of the spacecraft's motion state within the next integration step. This calculation process follows the basic principles of rigid body dynamics and orbital dynamics, enabling the calculation of the spacecraft's angular and linear accelerations in three-dimensional space, and further integration to obtain its complete six-degree-of-freedom state variables, including position, attitude, velocity, and angular velocity.

[0034] As one implementation method, the solution process can be executed by a dynamic link library based on the C++ language. This library receives the mapped control torque and thrust vector through the data input interface, performs the solution using the internal differential iteration calculation module in combination with parameters such as the spacecraft's mass and moment of inertia matrix, and finally outputs the six-degree-of-freedom state variables containing spatial position, attitude information, and relative velocity and distance in real time through the data output interface.

[0035] By mapping operational input signals to mechanical forces and combining them with spacecraft dynamic parameters for real-time calculation, this method can accurately simulate the motion response of spacecraft in real physical environments. This establishes a causal relationship between the user's operation and the spacecraft's state changes that conforms to physical laws, thereby significantly improving the realism and seriousness of interactive control.

[0036] S3, the head posture data is filtered and then mapped to the camera control parameters of the visual rendering engine to drive the simulation viewpoint to change synchronously with the user's head movement.

[0037] In the interactive control method, the collected head posture data is processed and used to drive the synchronous change of the simulation viewpoint. Specifically, the filtered head posture data is mapped to the camera control parameters of the visual rendering engine.

[0038] This step aims to establish a real-time correspondence between the user's head spatial motion and the virtual scene's viewing perspective, enabling the user to change their viewing orientation in the simulation environment through natural head rotation. To achieve this, the user's three-dimensional head posture information is first acquired through a visual perception module. This information is typically represented in Euler angles, including components such as pitch, yaw, and roll.

[0039] Because the raw head pose data may contain high-frequency noise caused by user physiological tremors or environmental interference, it needs to be filtered to extract smooth and stable pose change trends. Adaptive filtering algorithms, such as Kalman filtering or its variants, can be used to suppress noise while maintaining a fast response to real head movements. The filtered head pose data is then converted into input parameters for the camera control system in the visual rendering engine. These parameters directly or indirectly control the orientation of the virtual camera in three-dimensional space. Through this mapping, when the user's head pitches, yaws, or rolls, the viewing angle in the simulation scene will change accordingly, thus achieving an immersive interactive effect of "view-driven movement."

[0040] In one implementation, the process can locate and track the user's facial features in real time using a face detection algorithm, and then use a neural network model to regress the Euler angles of the head pose. After the raw data is smoothed by an adaptive Kalman filter, its Euler angle variables are directly mapped to the first-person camera control blueprint of the rendering engine, driving the viewpoint to change synchronously with the head movement.

[0041] This technical step directly links head posture data with camera control parameters, enabling real-time synchronization between the user's natural head movements and the simulated perspective. This significantly improves the intuitiveness and immersion of the interaction, while effectively alleviating motion sickness caused by the inconsistency between vision and proprioception, and enhancing the user's situational awareness and sense of presence in the virtual environment.

[0042] S4, the six-degree-of-freedom state variables are sent back to the rendering engine to update the visual scene, and real-time interactive control is achieved based on the closed-loop feedback between user input and state variables.

[0043] In the method described above, the calculated six-degree-of-freedom state variables are fed back to the rendering engine to update the visual scene, and real-time interactive control is achieved based on the closed-loop feedback between user input and state variables. This is a key step in ensuring deep coupling between the control response and the physical simulation.

[0044] This step aims to feed back the six-degree-of-freedom state information of the spacecraft in space, such as position, attitude, velocity, and angular velocity, obtained after dynamic calculation, to the visual rendering engine in a low-latency manner. This drives the pose update of the spacecraft model in the virtual scene, allowing users to perceive the physical effects of their actions through continuous changes in the visual scene.

[0045] Specifically, after receiving the returned state variables, the rendering engine recalculates and redraws the current frame's visual image based on these variables, ensuring that the spacecraft's trajectory and attitude changes in the virtual environment strictly follow the dynamic calculation results. Simultaneously, this step achieves real-time and continuous interactive control by constructing a closed-loop feedback mechanism between user input signals and state variables: each user input is converted into a change in state variables, and these updates are presented to the user through visual image refreshes. The user then adjusts subsequent operations accordingly, forming an iterative cycle of "input-solution-feedback-re-input." This closed-loop feedback process is completed within millisecond timescales, ensuring that the user can instantly perceive the impact of their actions on the spacecraft's motion, such as the inertial slip effect due to the conservation of angular momentum or the acceleration response when thrust counteracts gravity.

[0046] In one implementation, this step can be achieved by sending the six-degree-of-freedom state variables calculated by the dynamic link library back to the Unreal Engine-based real-time visual update module via a data interface. This module can then call the state variables to drive the position and attitude updates of the spacecraft model and simultaneously refresh the first-person camera view, thereby achieving a high degree of consistency between the visual image and the physical simulation state.

[0047] This step establishes a closed-loop interactive link between user operation and physical simulation by transmitting the dynamic solution results back in real time and driving the visual update. This significantly improves the real-time performance and physical consistency of the control response, enabling users to obtain an immersive experience similar to the control of a real spacecraft.

[0048] Example 2 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S1, "collecting user operation input signals and head attitude data, wherein the operation input signals include joystick displacement and throttle thrust, and the head attitude data includes pitch angle, yaw angle, and roll angle," in the spacecraft attitude interactive control method based on multi-dimensional perception.

[0049] In this embodiment, the process of collecting user operation input signals and head posture data in step S1 is specifically implemented in the following way.

[0050] For acquiring input signals, the system reads the USB digital interface signals of the Logitech X56 HOTAS flight stick through the PyGame library. This flight stick has a built-in high-precision Hall sensor that accurately acquires input signals such as stick displacement and knob angle, and outputs them as digital data packets via the USB digital interface.

[0051] The PyGame library, a multimedia and input device processing library in the Python environment, can directly access the underlying device communication protocols of the operating system and read the raw data stream output from the USB interface in real time. The read raw data is parsed and divided into four types of operational input signals: pitch, roll, yaw, and throttle. The pitch signal corresponds to the forward and backward movement of the joystick, the roll signal to the left and right movement of the joystick, the yaw signal to the torsional movement of the joystick or rudder input, and the throttle signal to the thrust of the throttle lever. These four types of operational input signals are used as input data for the mechanical calculations in the subsequent step S2.

[0052] For head pose data acquisition, the system employs a sequential visual algorithm architecture of "detect first, estimate later." First, the RetinaFace face detection algorithm performs high-precision face localization and cropping on the video stream. The RetinaFace model is a single-stage face detector based on deep learning, capable of quickly detecting face regions in each frame of the video stream and outputting accurate face bounding boxes. Based on this, the face region is cropped from the background, generating a sub-image containing only the face. Subsequently, this cropped face image is input into the 6DRepNet neural network for head pose regression. 6DRepNet is a neural network model for estimating continuous six-degree-of-freedom head pose. Its input is the cropped face image, and its output is raw head pose data including pitch, yaw, and roll angles. This raw data is in unfiltered Euler angle form, reflecting the instantaneous rotation angle of the user's head in three-dimensional space.

[0053] Furthermore, to improve the smoothness and stability of the data, the system smooths the raw head pose data using an adaptive Kalman filter. The adaptive Kalman filter dynamically adjusts the filtering parameters based on the noise characteristics of the observed data, effectively filtering out high-frequency jitter noise introduced by minor head movements or image acquisition noise, thereby generating filtered head pose Euler angle variables. These filtered Euler angle variables serve as input data for driving the camera control parameters of the visual rendering engine in subsequent step S3. Through the above process, step S1 achieves accurate conversion from physical input devices to digital signals and stable extraction from visual images to pose parameters, providing a reliable data foundation for subsequent mechanical calculations and visual synchronization.

[0054] This specific implementation method, by introducing the PyGame library and dedicated algorithm modules such as RetinaFace and 6DRepNet, achieves high-precision, low-latency acquisition and analysis of joystick input signals and head posture data. At the same time, it utilizes adaptive Kalman filtering to effectively suppress noise interference in head posture data, significantly improving the stability and reliability of input data, and providing high-quality data support for subsequent real-time interactive control.

[0055] Example 3 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S2 in the spacecraft attitude interaction control method based on multi-dimensional perception: "mapping the operation input signal into control torque and thrust vector, and performing real-time calculation based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables".

[0056] In this embodiment, the process of mapping the operation input signal into control torque and thrust vector in step S2, and performing real-time calculation based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables, is implemented through a low-level dynamic link library (DLL).

[0057] Specifically, the flight joystick data acquisition module, using the PyGame library in Python and the SimpleController plugin on the UE5 platform, collects and standardizes four types of signals—pitch, roll, yaw, and throttle—generated by the user's joystick inputs, and then injects them into the DLL as input. This DLL is written in C++ and includes a data input interface, an internal differential iteration calculation module, and a data output interface.

[0058] Upon receiving the operational input signal, the DLL first reads the spacecraft's current mass, moment of inertia matrix, atmospheric density, and gravitational gradient, and then converts the operational input signal into specific control torque and thrust vector. In one possible implementation, this conversion process further includes: the user inputs not a direct torque, but a desired angular velocity via a joystick. This desired angular velocity is input to the attitude kinematics PID control system based on the flywheel assembly. The system calculates the error between the desired attitude quaternion and the actual attitude quaternion in real time using a Lyapunov control law, and automatically allocates the rotational speed of the virtual reaction flywheel based on this error, thereby generating an auxiliary stabilizing torque, which ultimately forms the control torque acting on the spacecraft.

[0059] Subsequently, based on the orbital dynamics model, the DLL uses the transformed control torque and thrust vector, combined with the spacecraft's dynamic parameters, to calculate the spacecraft's angular and linear acceleration within the next integration step. This is then solved using an internal differential iterative calculation module, ultimately generating a six-degree-of-freedom state variable containing spatial position and attitude information. This state variable is transmitted back to the real-time visual update module via a data output interface, providing the data foundation for the synchronous update of the visual image. Through this millisecond-level closed-loop feedback—the complete link of user input, signal reading, dynamics calculation, result transmission, and visual update—deep coupling between the operational input signal and the spacecraft's dynamics model is achieved.

[0060] This specific implementation introduces a low-level dynamics DLL and a flywheel-based PID control system to accurately convert user operation signals into spacecraft control torque and thrust vectors that conform to physical laws, and calculates and generates six-degree-of-freedom state variables in real time. This significantly improves the physical realism and response accuracy of interactive control, while reducing the operating threshold for non-professional users.

[0061] Example 4 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S3 in the spacecraft attitude interaction control method based on multi-dimensional perception: "mapping the head attitude data to the camera control parameters of the visual rendering engine after filtering, so as to drive the simulation viewpoint to change synchronously with the user's head movement."

[0062] In this embodiment, the step S3, which involves filtering the head pose data and mapping it to the camera control parameters of the visual rendering engine, is implemented in the following way.

[0063] First, the head pose control module collects real-time spatial three-dimensional micro-motion data of the user's head using a face detection algorithm. This data includes pitch, yaw, and roll angles. Specifically, this module employs a sequential visual algorithm system of "detect first, estimate later." It first uses the RetinaFace model to achieve high-precision face localization and cropping in the video stream. Then, the cropped face image is fed into a 6DRepNet neural network for head pose regression, outputting the raw Euler angle data of the head pose. This raw data is then fed into an adaptive Kalman filter for filtering to smoothly remove high-frequency interference caused by natural body tremors or sensor noise, generating smooth and stable head pose Euler angle variables. Subsequently, the filtered head pose Euler angle variables are directly mapped to the first-person camera control blueprint of the UE5 engine.

[0064] During the mapping process, the pitch, yaw, and roll angles in the Euler angle variables correspond to the camera's rotation angles in the vertical, horizontal, and around the line of sight, respectively. Specifically, when the user's head moves in pitch, the filtered pitch angle variable drives the camera to rotate synchronously in the vertical direction, achieving vertical adjustment of the viewing angle; when the user's head moves in yaw, the filtered yaw angle variable drives the camera to rotate synchronously in the horizontal direction, achieving horizontal rotation of the viewing angle; and when the user's head moves in roll, the filtered roll angle variable drives the camera to rotate synchronously around the line of sight, achieving tilting of the viewing angle.

[0065] Through this direct numerical mapping, the pitch, yaw, and roll movements of the simulated viewpoint are synchronized with the pitch, yaw, and roll movements of the user's head in real time, thereby driving the simulated viewpoint to maintain a high degree of consistency with the user's head movements. This mapping process ensures absolute consistency between visual feedback and the perception of the human vestibular system, effectively alleviating motion sickness commonly experienced in virtual reality experiences, and giving users a stronger sense of situational awareness and cockpit presence.

[0066] Example 5 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S4 in the spacecraft attitude interactive control method based on multi-dimensional perception: "the six-degree-of-freedom state variables are fed back to the rendering engine to update the visual image, and real-time interactive control is achieved based on the closed-loop feedback between user input and state variables."

[0067] In this embodiment, the specific implementation method of sending the six-degree-of-freedom state variables back to the rendering engine, updating the visual scene, and realizing real-time interactive control based on the closed-loop feedback between user input and state variables in step S4 is as follows.

[0068] First, the flight mechanics algorithm analysis module, i.e., the underlying dynamics dynamic link library (DLL), encapsulates the calculated six-degree-of-freedom (DOF) state variables through its data output interface after completing the dynamics calculation within the current integration step. These six DFB state variables specifically include the spacecraft's position coordinates in three-dimensional space (linear displacements along the X, Y, and Z axes), attitude information (represented by pitch, yaw, roll angles, or corresponding quaternions), and linear and angular velocity components. This state variable data is transmitted back to the UE5 rendering engine in real time via a preset communication protocol. Upon receiving these six DFB state variables, the UE5 rendering engine uses them as core parameters to drive visual updates, inputting them into the transformation components of the spacecraft model and the camera control blueprint in the scene. Specifically, based on the received position and attitude data, the rendering engine recalculates and sets the spatial transformation matrix of the spacecraft model in the virtual scene, thereby driving the spacecraft model in the visual scene to perform synchronous displacement and rotation updates, ensuring that the displayed content is highly consistent with the flight state calculated by the flight mechanics algorithm analysis module.

[0069] Meanwhile, the updated visual feed is presented to the user as visual feedback through the display device. Based on this visual feedback—the spacecraft's real-time attitude and position in the virtual scene—and their own operational intentions, the user adjusts input devices such as the joystick and throttle, generating new operational input signals. These new input signals are then collected again by the flight joystick data acquisition module and transmitted to the flight mechanics algorithm analysis module for the next round of calculations, thus forming a complete closed-loop control circuit.

[0070] The specific process of this closed-loop feedback loop is as follows: user input, signal reading, dynamic calculation, result feedback, and visual update. The delay of the entire loop is controlled at the millisecond level, enabling users to perceive the correspondence between operation and visual changes in real time. For example, the physical lag that makes it difficult to stop immediately after a spacecraft "steers" in a vacuum due to the conservation of angular momentum, or the push-back feeling of the thrust-to-weight ratio against gravity when the retro-rocket is activated during the landing phase, thus achieving highly immersive real-time interactive control.

[0071] This specific implementation constructs a closed-loop feedback loop with millisecond-level latency, tightly coupling each user's operation input with the dynamic calculation results and visual updates, enabling users to instantly perceive the real physical impact of the operation on the spacecraft's attitude and position, significantly improving the seriousness and immersive realism of the operation.

[0072] Example 6 In this embodiment, the complete implementation of the spacecraft attitude interaction control method based on multi-dimensional perception will be described in detail. This method constructs a millisecond-level closed-loop interaction system from user operation input to visual feedback by deeply coupling the real-time mechanical mapping algorithm of the physics engine with the high-precision head attitude perception module.

[0073] At the interactive device level, this embodiment uses the Logitech X56 HOTAS flight joystick as the core control device. This device consists of a joystick and a throttle controller, and incorporates a high-precision Hall sensor to accurately collect input signals such as joystick displacement and knob angle, outputting them via a USB digital interface. The joystick is equipped with a FEEL spring tension system, supporting multiple levels of spring force adjustment to simulate the control force feel under different flight conditions. During the data acquisition phase, the PyGame library in Python reads the USB digital interface signals of the Logitech X56 HOTAS flight joystick, parsing the joystick displacement and throttle input into four types of operation input signals: pitch, roll, yaw, and throttle. On the UE5 platform, secondary development is performed based on the SimpleController plugin. This plugin uses the SDL framework to divide the joystick input data into four types of signals: pitch, roll, yaw, and throttle. When the user operates the joystick, the corresponding four types of data are transmitted in real time to the subsequent dynamics algorithm analysis module.

[0074] To further enhance the interactive immersion of the simulation project, this embodiment integrates a deep learning-based head pose control module on top of the existing hardware control. This module employs a sequential visual algorithm system of detection followed by estimation: first, the RetinaFace model is used to achieve high-precision face localization and cropping in the video stream; then, the cropped face image is fed into a 6DRepNet neural network for head pose regression, outputting raw head pose data including pitch, yaw, and roll angles. The captured raw head pose data is smoothed using an adaptive Kalman filter to remove high-frequency jitter noise and generate filtered Euler angle variables for the head pose. In module development, the Python environment was first configured, integrating the necessary libraries for visual detection, matrix operations, and data transmission, providing underlying support for head pose recognition and data processing; subsequently, the corresponding functional modules were compiled and integrated in the UE5 simulation project, embedding the head pose reception and parsing logic into the system kernel to ensure stable communication between the engine and the vision module. After the environment and engineering configuration are completed, start the deep space exploration simulation program. At the same time, create a new independent terminal to run the head attitude recognition server program. The server transmits data through the local network address and port by default, and sends the calculated head attitude angle to the UE5 simulation system in real time.

[0075] At the core algorithm level, this embodiment constructs a real-time mechanics mapping algorithm with a deeply coupled physics engine. The flight joystick data acquisition module injects the acquired operation input signals into the underlying dynamic link library (DLL), which is written in C++ and includes a data input interface, an internal differential iteration calculation module, and a data output interface. Based on the spacecraft's current mass, moment of inertia matrix, atmospheric density, and gravitational gradient, the DLL converts the operation input signals into control torque and thrust vectors. Specifically, the DLL inputs the desired angular velocity from the joystick to the attitude kinematics PID control system based on the flywheel assembly. It calculates the error between the desired attitude quaternion and the actual attitude quaternion in real time using a Lyapunov control law, and automatically allocates the rotational speed of the virtual reaction flywheel based on this error to generate auxiliary stabilizing torque. Subsequently, the DLL calculates the spacecraft's angular acceleration and linear acceleration within the next integral step based on the orbital dynamics model, generating six-degree-of-freedom state variables containing spatial position and attitude information.

[0076] At the level of visual rendering and closed-loop feedback, the real-time visual update module calls the DLL file to obtain the six-degree-of-freedom state variables of the spacecraft in real time and sends these state variables back to the UE5 rendering engine to drive the synchronous update of the visual image. Simultaneously, the filtered head attitude Euler angle variables are directly mapped to the first-person camera control blueprint of the UE5 engine, ensuring that the pitch, yaw, and roll motions of the simulated viewpoint change synchronously with the pitch, yaw, and roll motions of the user's head, achieving a visual-driven motion effect. The updated visual image is presented to the user as visual feedback, and the user adjusts the operation input signals based on this visual feedback, forming a millisecond-level latency closed-loop feedback system encompassing user input, signal reading, dynamics calculation, result feedback, and visual update. This closed-loop feedback allows the operator to realistically experience the physical lag of a spacecraft struggling to stop immediately after ruddering in a vacuum due to the conservation of angular momentum, or the thrust-to-weight ratio pushing against gravity when the retro-rockets activate during landing, greatly enhancing the seriousness and immersive realism of the operation. Meanwhile, the absolute consistency between visual feedback and the human vestibular system's perception effectively alleviates motion sickness, a common symptom in virtual reality experiences, and gives users a stronger sense of situational awareness and cockpit presence.

[0077] Example 7 This invention also provides a spacecraft attitude interaction control device based on multi-dimensional perception, comprising: The first module is used to collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust, and the head posture data includes pitch angle, yaw angle and roll angle. The second module is used to map the operation input signal into control torque and thrust vector, and perform real-time calculation based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables; The third module is used to map the head posture data to the camera control parameters of the visual rendering engine after filtering, so as to drive the simulation viewpoint to change synchronously with the user's head movement. The fourth module is used to send the six-degree-of-freedom state variables back to the rendering engine, update the visual scene, and realize real-time interactive control based on the closed-loop feedback between user input and state variables.

[0078] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0079] Example 8 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0080] Example 9 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A spacecraft attitude interactive control method based on multi-dimensional perception, characterized in that, Includes the following steps: S1, collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust. The head posture data includes pitch angle, yaw angle and roll angle. S2, the operation input signal is mapped into control torque and thrust vector, and a six-degree-of-freedom state variable is generated based on the dynamic parameters of the spacecraft in real time. S3, the head posture data is filtered and then mapped to the camera control parameters of the visual rendering engine to drive the simulation viewpoint to change synchronously with the user's head movement; S4, the six-degree-of-freedom state variables are sent back to the rendering engine to update the visual scene, and real-time interactive control is achieved based on the closed-loop feedback between user input and state variables.

2. The method as described in claim 1, characterized in that, The acquisition of user input signals and head posture data includes: By reading the USB digital interface signal of the Logitech X56 HOTAS flight stick through the PyGame library, the stick displacement and throttle input are resolved into four types of operation input signals: pitch, roll, yaw and throttle. The RetinaFace face detection algorithm is used to perform high-precision face localization and cropping on the video stream. The cropped face image is then input into the 6DRepNet neural network for head pose regression, and the output is raw head pose data including pitch angle, yaw angle and roll angle.

3. The method as described in claim 2, characterized in that, Also includes: The raw head attitude data, including pitch, yaw and roll angles, is smoothed by adaptive Kalman filtering to remove high-frequency jitter noise and generate filtered head attitude Euler angle variables.

4. The method as described in claim 1, characterized in that, The process of mapping the operational input signal into control torque and thrust vector, and performing real-time calculations based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables includes: The operation input signal is injected into the underlying dynamic link library (DLL). The DLL converts the operation input signal into control torque and thrust vector based on the spacecraft's current mass, moment of inertia matrix, atmospheric density, and gravitational gradient. The DLL calculates the angular and linear acceleration of the spacecraft in the next integration step based on the orbital dynamics model, generating a six-degree-of-freedom state variable containing spatial position and attitude information.

5. The method as described in claim 4, characterized in that, The process of converting the operation input signal into control torque and thrust vector includes: The user inputs the desired angular velocity via a joystick into the attitude kinematics PID control system based on the flywheel assembly. The Lyapunov control law calculates the error between the desired attitude quaternion and the actual attitude quaternion in real time, and automatically allocates the rotational speed of the virtual reaction flywheel based on the error to generate an auxiliary stabilizing torque.

6. The method as described in claim 1, characterized in that, The step of mapping the filtered head pose data to the camera control parameters of the visual rendering engine includes: directly mapping the filtered head pose Euler angle variables to the first-person camera control blueprint of the UE5 engine, so that the pitch, yaw, and roll motions of the simulated viewpoint change synchronously with the pitch, yaw, and roll motions of the user's head.

7. The method as described in claim 1, characterized in that, The step of sending the six-degree-of-freedom state variables back to the rendering engine to update the view and achieving real-time interactive control based on the closed-loop feedback between user input and state variables includes: The six-degree-of-freedom state variables are fed back to the UE5 rendering engine to drive the synchronous update of the visual scene. The updated visual scene is presented to the user as visual feedback. Based on the visual feedback, the user adjusts the operation input signal, forming a millisecond-level latency closed-loop feedback of user input, signal reading, dynamic calculation, result feedback, and visual scene update.

8. A spacecraft attitude interaction control device based on multi-dimensional perception, characterized in that, include: The first module is used to collect user operation input signals and head posture data. The operation input signals include joystick displacement and throttle thrust, and the head posture data includes pitch angle, yaw angle and roll angle. The second module is used to map the operation input signal into control torque and thrust vector, and perform real-time calculation based on the spacecraft's dynamic parameters to generate six-degree-of-freedom state variables; The third module is used to map the head posture data to the camera control parameters of the visual rendering engine after filtering, so as to drive the simulation viewpoint to change synchronously with the user's head movement. The fourth module is used to send the six-degree-of-freedom state variables back to the rendering engine, update the visual scene, and realize real-time interactive control based on the closed-loop feedback between user input and state variables.

9. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images