Digital Intelligent Space Flight Interaction Method and System Based on Multi-Sensor Perception

Abstract

This disclosure provides a method and system for intelligent space flight interaction based on multi-sensor perception. The method specifically includes: acquiring target control commands issued by the user when the intelligent flight interaction device is flying within a target intelligent space; when the target control commands are applied to the intelligent flight interaction device, collecting multimodal perception data through the multi-sensor perception module on the intelligent flight interaction device during the issuance of the target control commands; performing real-time fusion processing on the multimodal perception data to obtain fused perception data; identifying the user's target interaction intent based on the fused perception data, and executing corresponding interactive operations based on the target interaction intent and the fused perception data. This reduces the misidentification rate of interaction intents, ensures the execution accuracy of interactive operations, and ultimately improves the robustness and operational accuracy of the intelligent flight interaction device in complex environments.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent interactive control technology, and in particular to a digital intelligent space flight interaction method and system based on multi-sensor perception. Background Technology

[0002] As an intelligent carrier integrating digital technology and physical space, digital spaces are widely used in various scenarios such as smart exhibition halls, digital twins, and intelligent interaction. The intelligent interaction capabilities of digital spaces directly affect the user's experience and the presentation of the space's functions. However, with increasing demands for interaction in digital spaces, the current level of interaction falls far short of expectations, especially in spaces containing intelligent flight interaction devices. These devices primarily focus on flight control and lack effective means of user interaction, making it even more difficult to perceive user actions in complex environments, thus reducing interaction robustness and operational accuracy in complex environments. Summary of the Invention

[0003] The main objective of this disclosure is to propose a digital intelligent space flight interaction method and system based on multi-sensor perception, which can improve the interaction robustness and operational accuracy in complex environments.

[0004] To achieve the above objectives, a first aspect of this disclosure proposes a digital intelligent space flight interaction method based on multi-sensor perception, comprising: When the intelligent flight interaction device is flying in the target digital space, it acquires the target control commands made by the user. When the target control command is applied to the intelligent flight interaction device, during the process of issuing the target control command, multimodal perception data is collected by the multi-sensor perception module of the intelligent flight interaction device. The multimodal sensing data is fused in real time to obtain fused sensing data; The system identifies the user's target interaction intent based on the fused perception data, and performs corresponding interaction operations based on the target interaction intent and the fused perception data.

[0005] In some embodiments, identifying the user's target interaction intent based on the fused sensing data, and performing a corresponding interaction operation based on the target interaction intent and the fused sensing data, includes: Based on the pressure data and device movement trajectory in the fused sensing data, the user's target interaction intent is identified as an intention to write in the air. The location information in the fused sensing data is used to track the writing trajectory of the intelligent flight interactive device in the target digital space in real time, and the tactile information in the fused sensing data is used to confirm the pressure data corresponding to different trajectory points during the writing process. The writing motion trajectory is associated with the corresponding pressure data one by one to generate a trajectory point cloud fitting curve with pressure information. The thickness of the fitting curve is adjusted according to the pressure data of each trajectory point to obtain the adjusted fitting curve. A digital writing model is generated based on the adjusted fitting curve, and the digital writing model is transmitted to a display terminal for real-time display.

[0006] In some embodiments, the digital intelligent space flight interaction method based on multi-sensor perception further includes: When the target control command is not applied to the intelligent flight interaction device, the user's hand movement image is captured by the camera in the multi-sensor perception module; Based on the hand motion images, the user's air gestures are recognized to determine the user's non-contact target interaction intentions; Based on the non-contact target interaction intent, generate corresponding device control commands and control the intelligent flight interaction device to execute the device control commands.

[0007] In some embodiments, generating corresponding device control commands based on the non-contact target interaction intent and controlling the intelligent flight interaction device to execute the device control commands includes: When the non-contact target interaction intent is a gesture control intent, the intelligent flight interaction device is controlled to fly to the position of the target object indicated by the air gesture based on the device control command, execute the first interaction operation corresponding to the air gesture, and provide the user with corresponding vibration feedback and force feedback operations according to the attributes of the target object and the type of the first interaction operation. When the non-contact target interaction intent is a spatial interaction intent, the intelligent flight interaction device is controlled to perform the second interaction operation corresponding to the air gesture based on the device control command, and the corresponding tactile feedback operation is provided to the user.

[0008] In some embodiments, performing the corresponding interactive operation based on the target interactive intent and the fused perception data includes: When collaborative devices exist within the target intelligent space, the spatial location and sensing capability information of each collaborative device are acquired, wherein the collaborative devices include display devices and visual acquisition devices; Based on the perception capability information of each collaborative device and the target interaction intention, collaborative perception tasks are assigned to each collaborative device, wherein the display device is responsible for visual and auditory perception from a fixed perspective, the visual acquisition device is responsible for high-precision detail visual perception, and the intelligent flight interaction device is responsible for motion perception and three-dimensional interaction. The system receives sensing data collected by each of the collaborative devices, synchronizes all sensing data in time and aligns them in spatial coordinate system, and then fuses the aligned multi-device sensing data with the fused sensing data from multiple perspectives to generate a global collaborative sensing result.

[0009] In some embodiments, the digital intelligent space flight interaction method based on multi-sensor perception further includes: When multiple intelligent flight interaction devices are flying in the target digital space, the master control device and slave control device among the multiple intelligent flight interaction devices are predetermined; The main control device plans the formation shape and assigns a corresponding formation position and perception area to each intelligent flight interaction device, and distributes distributed perception tasks according to the perception area of ​​each intelligent flight interaction device. Control each of the intelligent flight interaction devices to fly in formation based on the corresponding formation position, and collect multimodal perception data within their respective perception areas in real time; The master control device receives multimodal sensing data collected by each slave control device and performs distributed data fusion to generate regional sensing results.

[0010] In some embodiments, the real-time fusion processing of the multimodal sensing data to obtain fused sensing data includes: The Kalman filter algorithm is used to perform multi-source fusion of position and attitude data in the multimodal sensing data to obtain real-time position and attitude information. Based on the real-time position and attitude information, the spatial coordinates of all the multimodal sensing data are transformed to a unified world coordinate system to obtain spatially transformed multimodal sensing data. The NTP protocol is used to perform timestamp synchronization processing on all the multimodal sensing data to obtain time-synchronized multimodal sensing data. The spatially transformed multimodal sensing data and the time-synchronized multimodal sensing data are integrated to generate fused sensing data containing spatiotemporal correlation information.

[0011] To achieve the above objectives, a second aspect of this disclosure proposes a digital intelligent space flight interaction system based on multi-sensor perception, comprising: The command response module is used to acquire target control commands issued by the user when the intelligent flight interaction device is flying in the target digital space; The sensing module is used to collect multimodal sensing data through the multi-sensor sensing module on the intelligent flight interaction device during the process of issuing the target control command when the target control command is applied to the intelligent flight interaction device. A multimodal fusion module is used to perform real-time fusion processing on the multimodal sensing data to obtain fused sensing data; The intent recognition and interaction module is used to identify the user's target interaction intent based on the fused perception data, and to perform corresponding interaction operations based on the target interaction intent and the fused perception data.

[0012] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the digital intelligent space flight interaction method based on multi-sensor perception described in the first aspect embodiment.

[0013] To achieve the above objectives, a fourth aspect of the present disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the digital intelligent space flight interaction method based on multi-sensor perception described in the first aspect embodiment.

[0014] The beneficial effects of the embodiments disclosed herein include: By integrating a multi-sensor perception module into the intelligent flight interaction device, specifically designed to collect multimodal perception data during the user's control commands, this approach overcomes the shortcomings of traditional devices that focus solely on flight control and lack interactive perception capabilities. This provides an effective technical means for user-flight interaction. Subsequently, the collected multimodal perception data undergoes real-time fusion processing. Leveraging the complementarity and redundancy of different sensor types in perception dimensions, this effectively overcomes the problem of single-sensor susceptibility to interference and perception failure in complex environments, significantly improving the stability and reliability of user interaction action perception and enhancing the robustness of the interaction. Finally, based on comprehensive and accurate fused perception data, the user's target interaction intent is identified, and corresponding interactive operations are executed. This reduces the misidentification rate of interaction intents, ensures the accuracy of interactive operations, and ultimately improves the robustness and operational precision of the intelligent flight interaction device in complex environments. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the digital intelligent space flight interaction method based on multi-sensor perception provided in the embodiments of this disclosure; Figure 2 yes Figure 1A flowchart further includes step S103; Figure 3 yes Figure 1 A flowchart further includes step S104; Figure 4 This is a flowchart illustrating the process of recognizing air gestures for interaction provided in an embodiment of this disclosure; Figure 5 yes Figure 1 Another process diagram further included in step S104; Figure 6 This is a flowchart illustrating the process of area perception in multiple intelligent flight interaction device scenarios provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the functional modules of the digital intelligent space flight interaction system based on multi-sensor perception provided in the embodiments of this disclosure; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0016] The accompanying drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of this disclosure. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0017] It is understood that in the specific embodiments of this disclosure, which involve retrieving target control commands, multimodal perception data and related data, when the above embodiments of this disclosure are applied to specific products or technologies, permission or consent from the target can be obtained first, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0018] Furthermore, when the embodiments of this disclosure require access to target control commands, multimodal perception data, and related data, separate permission or separate consent to the target control commands, multimodal perception data, and related data can be obtained through pop-up windows or by jumping to a confirmation page. After clearly obtaining separate permission or separate consent to the target control commands, multimodal perception data, and related data, the necessary target control commands, multimodal perception data, and related data for enabling the embodiments of this disclosure to operate normally can then be obtained.

[0019] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating the digital intelligent space flight interaction method based on multi-sensor perception provided in this disclosure. This digital intelligent space flight interaction method based on multi-sensor perception can be applied to a server, an intelligent flight interaction device, or a digital intelligent space flight interaction system based on multi-sensor perception (hereinafter referred to as the system), or can be jointly executed by an intelligent flight interaction device and a server. The digital intelligent space flight interaction method based on multi-sensor perception includes steps S101 to S104: Step S101: When the intelligent flight interaction device is flying in the target digital space, it acquires the target control commands made by the user. Step S102: When the target control command is applied to the intelligent flight interaction device, during the process of making the target control command, multimodal perception data is collected by the multi-sensor perception module on the intelligent flight interaction device. Step S103: Perform real-time fusion processing on the multimodal sensing data to obtain fused sensing data; Step S104: Identify the user's target interaction intent based on the fused perception data, and execute the corresponding interaction operation based on the target interaction intent and the fused perception data.

[0021] Regarding step S101 above, the target intelligent space can be a physical space requiring aerial interaction, such as a smart exhibition hall, a digital twin scene, or a family living room. The intelligent flight interaction device refers to an aircraft with autonomous flight capabilities and integrated multiple sensors; for example, it can be a quadcopter, hexacopter, or octocopter. In this embodiment, the intelligent flight interaction device can be called a smart pen. The target control commands issued by the user can be physical commands directly applied to the device, such as touching, pressing, or waving, or non-contact commands, such as aerial gestures or voice commands.

[0022] For example, when a user writes or touches on the aircraft, the pressure sensor and inertial measurement unit on the intelligent flight interaction device sense the force and motion trajectory, forming an action-based command; when the user waves or points at an object from a distance, the camera captures the gesture, forming a non-contact command. This embodiment of the disclosure provides a rich data source for subsequent interaction intent recognition by sensing various forms of user commands.

[0023] It should be noted that, by acquiring the user's control commands to the flight equipment and distinguishing whether the commands directly affect the equipment, this embodiment of the present disclosure can adopt differentiated perception strategies for different interaction methods, laying the foundation for subsequent multimodal fusion.

[0024] Regarding step S102 above, the multi-sensor perception module is a collection of sensors integrated into the intelligent flight interaction device. The multi-sensor perception module can include sensor units in various dimensions to collect perception data in the corresponding dimensions, and the setting position of each sensor unit can be determined based on actual collection needs.

[0025] Furthermore, embodiments of this disclosure can acquire device position and attitude data through a position sensor array. The position sensor array includes a GPS-RTK module, an Ultra Wide Band (UWB) module, a visual positioning module, and an Inertial Measurement Unit (IMU) module. GPS-RTK is used for centimeter-level positioning outdoors, UWB for high-precision positioning indoors, the visual positioning module provides relative pose through the SLAM algorithm, and the IMU provides acceleration and angular velocity information. These data together constitute the real-time pose of the aircraft. Secondly, scene visual data is acquired through a visual sensor array, which includes a front-facing RGB camera, a depth camera, and a downward-facing camera. The front-facing RGB camera is used for scene perception and object recognition, the depth camera for 3D perception and distance measurement, and the downward-facing camera for ground detection and landing assistance. Thirdly, environmental audio data is acquired through an auditory sensor array, which includes a microphone array and an acoustic sensor. The microphone array is used for sound source localization and speech recognition, and the acoustic sensor is used to acquire environmental sound characteristics. Then, contact and vibration data are collected through a tactile sensor array, which includes pressure sensors, temperature sensors, and vibration sensors. The pressure sensors detect writing and touching pressure, the temperature sensors detect ambient and device temperatures, and the vibration sensors provide tactile feedback and detect device vibration. Finally, ambient environmental data is collected through an environmental sensor array, which includes LiDAR, ultrasonic sensors, a barometer, and an optical flow sensor. The LiDAR is used to generate 3D point clouds and detect obstacles, the ultrasonic sensors are used for near-field obstacle detection, the barometer is used for altitude measurement, and the optical flow sensor is used for speed measurement and hovering assistance.

[0026] Specifically, five types of sensor arrays are deployed at different locations on the aircraft. For example, the GPS-RTK antenna is on top, the UWB sensor is on the bottom, the RGB camera is on the front, the depth cameras are side-by-side, the microphone array is distributed around the perimeter, the pressure sensors are on the bottom and sides, and the lidar is rotatable on top. Each sensor operates independently according to its optimal sampling frequency, and the data includes timestamps and local coordinates in the device coordinate system. During integration, embodiments of this disclosure can add modal tags to each type of data to form a unified data structure, such as JSON format containing fields such as modality type, timestamp, sensor ID, and data value. This standardized integration facilitates subsequent fusion processing.

[0027] When the target control command is applied to the device, such as when a user touches the aircraft or writes in the air, the device experiences pressure and movement. During the user's command process, the device continuously collects data from various sensors. For example, when a user presses their finger on the surface of the aircraft to write the word "Welcome," the pressure sensor records the pressure value at each trajectory point, the IMU and visual positioning module record the aircraft's three-dimensional motion trajectory, the microphone array records the ambient sound during writing, and the depth camera simultaneously records the relative position of the user's hand and the device. All of this data constitutes multimodal perception data.

[0028] It should be noted that the embodiments of this disclosure integrate multiple sensor arrays, such as position, vision, hearing, touch, and environment, into the flight equipment, thereby achieving comprehensive perception of user interaction actions. This makes up for the shortcomings of traditional flight equipment, which only has flight control sensors and lacks interactive perception capabilities, and provides a data foundation for robust interaction in complex environments.

[0029] Regarding step S103 above, real-time fusion processing refers to aligning and integrating multimodal sensing data in three dimensions: time, space, and semantics. Because different sensors have different sampling frequencies and data formats—for example, IMUs have a sampling frequency as high as 400 Hz while GPS only has 10 Hz, and cameras only have 30 frames per second—Kalman filtering or extended Kalman filtering is needed to fuse position and attitude data from multiple sources to obtain real-time pose information with centimeter-level accuracy. Simultaneously, the Network Time Protocol (NTP) can be used to timestamp and synchronize all sensor data, and the spatial coordinates of each sensor can be transformed to a unified world coordinate system, such as a right-handed coordinate system with the exhibition hall entrance as the origin. The fused data not only retains the original information of each modality but also establishes temporal and spatial relationships, forming structured fused sensing data.

[0030] It should be noted that the embodiments of this disclosure solve the problem of inconsistency between data from different sensors in time and space through multi-sensor fusion processing, which significantly improves the accuracy and robustness of interactive action perception and avoids perception failure caused by environmental interference such as changes in light or GPS signal obstruction.

[0031] Regarding step S104 above, the target interaction intent refers to the purpose the user wants to achieve through control commands, such as writing in the air, controlling objects with gestures, or interacting with digital content. Based on fused perception data, embodiments of this disclosure can utilize preset recognition models, such as neural network-based air writing trajectory recognition or MediaPipe-based gesture classification, to analyze the user's intent. For example, when the pressure sensor values ​​in the fused data change regularly and the aircraft trajectory forms a continuous curve, it is recognized as an air writing intent; when the hand key point movement trajectory forms a pointing action, it is recognized as a gesture control intent. After determining the intent, embodiments of this disclosure can perform corresponding interactive operations based on the intent type and detailed information in the fused data, such as the pressure magnitude of the writing trajectory and the coordinates of the object the gesture is pointing to. For example, the writing trajectory with pressure information can be generated into a three-dimensional digital model and projected onto a screen, or the aircraft can be controlled to fly towards and grab a target object.

[0032] It should be noted that the embodiments of this disclosure accurately identify user intentions and execute operations by fusing sensor data, transforming the originally scattered sensor data into meaningful interactive behaviors, enabling users to interact with the flying device in the air as naturally as using a smart pen, such as writing, pointing, and grasping, which significantly improves the accuracy of operation and user experience in complex environments.

[0033] In summary, this embodiment of the present disclosure, through the execution of the multi-sensor-based intelligent space flight interaction method in steps S101 to S104, integrates a multi-sensor perception module into the intelligent flight interaction device. This module is specifically designed to collect multimodal perception data during the user's control commands to the device, overcoming the shortcomings of existing devices that focus solely on flight control and lack interactive perception capabilities. This provides an effective technical means for user-flight interaction. Subsequently, the collected multimodal perception data is fused in real time. By utilizing the complementarity and redundancy of different types of sensors in the perception dimension, the problem of single sensors being susceptible to interference and perception failure in complex environments is effectively overcome. This significantly improves the stability and reliability of user interaction action perception and enhances the robustness of the interaction. Finally, based on comprehensive and accurate fused perception data, the user's target interaction intent is identified, and the corresponding interactive operation is executed in conjunction with the fused perception data. This reduces the misidentification rate of the interaction intent, ensures the execution accuracy of the interactive operation, and ultimately improves the interaction robustness and operational accuracy of the intelligent flight interaction device in complex environments.

[0034] Please see Figure 2 , Figure 2 yes Figure 1 The flowchart further includes step S103. In some embodiments, the process of performing real-time fusion processing on multimodal sensing data to obtain fused sensing data may also include steps S201 to S203: Step S201: The Kalman filter algorithm is used to perform multi-source fusion of position and attitude data in multimodal sensing data to obtain real-time position and attitude information. Based on the real-time position and attitude information, the spatial coordinates of all multimodal sensing data are transformed to a unified world coordinate system to obtain spatially transformed multimodal sensing data. Step S202: Use the NTP protocol to perform timestamp synchronization processing on all multimodal sensing data to obtain time-synchronized multimodal sensing data; Step S203: Integrate the spatially transformed multimodal sensing data and the time-synchronized multimodal sensing data to generate fused sensing data containing spatiotemporal correlation information.

[0035] In the steps described above, the Kalman filter algorithm is a recursive state estimation method suitable for fusing noisy measurement data from different sensors. Position and attitude data come from GPS-RTK providing outdoor absolute coordinates, UWB providing indoor relative coordinates, visual SLAM providing relative pose, and IMU providing acceleration and angular velocity. The filter's state vector includes 3D position, 3D velocity, 3D attitude angles, and sensor bias. Through iterative prediction and update steps, the filter outputs a smooth and high-frequency pose estimate. Spatial coordinate transformation refers to converting the spatial coordinates from each sensor's data, such as object coordinates in a depth camera, to a unified world coordinate system, such as a right-handed coordinate system with the exhibition hall entrance as the origin, using the current pose transformation matrix.

[0036] Network Time Protocol (NTP) is used to synchronize clocks in a distributed system. All sensor devices periodically request time calibration from the NTP server to keep local time errors within a preset range. Since different sensors have significantly different sampling frequencies—for example, IMUs can reach 400 Hz while GPS is only 10 Hz—this embodiment of the present disclosure can use a linear interpolation method to interpolate low-frequency data onto the sampling time grid of high-frequency data, thereby giving all data a unified time label.

[0037] Integration refers to merging all spatially aligned and temporally synchronized sensor data into a unified data structure. This data structure includes the modality type, timestamp, spatial coordinates, and specific numerical value for each observation. This fused sensor data with temporal and spatial labels enables subsequent intent recognition and interaction execution to be based on precise spatiotemporal context. For example, embodiments of this disclosure can package the aircraft's position at a certain moment, the RGB image acquired at that moment, the depth map, pressure values, microphone audio, etc., into a single data frame for use by the recognition algorithm.

[0038] It should be noted that the embodiments disclosed herein solve the temporal and spatial inconsistencies of multi-sensor data through Kalman filtering and NTP synchronization, enabling data from different modalities to be jointly analyzed within the same spatiotemporal framework, thus providing a high-quality data foundation for accurate intent recognition and execution.

[0039] Please see Figure 3 , Figure 3 yes Figure 1 The flowchart further includes step S104. In some embodiments, the process of identifying the user's target interaction intent based on fused sensing data and performing the corresponding interaction operation based on the target interaction intent and fused sensing data may further include steps S301 to S304: Step S301: Based on the pressure data and device motion trajectory in the fused sensing data, identify the user's target interaction intent as an air writing intent; Step S302: Track the writing trajectory of the intelligent flight interactive device in the target digital space in real time by using the location information in the fused sensing data, and confirm the pressure data corresponding to different trajectory points during the writing process by using the tactile information in the fused sensing data. Step S303: Associate the writing motion trajectory with the corresponding pressure data one by one to generate a trajectory point cloud fitting curve with pressure information, and adjust the thickness of the fitting curve according to the pressure data of each trajectory point to obtain the adjusted fitting curve. Step S304: Generate a digital writing model based on the adjusted fitting curve, and transmit the digital writing model to the display terminal for real-time display.

[0040] In the above steps, pressure data refers to the values ​​collected by the pressure sensors in the tactile sensor array, typically located on the bottom or side of the aircraft, used to detect the force applied by the user when pressing or writing. Device motion trajectory refers to the continuous displacement path of the aircraft in three-dimensional space obtained by the position sensor array through the fusion of GPS-RTK, UWB, visual SLAM, and IMU data. When the pressure values ​​change continuously over time and the aircraft's position undergoes regular displacement, this embodiment of the disclosure can determine that the user is writing in mid-air. For example, when a user presses their finger on the surface of the aircraft and moves it in space, the pressure sensor records the changes in force during the initial stroke, movement, and final stroke, while the position sensor records the aircraft's motion trajectory. This embodiment of the disclosure identifies this as an intention to write in mid-air.

[0041] Position information refers to centimeter-level real-time coordinates after Kalman filtering and fusion. Tactile information mainly refers to the pressure value corresponding to each trajectory point recorded by the pressure sensor and the feedback frequency recorded by the vibration sensor. This embodiment of the disclosure can bind the aircraft's position coordinates at each moment with the pressure value at that moment, forming a series of three-dimensional spatial points with pressure labels. For example, when a user writes a stroke, this embodiment of the disclosure can record a trajectory point every few milliseconds and simultaneously save the pressure sensor reading at that point, thereby obtaining precise force information for each trajectory point.

[0042] A trajectory point cloud refers to a set of all three-dimensional spatial points labeled with pressure. A fitted curve is a continuous, smooth curve formed by connecting these discrete points using B-splines or NURBS algorithms. Adjusting the line thickness dynamically maps the pressure value to the curve's radius: the greater the pressure value, the larger the curve's radius at that point, and the thicker the line; the smaller the pressure value, the thinner the line. For example, at the start of a stroke, the pressure is 0.2 Newtons, and the curve radius is 0.5 millimeters; during the stroke, the pressure increases to 0.8 Newtons, and the curve radius increases to 2 millimeters; at the end of the stroke, the pressure returns to 0.1 Newtons, and the radius decreases to 0.3 millimeters, thus simulating the brushstroke effect of a traditional Chinese calligraphy brush.

[0043] A digital writing model refers to a three-dimensional mesh model generated by triangulating the adjusted fitted curve, and can use common formats such as OBJ or GLTF. The display terminal can be a large screen in a smart exhibition hall, a user's augmented reality glasses, or a mobile phone screen—any terminal device capable of displaying images. In this embodiment, the model can be sent to the display terminal via wireless communication such as Wi-Fi or Bluetooth, and the terminal renders the user's written three-dimensional text or graphics in real time. For example, if a user writes the characters "Welcome" in space, this embodiment can generate a three-dimensional text model with varying thickness in real time and project it onto the exhibition hall wall, allowing viewers to see the writing effect instantly.

[0044] It should be noted that, through the joint modeling of pressure and trajectory, the embodiments of this disclosure enable air writing not only to record spatial trajectories but also to simulate the force of real brushstrokes, greatly enhancing the expressive power of digital creation and providing an intuitive three-dimensional interactive means for design, education, display and other scenarios.

[0045] Please see Figure 4 , Figure 4 This is a flowchart illustrating the process of recognizing aerial gestures for interaction provided in an embodiment of this disclosure. In some embodiments, the digital intelligent space flight interaction method based on multi-sensor perception may further include steps S401 to S403: Step S401: When the target control command is not applied to the intelligent flight interaction device, the user's hand movement image is collected by the camera in the multi-sensor perception module. Step S402: Based on hand motion image recognition, identify the user's air gestures to determine the user's non-contact target interaction intent; Step S403: Generate corresponding device control commands based on the non-contact target interaction intent, and control the intelligent flight interaction device to execute the device control commands.

[0046] In the above steps, the camera mainly refers to the RGB camera located at the front of the aircraft, used to capture a continuous sequence of images of the user's hand. The hand movement images refer to visual data including the fingers, palm, and wrist. When the user does not directly touch the aircraft but issues commands through gestures, this embodiment of the disclosure can begin to collect the user's hand movements through the camera.

[0047] Air gestures refer to commands expressed by users through hand postures and movement trajectories, such as waving, pointing, clenching a fist, and opening a palm. Furthermore, embodiments of this disclosure can use keypoint detection frameworks such as MediaPipe or OpenPose to extract hand joint coordinates from images, and then match them with a preset gesture library to identify the specific gesture type. For example, if a user stands at a distance and waves their right hand left and right, this disclosure can identify it as a gesture to call for an aircraft; if a user extends their index finger to point at an exhibit, this disclosure can identify it as a pointing intention.

[0048] Equipment control commands refer to instructions that cause the aircraft to perform specific actions, such as flying to a target location, hovering, rotating, or grabbing an object. In this embodiment, the aircraft can look up the corresponding control command in the command mapping table based on the recognized gesture type, and then execute it through the flight control system. For example, when a user points to an exhibit and makes a grabbing gesture, this embodiment can generate a command to fly towards the exhibit and activate the grabbing mechanism. The aircraft then autonomously flies above the exhibit and attempts to grab it.

[0049] It should be noted that the embodiments disclosed herein achieve aerial gesture recognition through a camera, enabling users to interact remotely without touching the aircraft, greatly expanding the application scenarios of intelligent flight devices, and making them suitable for use in large exhibition halls, dangerous environments, or occasions where contact is inconvenient.

[0050] In one embodiment, step S403 above, which generates corresponding device control commands based on the non-contact target interaction intent and controls the intelligent flight interaction device to execute the device control commands, may further include: When the non-contact target interaction intent is a gesture control intent, the intelligent flight interaction device is controlled to fly to the position of the target object indicated by the air gesture based on the device control command, execute the first interaction operation corresponding to the air gesture, and provide the user with corresponding vibration feedback and force feedback operation according to the attributes of the target object and the type of the first interaction operation. When the non-contact target interaction intent is a spatial interaction intent, the intelligent flight interaction device is controlled to perform the second interaction operation corresponding to the air gesture based on the device control command, and the corresponding tactile feedback operation is provided to the user.

[0051] In the above steps, gesture control intent refers to the user directly manipulating the aircraft to physically interact with an object through gestures, such as pointing and grasping, or pointing and touching. The target object's attributes include its hardness, weight, and surface material, which can be estimated using pressure sensors and visual recognition. Vibration feedback refers to generating vibrations of different intensities and frequencies through a vibration motor inside the aircraft, while force feedback refers to simulating drag or thrust through aircraft attitude adjustment. For example, if a user points to a hard ceramic vase and makes a grasping gesture, and the aircraft flies close and lightly touches the vase, the pressure sensor detects high hardness. This embodiment of the present disclosure can generate strong vibration feedback and upward drag to simulate the feeling of grasping a heavy object.

[0052] Spatial interaction refers to users interacting with virtual content or spatial atmosphere through gestures, such as drawing circles in the air to control orbital flight or waving to switch exhibition content. This second type of interaction does not involve physical grasping but rather controlling the aircraft's flight mode or linking with digital systems. Haptic feedback primarily provides simple confirmation prompts through vibration motors, such as a short vibration for successful execution and a long vibration for failure. For example, if a user draws a circle in the air, the aircraft will recognize it, execute orbital flight mode, and emit a short vibration to indicate successful activation.

[0053] It should be noted that the embodiments of this disclosure integrate the tactile dimension into non-contact interaction by combining vibration feedback and force feedback, enabling users to obtain a sense of touch without air contact, which significantly improves the realism and safety of the interaction.

[0054] Please see Figure 5 , Figure 5 yes Figure 1 Another flowchart further includes step S104. In some embodiments, the process of performing the corresponding interactive operation based on the target interactive intent and fused perception data may also include steps S501 to S503: Step S501: When there are collaborative devices in the target digital space, acquire the spatial location and perception capability information of each collaborative device, wherein the collaborative devices include display devices and visual acquisition devices; Step S502: Based on the perception capability information and target interaction intention of each collaborative device, assign collaborative perception tasks to each collaborative device. Among them, the display device is responsible for visual and auditory perception from a fixed perspective, the visual acquisition device is responsible for high-precision detail visual perception, and the intelligent flight interaction device is responsible for motion perception and three-dimensional interaction. Step S503: Receive the sensing data collected by each collaborative device, synchronize all sensing data in time and align them in spatial coordinate system, and fuse the aligned multi-device sensing data with the fused sensing data from multiple perspectives to generate a global collaborative sensing result.

[0055] In the above steps, "collaborating devices" refers to other intelligent devices that work in conjunction with the intelligent flight interaction device, such as intelligent displays and fixed high-definition cameras. Device type is used to distinguish between fixed sensing and mobile sensing; spatial location refers to the device's three-dimensional coordinates in a unified world coordinate system; and sensing capability information includes parameters such as resolution, sampling frequency, and field of view. Embodiments of this disclosure can automatically obtain this information through device discovery protocols such as mDNS or UPnP.

[0056] Collaborative perception tasks refer to specific data acquisition responsibilities assigned based on device characteristics. For example, display devices are typically fixed in position, making them suitable for acquiring panoramic vision and environmental audio; visual acquisition devices have high-resolution lenses, making them suitable for capturing object details; and intelligent flying interactive devices are mobile, making them suitable for acquiring depth information and 3D interactive data from multiple angles. For instance, when a user intends to fully understand a cultural relic, this embodiment of the disclosure can assign a display screen to acquire overall environmental sound, a high-definition camera to capture macro images of the relic's patterns, and an aircraft to fly around the relic to acquire 3D point clouds from its sides and bottom.

[0057] Time synchronization refers to aligning the timestamps of all devices to the same reference using the NTP protocol, with the error controlled within a preset range. Spatial coordinate system alignment refers to transforming the data from each device's local coordinate system to a unified world coordinate system based on the device's position and orientation. Multi-view fusion refers to fusing image, point cloud, and audio data collected from different perspectives into a complete 3D model or spatiotemporal context. For example, embodiments of this disclosure can fuse panoramic images from a display screen, detailed textures from a high-definition camera, and depth point clouds from an aircraft to generate a high-precision digital twin model of an artifact, which users can observe from any angle using augmented reality glasses.

[0058] For example, in a smart retail sales scenario, the target digital space is an offline experience store or live streaming room. Collaborative devices include a wall-mounted smart screen, a smart camera suspended above the product display area, and two movable smart pens. The smart screen is located at the store entrance and features a panoramic display and an omnidirectional microphone array; the smart camera is located above the product display stand and is equipped with a macro camera and a spectral sensor; the two smart pens are in standby mobile mode. Based on this, according to the core intent of product display and interaction, the system assigns collaborative perception tasks: the smart screen is responsible for collecting panoramic images of the store and customer inquiries, and displaying product prices, parameters, and promotional information in real time; the smart camera is responsible for capturing high-definition close-ups of product surface textures, material details, and color reproduction; the two smart pens are responsible for mobile 3D scanning of the product from the front and sides, respectively, while simultaneously detecting material hardness and elasticity by lightly touching the product with a bottom pressure sensor. All devices convert their collected data into a unified global coordinate system with the store's floor center as the origin. The data fusion engine performs multimodal fusion of the panoramic image from the smart screen, the detailed texture of the smart screen, the 3D point cloud of the smart pen, and tactile data to generate a high-precision digital twin model of the product. This model can be pushed to the customer's mobile phone or AR glasses in real time. Customers can rotate the product 360 degrees to view its internal structure, or use gestures to control the smart pen to grasp the digital model in the air for disassembly demonstrations. The tactile feedback from the pressure sensor is transmitted to the customer through vibration, greatly enhancing the immersive experience and conversion rate of offline shopping.

[0059] For example, in a smart classroom teaching scenario, the target digital space is a physics or chemistry laboratory. Collaborative devices include a smart screen installed at the front of the classroom, smart cameras distributed above each experimental table, and a smart pen for each student. The system acquires their spatial location and perception parameters: the smart screen is located directly in front of the classroom, possessing high-definition display and classroom-wide audio acquisition capabilities; the smart cameras are distributed above each experimental table, equipped with high-speed cameras for capturing experimental operation details; the smart pen is held by the student and possesses multi-sensor perception and aerial interaction capabilities. Based on the interactive intent of the experimental teaching, the system assigns collaborative perception tasks: the smart screen is responsible for displaying the experimental principles, operating steps, and a summary of the class's experimental data, while also collecting the teacher's explanations; the smart cameras are responsible for capturing close-up images of the teacher's demonstration and the students' experimental process, capturing the dynamic changes of experimental phenomena in real time; the smart pen is responsible for students writing experimental reports in the air, controlling virtual experimental equipment, and collecting tactile data during experimental operations. All devices synchronize their time via a local area network NTP server, uniformly converting the data to a world coordinate system with the lower left corner of the classroom podium as the origin. The data fusion engine performs multimodal fusion of the teacher's operation video, students' aerial writing trajectories, interactive data from virtual experiments, and tactile feedback collected by the Smart Pen, constructing a complete teaching spatiotemporal context. This context can be synchronized in real time to the students' AR glasses, allowing students to grasp and operate virtual experimental equipment using the Smart Pen. The tactile feedback module simulates the weight and resistance of different equipment. The system can also automatically analyze the students' operational standardization based on the fused data, generating personalized experimental guidance reports, significantly improving the intuitiveness and efficiency of experimental teaching.

[0060] It should be noted that the embodiments disclosed herein achieve comprehensive, multi-scale data collection that cannot be accomplished by a single device through multi-device collaborative sensing, complementing the advantages of fixed sensing and mobile sensing, and providing a complete spatiotemporal context for intelligent spaces.

[0061] Please see Figure 6 , Figure 6 This is a flowchart illustrating the area perception process in multiple intelligent flight interaction device scenarios provided in the embodiments of this disclosure. In some embodiments, the digital intelligent space flight interaction method based on multi-sensor perception may further include steps S601 to S604: Step S601: When multiple intelligent flight interaction devices are flying in the target digital space, the master control device and slave control device among the multiple intelligent flight interaction devices are determined in advance; In step S602, the main control device plans the formation shape, assigns a corresponding formation position and perception area to each intelligent flight interaction device, and assigns distributed perception tasks according to the perception area of ​​each intelligent flight interaction device. Step S603: Control each intelligent flight interaction device to fly in formation based on the corresponding formation position, and collect multimodal perception data in their respective perception areas in real time. In step S604, the master control device receives multimodal sensing data collected by each slave control device and performs distributed data fusion to generate regional sensing results.

[0062] In the above steps, the master control device refers to the central aircraft responsible for formation planning, task allocation and data fusion, while the slave control device refers to the follower aircraft that receives master control commands and performs distributed sensing. The master control device is usually the one with the best communication quality or the strongest computing power, or it can be specified by the user.

[0063] Formation shape refers to the geometric arrangement of multiple aircraft in the air, such as triangular formation, straight-line formation, and grid formation. The perception area refers to the three-dimensional spatial range within which each aircraft is responsible for collecting data, usually corresponding to the formation's position. Distributed perception tasks refer to the specific data collection work assigned based on the area where each aircraft is located. For example, the aircraft in front is responsible for visual obstacle avoidance, the aircraft to the side is responsible for sound source localization, and the aircraft behind is responsible for environmental monitoring. For instance, three aircraft form an equilateral triangle formation, each responsible for visual and lidar data collection in a 120-degree sector in front.

[0064] Formation flight refers to the synchronous movement of multiple aircraft while maintaining their relative positions. Real-time data acquisition refers to each aircraft continuously acquiring data from its own sensor array according to its assigned sensing tasks, and sending the data to the main control device through short-range communication such as UWB or Wi-Fi Mesh.

[0065] Distributed data fusion refers to the process by which the master control device performs weighted averaging or Kalman fusion on position, visual, auditory, and environmental data from different aircraft to obtain more complete and accurate perception results than a single device. These regional perception results can be 3D point cloud maps, heat maps, sound field distribution maps, or perception results of user interactions across multiple areas. This facilitates real-time interactive operations across multiple areas and devices based on these perception results. For example, in an exhibition hall scenario, three aircraft fly in formation, collecting images and LiDAR data from the front, left, and right sides respectively. Users can interact with the formation, controlling devices in different areas through different gestures to perform corresponding interactive operations, thus completing multi-aircraft interaction in complex scenarios.

[0066] It should be noted that the embodiments disclosed herein expand the sensing range of a single device by multi-machine collaborative formation and distributed sensing, and improve sensing accuracy and robustness by using data fusion, making them suitable for interactive operations under various complex tasks.

[0067] In summary, and with reference to several specific embodiments, the digital intelligent space flight interaction method based on multi-sensor perception in the above embodiments of this disclosure will be illustrated by examples. The intelligent flight interaction device in the above embodiments can be defined as a smart pen, or it can also be called an aircraft. Specific embodiments are as follows: In a specific application scenario, when a user wants to write in the air using the Smart Pen in a smart exhibition hall, the user first wakes up the device and enters writing mode by waving their hand or using a voice command. The front-facing RGB camera on the Smart Pen captures the user's hand movements, and a gesture recognition algorithm confirms the user's writing intention. Subsequently, the Smart Pen moves in three-dimensional space according to the user's hand movement trajectory, and its position sensor array records the three-dimensional spatial coordinates of the aircraft in real time, forming a continuous writing trajectory. Simultaneously, a pressure sensor located at the bottom of the aircraft continuously detects the writing pressure applied by the user's fingertips or pen tip, and associates the pressure value of each trajectory point with its position coordinates. This embodiment of the present disclosure can fit these pressure-labeled trajectory points to a B-spline curve, dynamically adjusting the curve radius according to the pressure value to generate a three-dimensional point cloud curve of the Chinese characters for "Welcome" with varying thickness. Further, this curve is triangulated into a three-dimensional digital model and transmitted wirelessly to an augmented reality display device. The user can then see the Chinese characters for "Welcome" floating in the air in a three-dimensional form in real time through AR glasses, achieving an immersive aerial writing interactive experience.

[0068] In another application scenario, users want to use the Smart Pen to remotely control physical exhibits in an exhibition hall. The user first extends their index finger towards the target exhibit, and the Smart Pen's front-facing RGB camera captures an image of the user's hand. Using MediaPipe keypoint detection, the pen recognizes the pointing gesture and uses a depth camera to calculate the spatial intersection point between the finger's direction and the exhibit, accurately locating the exhibit in three-dimensional space. Based on these coordinates, the Smart Pen autonomously flies towards the exhibit, maintaining a safe distance during approach using LiDAR and ultrasonic sensors. Once the pen is stably hovering above the exhibit, the user makes a fist-grabbing gesture. The Smart Pen recognizes this gesture and activates its micro-robotic arm or electromagnetic adsorption device to grasp the exhibit. Subsequently, the user moves their arm in a lateral gesture, and the Smart Pen controls its own movement based on the direction and amplitude of the gesture, thus moving the exhibit synchronously to the new position. Throughout the grasping and moving process, a pressure sensor on the bottom of the pen detects the grasping contact force and the weight of the object in real time, while a vibration sensor generates vibration feedback of appropriate intensity based on the exhibit's hardness, simulating a realistic grasping and moving sensation, giving the user a sense of immersive force perception. This interaction method significantly improves the precision and immersion of remote control of physical objects within the digital space.

[0069] In a specific application scenario, a smart exhibition hall is equipped with a smart screen, a smart camera, and multiple smart pens. These three devices work together to achieve comprehensive perception and spatiotemporal context construction of a bronze exhibit. This embodiment first automatically obtains the location information of each device via the mDNS device discovery protocol: the smart screen is fixed at the entrance of the exhibition hall, the smart camera is suspended directly above the exhibit, and the two smart pens are in a mobile state. Based on the perception capabilities of each device, collaborative perception tasks are assigned: the smart screen is responsible for collecting fixed-view visual images and environmental audio from the entrance area; the smart camera uses its high-resolution camera to capture the details of the exhibit's surface texture; and the two smart pens perform mobile multimodal perception from the front and sides of the exhibit, including 3D point cloud scanning, tactile pressure detection, and near-field ultrasonic ranging. All devices collect data in parallel and achieve timestamp synchronization via the NTP protocol, with errors controlled within a preset range.

[0070] Meanwhile, two smart pens can form a formation, with one acting as the master control device to plan the formation shape and the other as the slave control device to maintain relative position and fly in coordination, covering the front and side areas of the exhibits respectively, achieving multi-angle coverage. The collected multi-source data is then uniformly converted to a world coordinate system with the exhibition hall entrance as the origin, completing spatial alignment. Subsequently, the data fusion engine performs multimodal fusion of information collected from position sensors, visual sensors, auditory sensors, tactile sensors, and environmental sensors to generate a complete spatiotemporal context including the exhibit's 3D geometric model, surface texture, environmental sound field, and interactive pressure distribution. This spatiotemporal context is deeply integrated with the intelligent space system, supporting subsequent data sharing, collaborative processing, and AR visualization, providing visitors with an immersive experience for appreciating cultural relics.

[0071] Please see Figure 7 This disclosure also provides a digital intelligent space flight interaction system based on multi-sensor perception, which can realize the above-mentioned digital intelligent space flight interaction method based on multi-sensor perception. The digital intelligent space flight interaction system based on multi-sensor perception includes: The command response module 701 is used to acquire the target control commands made by the user when the intelligent flight interaction device is flying in the target digital space; The perception module 702 is used to collect multimodal perception data through the multi-sensor perception module on the intelligent flight interaction device during the process of issuing the target control command when the target control command is applied to the intelligent flight interaction device. The multimodal fusion module 703 is used to perform real-time fusion processing on multimodal sensing data to obtain fused sensing data; The intent recognition and interaction module 704 is used to identify the user's target interaction intent based on fused perception data, and to execute the corresponding interaction operation based on the target interaction intent and fused perception data.

[0072] In summary, the multi-sensor-based intelligent space flight interaction system, by implementing the multi-sensor-based intelligent space flight interaction method described in the above embodiments, integrates a multi-sensor perception module into the intelligent flight interaction device. This module is specifically designed to collect multi-modal perception data during the user's control commands to the device, overcoming the shortcomings of traditional devices that focus solely on flight control and lack interactive perception capabilities. This provides an effective technical means for user-flight interaction. Subsequently, the collected multi-modal perception data is fused in real time. By utilizing the complementarity and redundancy of different types of sensors in the perception dimension, the system effectively overcomes the problem of single sensors being susceptible to interference and perception failure in complex environments. This significantly improves the stability and reliability of user interaction action perception and enhances the robustness of the interaction. Finally, based on comprehensive and accurate fused perception data, the system identifies the user's target interaction intent and executes the corresponding interactive operation. This reduces the misidentification rate of interaction intent, ensures the execution accuracy of interactive operations, and ultimately improves the robustness and operational precision of the intelligent flight interaction device in complex environments.

[0073] The specific implementation of this multi-sensor-based intelligent space flight interaction system is basically the same as the specific embodiment of the multi-sensor-based intelligent space flight interaction method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this disclosure, the multi-sensor-based intelligent space flight interaction system may also be equipped with other functional modules to implement the multi-sensor-based intelligent space flight interaction method described above.

[0074] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned digital intelligent space flight interaction method based on multi-sensor perception. This electronic device can be any intelligent terminal, including tablet computers, in-vehicle computers, etc.

[0075] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to execute the digital intelligent space flight interaction method based on multi-sensor perception of the embodiments of this disclosure. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0076] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described digital intelligent space flight interaction method based on multi-sensor perception.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0079] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0080] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such orders can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0084] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of the units described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0085] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A digital intelligent space flight interaction method based on multi-sensor perception, characterized in that, include: When the intelligent flight interaction device is flying in the target digital space, it acquires the target control commands made by the user. When the target control command is applied to the intelligent flight interaction device, during the process of issuing the target control command, multimodal perception data is collected by the multi-sensor perception module of the intelligent flight interaction device. The multimodal sensing data is fused in real time to obtain fused sensing data; The system identifies the user's target interaction intent based on the fused perception data, and performs corresponding interaction operations based on the target interaction intent and the fused perception data.

2. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 1, characterized in that, The step of identifying the user's target interaction intent based on the fused sensing data, and executing the corresponding interaction operation based on the target interaction intent and the fused sensing data, includes: Based on the pressure data and device movement trajectory in the fused sensing data, the user's target interaction intent is identified as an intention to write in the air. The location information in the fused sensing data is used to track the writing trajectory of the intelligent flight interactive device in the target digital space in real time, and the tactile information in the fused sensing data is used to confirm the pressure data corresponding to different trajectory points during the writing process. The writing motion trajectory is associated with the corresponding pressure data one by one to generate a trajectory point cloud fitting curve with pressure information. The thickness of the fitting curve is adjusted according to the pressure data of each trajectory point to obtain the adjusted fitting curve. A digital writing model is generated based on the adjusted fitting curve, and the digital writing model is transmitted to a display terminal for real-time display.

3. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 1, characterized in that, The digital intelligent space flight interaction method based on multi-sensor perception also includes: When the target control command is not applied to the intelligent flight interaction device, the user's hand movement image is captured by the camera in the multi-sensor perception module; Based on the hand motion images, the user's air gestures are recognized to determine the user's non-contact target interaction intentions; Based on the non-contact target interaction intent, generate corresponding device control commands and control the intelligent flight interaction device to execute the device control commands.

4. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 3, characterized in that, The step of generating corresponding device control commands based on the non-contact target interaction intent and controlling the intelligent flight interaction device to execute the device control commands includes: When the non-contact target interaction intent is a gesture control intent, the intelligent flight interaction device is controlled to fly to the position of the target object indicated by the air gesture based on the device control command, execute the first interaction operation corresponding to the air gesture, and provide the user with corresponding vibration feedback and force feedback operations according to the attributes of the target object and the type of the first interaction operation. When the non-contact target interaction intent is a spatial interaction intent, the intelligent flight interaction device is controlled to perform the second interaction operation corresponding to the air gesture based on the device control command, and the corresponding tactile feedback operation is provided to the user.

5. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 1, characterized in that, The step of performing the corresponding interactive operation based on the target interactive intent and the fused perception data includes: When collaborative devices exist within the target intelligent space, the spatial location and sensing capability information of each collaborative device are acquired, wherein the collaborative devices include display devices and visual acquisition devices; Based on the perception capability information of each collaborative device and the target interaction intention, collaborative perception tasks are assigned to each collaborative device, wherein the display device is responsible for visual and auditory perception from a fixed perspective, the visual acquisition device is responsible for high-precision detail visual perception, and the intelligent flight interaction device is responsible for motion perception and three-dimensional interaction. The system receives sensing data collected by each of the collaborative devices, synchronizes all sensing data in time and aligns them in spatial coordinate system, and then fuses the aligned multi-device sensing data with the fused sensing data from multiple perspectives to generate a global collaborative sensing result.

6. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 1, characterized in that, The digital intelligent space flight interaction method based on multi-sensor perception also includes: When multiple intelligent flight interaction devices are flying in the target digital space, the master control device and slave control device among the multiple intelligent flight interaction devices are predetermined; The main control device plans the formation shape and assigns a corresponding formation position and perception area to each intelligent flight interaction device, and distributes distributed perception tasks according to the perception area of ​​each intelligent flight interaction device. Control each of the intelligent flight interaction devices to fly in formation based on the corresponding formation position, and collect multimodal perception data within their respective perception areas in real time; The master control device receives multimodal sensing data collected by each slave control device and performs distributed data fusion to generate regional sensing results.

7. The digital intelligent space flight interaction method based on multi-sensor perception according to claim 1, characterized in that, The real-time fusion processing of the multimodal sensing data to obtain fused sensing data includes: The Kalman filter algorithm is used to perform multi-source fusion of position and attitude data in the multimodal sensing data to obtain real-time position and attitude information. Based on the real-time position and attitude information, the spatial coordinates of all the multimodal sensing data are transformed to a unified world coordinate system to obtain spatially transformed multimodal sensing data. The NTP protocol is used to perform timestamp synchronization processing on all the multimodal sensing data to obtain time-synchronized multimodal sensing data. The spatially transformed multimodal sensing data and the time-synchronized multimodal sensing data are integrated to generate fused sensing data containing spatiotemporal correlation information.

8. A digital intelligent space flight interaction system based on multi-sensor perception, characterized in that, include: The command response module is used to acquire target control commands issued by the user when the intelligent flight interaction device is flying in the target digital space; The sensing module is used to collect multimodal sensing data through the multi-sensor sensing module on the intelligent flight interaction device during the process of issuing the target control command when the target control command is applied to the intelligent flight interaction device. A multimodal fusion module is used to perform real-time fusion processing on the multimodal sensing data to obtain fused sensing data; The intent recognition and interaction module is used to identify the user's target interaction intent based on the fused perception data, and to perform corresponding interaction operations based on the target interaction intent and the fused perception data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the digital intelligent space flight interaction method based on multi-sensor perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the digital intelligent space flight interaction method based on multi-sensor perception as described in any one of claims 1 to 7.

Images