An immersive multi-display interaction aiming method and system

Abstract

The application belongs to the field of human-computer interaction, and discloses an immersive multi-display interaction aiming method and system. The method comprises the following steps: acquiring a posture angle output by an inertial sensor in a somatosensory device, wherein the posture angle comprises a yaw angle and a pitch angle; constructing a space direction vector according to the posture angle, wherein the space direction vector starts from a fixed reference position in a space; taking the space direction vector as a ray, and performing intersection calculation with multiple display surfaces of an immersive display environment respectively to determine corresponding intersection points of each display surface and distances between the intersection points and the starting point of the ray; selecting a display surface where the nearest intersection point is located as a target display surface according to the distances, and taking the intersection point as a target position of an interaction indication mark; mapping the target position to a display coordinate on the target display surface; and rendering the interaction indication mark on the target display surface. The somatosensory aiming method has low delay and high precision, supports smooth transition across display surfaces, and is suitable for game entertainment and immersive interaction scenes.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and in particular to an immersive multi-display interactive aiming method and system. Background Technology

[0002] With the rapid development of virtual reality and immersive display technologies, human-computer interaction methods are also constantly evolving. Immersive display environments, such as multi-projection surface surround display systems (CAVE systems) and LED display walls, can provide users with an immersive visual experience and have broad application prospects in fields such as gaming, training simulation, and exhibitions. In these application scenarios, how to achieve natural interaction between users and the virtual environment, especially precise aiming interaction, is a key technical issue for improving user experience.

[0003] The existing immersive interaction solutions mainly fall into the following categories.

[0004] The first type is virtual reality head-mounted display solutions. Users wear head-mounted displays, and the system controls the field of view by tracking the user's head movements. The crosshair is usually fixed in the center of the screen. The drawbacks of this type of solution are: users are enclosed in the head-mounted display and cannot perceive the surrounding environment, which can easily cause dizziness after prolonged use; in commercial scenarios with multiple people, there is a lack of social interaction between users, which limits the experience; and the head-mounted display adds additional hardware costs and maintenance burdens.

[0005] The second type is the traditional CAVE projection system. This type of system uses multi-projection to achieve immersive display, supporting synchronized images across multiple projection surfaces through technologies such as nDisplay. However, existing CAVE applications are mainly geared towards passive viewing and controller interaction scenarios, lacking active aiming interaction solutions based on motion-sensing devices, thus preventing users from obtaining an experience similar to real shooting.

[0006] The third type is a laser pointer positioning solution based on an external camera. For example, a laser pointer can be used to track the laser pointer's light point to achieve cross-screen indication. The drawback of this type of solution is that it relies on an external camera for closed-loop tracking, resulting in high system latency and positioning accuracy that is greatly affected by ambient light.

[0007] The fourth category is inertial measurement unit (IMU) motion-sensing mouse or pointer control solutions, such as technologies that use IMUs to implement head-controlled mice or 2D cursor control. While these solutions achieve some motion-sensing functionality, they are all designed for single-screen 2D cursor control scenarios and do not address the issue of crosshair positioning in 3D space across multiple display surfaces. The fifth category is cross-screen cursor transition solutions for multi-screen splicing displays, such as multi-monitor management in Windows operating systems or Microsoft Research's MeetAlive system. These solutions all involve 2D screen coordinate mapping and do not involve the physical continuity transition between non-coplanar display surfaces in 3D space. Summary of the Invention

[0008] The purpose of this invention is to provide an immersive multi-display surface interactive aiming method that solves the technical problem of how to accurately map the posture information of a motion-sensing device to the position of the interactive indicator in an immersive multi-display surface environment, so as to achieve motion-sensing aiming interaction without a head-mounted display device, with low latency and high precision, and support flexible adaptation to various display forms and smooth transition across display surfaces.

[0009] Specifically, the present invention provides an immersive multi-display-surface interactive aiming method for an immersive space containing multiple projection display surfaces, comprising the following steps: S1. Obtain the attitude angles output by the inertial sensor in the motion sensing device, wherein the attitude angles include yaw angle and pitch angle; S2. Construct a spatial direction vector based on the attitude angle. The spatial direction vector starts from a fixed reference position in space, and its direction is determined by the yaw angle and the pitch angle. S3. Using the spatial direction vector as a ray, perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray; S4. Select the display surface where the nearest intersection point is located as the target display surface according to the distance, and use the intersection point as the target location of the interactive indicator in three-dimensional space; S5. Map the target position to display coordinates on the target display surface; S6. Render the interactive indicator on the target display surface at the display coordinates.

[0010] An immersive multi-display interactive aiming system includes: The motion input module includes an inertial sensor for outputting attitude angles; The orientation calculation module is used to determine a spatial orientation vector based on the attitude angle, wherein the spatial orientation vector is constructed from a fixed reference position in space. The intersection positioning module is used to take the spatial direction vector as a ray and perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray. Based on the distance, the display surface where the closest intersection point is located is selected as the target display surface, and the intersection point is used as the target position of the interactive indicator in three-dimensional space. A coordinate mapping module is used to map the target position to display coordinates on the target display surface; A rendering module is used to render the interactive indicator on the target display surface at the display coordinates.

[0011] The beneficial effects of this invention are as follows: By converting the posture angle of the motion-sensing device into a spatial direction vector and calculating its intersection with multiple display surfaces, and setting the starting point of the direction vector as a fixed reference point instead of following the user's head movement, this invention achieves real-time aiming interaction based on the motion-sensing device in a multi-display-surface environment. Compared with existing technologies, users do not need to wear head-mounted displays, eliminating the dizziness caused by the feeling of being enclosed. Users can maintain visual communication with their surroundings, making it suitable for multi-person experiences in commercial venues. Compared with the controller interaction of traditional CAVE systems, the motion-sensing aiming method is more natural and intuitive. The system architecture is not limited to a specific display method. Through parameterized adaptation of the display surface geometry, it can flexibly support various display forms such as front projection, rear projection, LED displays, transparent display layers, curved screens, and spherical screens. The cross-display-surface transition method based on physical spatial continuity presets a transition area at the intersection of adjacent display surfaces and adjusts it according to... The transparency is dynamically adjusted based on the distance from the boundary line, enabling a smooth and continuous transition of interactive indicators between different display surfaces. This avoids the jump and flickering issues common in existing 2D multi-screen solutions. The fixed reference point design ensures that the position and orientation of the reference point are completely fixed, and the movement of interactive indicators is driven solely by the posture angle of the motion sensing device. Unlike the camera-following mode of virtual reality headsets, this design offers lower system complexity and better stability. It naturally supports multi-user scenarios, with each user holding an independent motion sensing input module. The system renders interactive indicators independently for each module, facilitating interactive experiences for multi-user battles or multi-user collaborations. Attached Figure Description

[0012] Figure 1 This is a simplified flowchart of the method of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0014] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0015] Example 1 Please refer to Figure 1 The present invention provides an immersive multi-display-surface interactive aiming method for an immersive space containing multiple projection display surfaces, comprising the following steps: S1. Obtain the attitude angles output by the inertial sensor in the motion sensing device, wherein the attitude angles include yaw angle and pitch angle; As one embodiment, in step S1 of the present invention, the motion-sensing device is an interactive device for a user to hold or wear, which integrates an inertial sensor. The inertial sensor is preferably an inertial measurement unit (IMU), which integrates a three-axis gyroscope and a three-axis accelerometer.

[0016] In practical implementation, the inertial measurement unit (IMU) can be either the InvenSense MPU6050 or MPU9250 chip. These chips integrate a three-axis gyroscope and a three-axis accelerometer (the MPU9250 also integrates a three-axis magnetometer). The gyroscope's measurement range is ±250° / s to ±2000° / s, and the accelerometer's range is ±2g to ±16g. The sampling rate is configurable, up to 8kHz. The microcontroller uses the Espressif Systems ESP32 chip, which features a dual-core processor with a clock speed of up to 240MHz, rich communication interfaces, and low power consumption, making it suitable for battery-powered portable devices.

[0017] The attitude data processing flow is as follows: First, raw gyroscope angular velocity data and accelerometer data are acquired from the inertial measurement unit (IMU). Then, a complementary filtering algorithm or the Mahony algorithm is used for data fusion, fusing the gyroscope and accelerometer data into a quaternion representation of the attitude. Next, the quaternion is converted into Euler angles to obtain the yaw and pitch angles. The yaw angle represents the rotation angle of the motion sensing device in the horizontal plane, and the pitch angle represents the rotation angle of the motion sensing device in the vertical plane. Finally, the attitude angle data is transmitted to the host computer via Bluetooth or low-latency wireless communication. The preferred sampling frequency for the attitude angles is set to 100Hz to 200Hz, which ensures real-time response without excessive data transmission burden.

[0018] S2. Construct a spatial direction vector based on the attitude angle. The spatial direction vector starts from a fixed reference position in space, and its direction is determined by the yaw angle and the pitch angle. As one embodiment, in step S2 of the present invention, the fixed reference position is a fixed point in space from which all directional vectors originate. The specific setting of the fixed reference position is determined according to the system configuration and application scenario. Typical setting methods include: the user's chest area (5cm to 10cm in front of the midpoint of the clavicle), the mounting base position of the motion sensing device (center of the pivot or universal joint fulcrum), the midpoint between the user's eyes, or a pre-designated game standing position in the venue.

[0019] The direction vector is constructed based on the yaw and pitch angles, and the calculation formula is as follows:

[0020] Where yaw represents the yaw angle and pitch represents the pitch angle. dir X、 dir Y、 dir Z represents the three components of the normalized direction vector.

[0021] The angles and coordinates used in the above formulas follow these definitions: a right-handed coordinate system is used, with the Y-axis pointing vertically upwards, the Z-axis pointing straight ahead, and the X-axis pointing horizontally to the right. Yaw is the angle between the projection of the direction vector onto the XZ plane (horizontal plane) and the Z-axis, positive to the right; pitch is the angle between the direction vector and the XZ horizontal plane, positive upwards. All angles are defined relative to the world coordinate system (or site coordinate system), not the motion sensing device's own coordinate system. The yaw angle ranges from -180° to +180°, and the pitch angle ranges from -90° to +90°.

[0022] The direction vector originates from the fixed reference position and points in the user's desired aiming direction. The starting point of the direction vector remains fixed; its direction is changed only by variations in the yaw and pitch angles, thus achieving an attitude control method where the reference point position and orientation are completely fixed.

[0023] S3. Using the spatial direction vector as a ray, perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray; As one embodiment, in step S3 of the present invention, the spatial direction vector constructed above is used as a ray, the starting point of which is the fixed reference position, and the direction is the spatial direction vector. The multiple display surfaces in the immersive display environment are physically existing display media, including but not limited to projection screens, LED displays, etc.

[0024] For each display surface, a mathematical equation for the space containing that surface is established based on its geometric parameters (including surface type, spatial location, size, and normal vector). Then, the ray equation is substituted into the surface's mathematical equation to solve for the coordinates of the intersection point between the ray and the surface. Different intersection algorithms are used for different types of display surfaces. For each valid intersection point, the coordinates of the intersection point and the distance parameter t from the ray's origin to that intersection point are recorded.

[0025] S4. Select the display surface where the nearest intersection point is located as the target display surface according to the distance, and use the intersection point as the target location of the interactive indicator in three-dimensional space; In one embodiment, in step S4 of the present invention, since multiple intersection points of the ray and multiple display surfaces may be obtained in step S3, it is necessary to select one as the final target display surface. The selection principle is: compare the distances of each intersection point to the starting point of the ray (i.e., the fixed reference position), and select the display surface where the closest intersection point is located as the target display surface.

[0026] The physical meaning of this selection logic is: when a user's aiming direction simultaneously passes through multiple display surfaces (e.g., near the intersection of display surfaces), the user actually sees the content on the display surface closest to them. Therefore, the interactive indicator should be displayed on the nearest display surface to align with the user's visual intuition. The nearest intersection point is selected as the target position of the interactive indicator in three-dimensional space.

[0027] S5. Map the target position to display coordinates on the target display surface; In one embodiment, in step S5 of the present invention, the target position is a coordinate point in three-dimensional space, while rendering on the display surface requires two-dimensional pixel coordinates. Therefore, it is necessary to map the target position in three-dimensional space to the two-dimensional coordinate system of the target display surface.

[0028] The basic idea of ​​mapping is to establish a local two-dimensional coordinate system (UV coordinate system) on the target display surface, calculate the projection position of the target position in this local coordinate system, and then convert the projection position into pixel coordinates according to the physical size and pixel resolution of the display surface. The mapping method differs for different types of display surfaces (planar, curved, spherical, etc.).

[0029] S6. Render the interactive indicator on the target display surface at the display coordinates.

[0030] As one embodiment, in step S6 of the present invention, the interactive indicator is a visual element used to indicate the user's aiming position, typically in the form of a crosshair or a dot crosshair. The color, size, style, and other parameters of the crosshair can be dynamically adjusted during runtime.

[0031] Rendering utilizes the game engine's multi-viewport rendering mechanism. For multi-display-plane systems, each display plane corresponds to an independent rendering viewport, and the viewport's resolution is consistent with the display plane's pixel resolution. During rendering, rendering commands with interactive indicators are sent to the corresponding viewport based on the target display plane's number. The crosshair can be rendered using a user interface (UI) component overlaid on the game screen, ensuring the crosshair is always visible and unaffected by depth testing.

[0032] For example, suppose in a four-sided projection immersive gaming environment, the user stands in the center of the field, with a fixed reference position set in front of the user's chest (world coordinates P0=(0, 1.5, 0)). The user holds a motion-sensing gun, with the muzzle pointing slightly upwards and directly forward. The attitude angles output by the inertial sensor are: yaw=0°, pitch=15°.

[0033] After obtaining the attitude angle in step S1, step S2 calculates the direction vector according to the formula: dirX=cos(15°)×cos(0°)≈0.966, dirY=sin(15°)≈0.259, dirZ=cos(15°)×sin(0°)=0, that is, the direction vector is (0.966,0.259, 0).

[0034] Step S3: Using this direction vector as a ray, find its intersections with the four display surfaces: front, rear, left, and right. The front display surface is located in the positive Z direction, and the ray intersects it at a point on the front display surface; the rear, left, and right display surfaces do not intersect the ray (or their intersections are behind the ray's origin). Step S4: Select the front display surface as the target display surface, and the intersection point is the aiming point on the front display surface. Step S5: Map this three-dimensional intersection point to the pixel coordinates of the front display surface (e.g., (960, 540)). Step S6: Render a red crosshair at the (960, 540) pixel position on the front display surface. When the user observes through the scope of the motion-sensing gun, the crosshair falls precisely on the user's aiming position on the front display surface, achieving precise motion-sensing aiming interaction.

[0035] Example 2 The construction of the spatial direction vector in step S2 includes the following sub-steps: S21. The raw data output by the inertial sensor is fused into an attitude quaternion q=(q0, q1, q2, q3); As one embodiment, in step S21 of the present invention, the raw data output by the inertial sensor includes angular velocity data from the three-axis gyroscope and acceleration data from the three-axis accelerometer. This raw data cannot be directly used to represent attitude and needs to be converted into attitude quaternions through a data fusion algorithm.

[0036] Specifically, complementary filtering or Mahony algorithm is used for data fusion. Taking the Mahony algorithm as an example, its basic principle is as follows: First, the direction of gravity is estimated using accelerometer data, and the error between the current attitude and the direction of gravity is calculated; then, this error is used to correct the angular velocity data of the gyroscope, eliminating the zero-bias drift of the gyroscope; finally, the attitude quaternion is updated through quaternion differential equations. The quaternion q=(q0, q1, q2, q3) is a unit quaternion that satisfies q0²+q1²+q2²+q3²=1, and is used to describe the orientation of the motion sensing device in three-dimensional space.

[0037] S22. Calculate the spatial direction vector based on the attitude quaternion, using at least one of the following methods: The attitude quaternion is converted into the yaw angle and the pitch angle according to the ZYX internal rotation order, and then the spatial direction vector is calculated using trigonometric formulas based on the yaw angle and the pitch angle. The spatial direction vector is calculated directly using quaternion rotation: dir world = q · dir local · q - ¹, where dir local Let q be the initial direction vector in the local coordinate system. - ¹ represents the conjugate of the attitude quaternion, dir world The spatial direction vector; As one embodiment, in step S22 of the present invention, the spatial direction vector is calculated based on the attitude quaternion in the following two ways: Method 1: Convert the attitude quaternion into the yaw angle and pitch angle according to the ZYX rotation order, and then calculate the spatial direction vector using trigonometric formulas based on the yaw angle and pitch angle.

[0038] The conversion formula from quaternions to Euler angles is as follows (ZYX internal rotation order):

[0039] After obtaining the yaw and pitch angles, the direction vector is then calculated.

[0040] Method 2: Directly calculate the spatial direction vector using quaternion rotation:

[0041] in, Let q be the initial direction vector in the local coordinate system (usually (0, 0, 1), i.e., directly in front along the Z-axis), and let q be the attitude quaternion. - ¹ is its conjugate (for a unit quaternion, the conjugate is equal to the inverse). Let be the spatial direction vector.

[0042] The quaternion direct rotation method avoids the gimbal lock problem of Euler angles and maintains numerical stability across the entire attitude range, making it a preferred embodiment of the present invention.

[0043] S23. Normalize the spatial direction vector.

[0044] As one embodiment, in step S23 of the present invention, the direction vector calculated in step S22 is divided by its magnitude to make it a unit vector. The normalized direction vector satisfies This ensures the consistency of the ray direction.

[0045] Taking Method 2 as an example, assume the attitude quaternion q = (0.966, 0, 0.259, 0), representing a 30° rotation around the X-axis (30° pitch angle). The local initial direction vector dir_local = (0, 0, 1). Calculate using the quaternion rotation formula:

[0046] Calculated =(0, sin30°, cos30°)=(0, 0.5, 0.866), meaning the direction vector points 30° upwards and slightly forward. This result is consistent with the result calculated using the Euler angle formula, but the quaternion method avoids the accuracy loss during angle conversion.

[0047] Example 3 The fixed reference position is set in at least one of the following positions: User's chest area; The mounting base location for the motion sensing device is specifically the center of the device's rotating shaft or the universal joint fulcrum. The midpoint between the user's eyes; The game's designated positions are preset in the game area.

[0048] As one embodiment, the specific method of setting the fixed reference position in this invention varies depending on the different system application scenarios, and will be described separately below: (1) User's chest area. For a standing immersive shooting experience, the fixed reference position is usually set in the user's chest area, specifically 5cm to 10cm in front of the midpoint of the collarbone, corresponding to coordinates of approximately (0, 1.5m, 0) in three-dimensional space (with the projection of the midpoint of the user's feet on the ground as the origin of the world coordinate system, the Y-axis pointing vertically upwards, and the Z-axis pointing directly forward). This position is close to the midline of the user's body and at the same height as the user's chest, making the aiming mapping relationship most ergonomic and intuitive.

[0049] (2) Mounting base position of the motion sensing device. For scenarios where the motion sensing gun is mounted on a fixed bracket or motion platform, the fixed reference position is set as the center of the motion sensing device's rotation axis or the universal joint pivot point. This position remains fixed in space, and the motion sensing device performs pitch and yaw rotations around this point. With this setup, the motion trajectory of the motion sensing device is a rotation around a fixed point, and the starting point of the direction vector coincides with the physical rotation center of the device, resulting in the simplest kinematic model.

[0050] (3) The midpoint between the user's eyes. For scenarios requiring more precise viewpoint matching, the fixed reference position can be set at the midpoint between the user's eyes (i.e., the root of the nose), at a height of approximately 1.6m to 1.7m. This position coincides with the user's visual focus, resulting in the highest aiming accuracy, but requires additional head tracking equipment to obtain head position information.

[0051] (4) Pre-set game station designation points in the venue. For large multiplayer venues, fixed reference positions can be pre-set at designated points for each game station according to the venue layout. All players' reference points are uniformly marked, which facilitates system management and calibration.

[0052] For example, in a standard four-sided projection immersive shooting experience hall, the player's standing position is marked on the ground in the center of the arena. The system defaults the world coordinates of this position to (0, 0, 0) (ground projection point), and a fixed reference position is set 1.5m directly above this point, i.e., (0, 1.5m, 0). All players' motion sensing devices use this point as the starting point of their direction vector. Regardless of the player's height, the aiming mapping relationship remains consistent, ensuring a consistent experience for multiple players.

[0053] Example 4 The intersection calculation in step S3 includes the following sub-steps: S31. Obtain the geometric parameters of each display surface. The geometric parameters include at least the type, spatial position, size, and normal vector of the display surface. The type of the display surface includes at least one of planar display surface, curved display surface, spherical display surface, and hemispherical display surface. As one embodiment, in step S31 of the present invention, the geometric parameters of the display surface are obtained through a calibration process. The calibration methods include the following: Manual measurement and data entry: The size, position, and orientation angle of each display surface are measured using tools such as a tape measure and laser rangefinder, and the measurement data is manually entered into the system. This method is suitable for simple flat panel display systems, and the accuracy depends on the operator's measurement skill, typically at the centimeter level.

[0054] Automatic calculation using visual calibration algorithms: A camera or depth camera is used to capture the calibration pattern (such as a checkerboard or AprilTag marker) on the display surface. Computer vision algorithms then automatically calculate the position and normal vector of the display surface in space. Common methods include Zhang Zhengyou calibration and homography matrix decomposition. This method offers high accuracy (millimeter-level) and is the mainstream solution for commercial systems.

[0055] CAD Model Import: Directly imports the display surface geometric parameters from CAD models exported from 3D design software into the system. Offers the highest accuracy and is suitable for the design phase of new venues.

[0056] Infrared tracking calibration: Using an infrared tracking system, the calibration tool is positioned at different locations on the display surface, and the plane equation of the display surface is fitted by the coordinates of the marked points on the tracking tool. The accuracy can reach the sub-millimeter level.

[0057] Among the geometric parameters, the type of the display surface determines the intersection algorithm used subsequently; the spatial position (such as the D value in the plane equation, the coordinates of the center of the sphere) determines the intersection relationship between the ray and the display surface; the dimensions (width, height, radius, etc.) determine the range of UV coordinate mapping; and the normal vector (for a plane) determines the denominator of the intersection of the ray and the plane.

[0058] S32. Select the corresponding intersection algorithm according to the type of the display surface; S33. Perform the corresponding intersection algorithm on each display surface to obtain the coordinates of the intersection point of the spatial direction vector with each display surface and the intersection point parameter t, where t represents the distance parameter from the fixed reference position to the intersection point; In one embodiment, in step S33 of the present invention, the ray equation is substituted into the mathematical equation of the display surface to obtain the intersection parameter t. For a planar display surface, t is a scalar parameter in the ray parameter equation; for a spherical display surface, t is a solution to a quadratic equation. The physical meaning of t is the distance from the ray origin (fixed reference position) along the ray direction to the intersection point.

[0059] S34. Select intersections where t>0 as valid intersections.

[0060] As one embodiment, in step S34 of the present invention, t>0 indicates that the intersection point is in front of the ray (i.e., reachable from the fixed reference position along the aiming direction), t<0 indicates that the intersection point is behind the ray (i.e., the user needs to turn their head to see it), and t=0 indicates that the intersection point is at the reference point position (which will not happen in reality). Only when t>0 is the intersection point actually aimed at by the user. When t=0 or t<0, the intersection point is invalid and excluded.

[0061] For example, in a four-view projection system, the geometric parameters of the four display surfaces are pre-calibrated. The plane equation of the front display surface is z=3m (normal vector is (0,0,1)), the plane equation of the rear display surface is z=-3m (normal vector is (0,0,-1)), the plane equation of the left display surface is x=-3m (normal vector is (-1,0,0)), and the plane equation of the right display surface is x=3m (normal vector is (1,0,0)). The fixed reference position is P0=(0,1.5,0), and the direction vector is dir=(0.5,0,0.866) (pointing to the right front).

[0062] For the front display surface: Substituting into the ray equation P=P0+t·dir, we get t=(3-0) / 0.866≈3.46>0, which is valid.

[0063] For the right-side display: the solution t=(3-0) / 0.5=6>0, which is valid.

[0064] For the rear display surface: the Z component of the ray direction is positive, and it is impossible for it to intersect with z=-3m, so there is no solution.

[0065] For the left display surface: the X component of the ray direction is positive, and it is impossible for it to intersect with x=-3m, so there is no solution.

[0066] Step S34 filters out two valid intersection points (front display surface and right display surface), and subsequent step S4 selects the closest one (front display surface, t=3.46) as the target display surface.

[0067] Example 5 The intersection algorithm described in step S32 includes at least one of the following: If the display surface is a planar display surface, then the intersection algorithm of a ray and a plane is used, and the calculation formula is as follows:

[0068] in,( A , B , C () is the unit normal vector of the display surface. D The signed distance from the origin to the display surface. P 0=( , , ) is the fixed reference position. dir =( , , ) is the spatial direction vector; If the display surface is an arc surface, then the arc surface is discretized into multiple planar segments and the intersection calculation of the ray and the plane is performed respectively, or the intersection algorithm of the ray and the parameterized arc surface is used. If the display surface is a spherical or hemispherical display surface, then the intersection algorithm between the ray and the sphere is used, and the calculation formula is as follows:

[0069] in,( , , ( ) represents the coordinates of the sphere's center. R Let t be the radius of the sphere. Solve the quadratic equation in one variable and take the solution with the smallest value of t that is greater than zero as the intersection point parameter.

[0070] As one embodiment, the intersection algorithm includes at least one of the following: (i) If the display surface is a planar display surface, then the intersection algorithm of ray and plane is used.

[0071] As one embodiment, for a planar display surface, let the plane equation be:

[0072] Where (A, B, C) is the unit normal vector of the plane (satisfying A² + B² + C² = 1), and D is the signed distance from the origin to the plane. The ray originates at P0 = ( , , ), the direction vector is dir=( , , The parametric equation for the ray is P = P0 + t·dir. Substituting the ray equation into the plane equation, we get:

[0073] Where the denominator A· +B· +C· This is the dot product of the ray's direction vector and the plane's normal vector. When the denominator is 0, the ray is parallel to (or coplanar with) the plane, with no intersection points or infinitely many intersection points; in this case, the calculation of that display plane is skipped.

[0074] Explanation of parameters: (A,B,C) in the plane equation is the unit normal vector, which is dimensionless; the unit of D is meters (m), representing the signed distance from the origin to the plane; , , The unit is meter (m); , , The normalized direction vector is dimensionless; the unit of t is meters (m), representing the distance from the ray's origin to the intersection point.

[0075] For example, the plane equation of the front display surface is z-3=0, i.e., A=0, B=0, C=1, D=-3. The fixed reference position is P0=(0,1.5,0), and the direction vector is dir=(0.5,0,0.866). Substituting into the formula:

[0076] The intersection point has coordinates P=(0+0.5×3.46, 1.5+0×3.46, 0+0.866×3.46)=(1.73, 1.5,3.0), and is located on the front display surface.

[0077] (ii) If the display surface is an arc surface, the arc surface is discretized into multiple planar segments and then the intersection calculation of the ray and the plane is performed respectively, or the intersection algorithm of the ray and the parameterized arc surface is used.

[0078] As one example, there are two processing methods for curved display surfaces: Method 1 (Discretization Method): Divide the arc surface into N planar segments along the horizontal direction (the value of N depends on the curvature of the arc surface and the accuracy requirements, typically 10-50). Each segment is approximated by a tangent plane at its center. Then, perform ray-plane intersection calculations on each planar segment and take the intersection point with the smallest t>0 as the final intersection point.

[0079] Method 2 (Parametric Method): Taking a cylindrical surface as an example. Assume the axis of the cylindrical surface is parallel to the Z-axis, the center is located at (Xc, Yc), and the radius is R. The equation of the cylindrical surface is:

[0080] Substituting the ray equation P = P0 + t·dir into the cylindrical surface equation, we obtain a quadratic equation in t:

[0081] Solve the equation and take the solution with the smallest t value that is greater than zero as the intersection point parameter.

[0082] For example, suppose the curved display surface is part of a cylindrical surface, with its center at (0, 0), radius R = 4m, and axis along the Z direction. Fix the reference position P0 = (0, 1.5, 0), and the direction vector dir = (1, 0, 0) (horizontally to the right). Substitute into the cylindrical surface equation:

[0083] The intersection point is P=(0+1×3.708, 1.5+0, 0)=(3.708, 1.5, 0), located on the curved display surface.

[0084] (iii) If the display surface is a spherical display surface or a hemispherical display surface, the intersection algorithm of the ray and the sphere shall be used.

[0085] As one embodiment, for a spherical display surface, let the coordinates of the sphere's center be (Cx, Cy, Cz), and the radius of the sphere be R. The equation of the sphere is:

[0086] Substituting the ray equation P = P0 + t·dir into the sphere equation, we obtain a quadratic equation in t:

[0087] Since dir is a unit direction vector The above formula can be simplified to:

[0088] This quadratic equation may have 0, 1, or 2 real solutions: When the discriminant Δ < 0, the ray has no intersection with the sphere. When Δ=0, the ray is tangent to the sphere (1 intersection point); When Δ>0, the ray intersects the sphere at two points. The solution with the smallest t value greater than zero is taken as the target intersection point (i.e., the first point where the ray enters the sphere).

[0089] For example, suppose the center of the spherical display surface is at C=(0, 0, 0), and the radius is R=5m. Fix the reference position P0=(0, 1.5, 3) (the user stands in front of the center of the sphere), and the direction vector dir=(0, -0.5, 0.866) (pointing diagonally downwards). Substitute into the spherical equation:

[0090] Solving for t, we get t≈2.377m (another solution is negative and is discarded). The intersection point is P=(0+0, 1.5+(-0.5)×2.377, 3+0.866×2.377)=(0, 0.312, 5.058), located on the spherical display surface.

[0091] Example 6 For flat or curved display surfaces, the mapping in step S5 includes the following sub-steps: S511. Establish a local coordinate system on the target display surface, wherein the U-axis of the local coordinate system is parallel to the horizontal direction of the target display surface, and the V-axis is parallel to the vertical direction of the target display surface; As one embodiment, in step S511 of the present invention, a two-dimensional local coordinate system is established on the selected target display surface. For a planar display surface, the U-axis is along the horizontal edge of the display surface, and the V-axis is along the vertical edge of the display surface. For a curved display surface, the U-axis is along the horizontal arc direction of the curved surface (i.e., the generatrix direction of the curved surface), and the V-axis is along the vertical direction of the curved surface (i.e., the straight generatrix direction of the curved surface). The origin of the local coordinate system is usually set at the lower left or upper left corner of the display surface.

[0092] S512. Calculate the offset of the target position in the local coordinate system, and project the offset along the U-axis and V-axis directions of the target display surface to obtain the projection values ​​u and v; As one embodiment, in step S512 of the present invention, let the coordinates of the lower left corner of the display surface in three-dimensional space be P_origin, the U-axis direction vector be U_vec (unit vector), and the V-axis direction vector be V_vec (unit vector). The target position is P_target, then the offset is:

[0093] The projection value is calculated as follows:

[0094] The units for u and v are meters (m), representing the distance of the target position on the display surface along the U-axis and V-axis directions.

[0095] S513. Based on the width and height of the target display surface, normalize the projection value into UV coordinates:

[0096] Where width and height represent the physical width and height of the display surface, in meters (m). Normalized UV coordinates. and It is a dimensionless value, ranging from 0 to 1. =0 indicates the left edge of the displayed surface. =1 indicates the right edge; =0 indicates the bottom edge of the displayed surface. =1 indicates the top edge.

[0097] S514. Map the normalized UV coordinates to the screen pixel coordinates of the target display surface:

[0098] in screen_width and screen_height The rendering resolution corresponding to the target display surface.

[0099] For example, suppose the target display surface is a frontal plane display surface with width = 4m and height = 3m, and the coordinate of the lower left corner is P. origin =(-2, 0, 3), U vec =(1, 0, 0) (horizontally to the right), V vec =(0, 1, 0) (vertically upward). Target position P target =(0.5, 1.2, 3).

[0100] Offset = (0.5 - (-2), 1.2 - 0, 3 - 3) = (2.5, 1.2, 0). Projected values ​​u = dot((2.5, 1.2, 0), (1, 0, 0)) = 2.5m, v = dot((2.5, 1.2, 0), (0, 1, 0)) = 1.2m. Normalized UV coordinates: =2.5 / 4=0.625, =1.2 / 3=0.4.

[0101] If the display surface rendering resolution is 1920×1080, then the pixel coordinates are: =0.625×1920=1200, =0.4×1080=432.

[0102] Example 7 For spherical or hemispherical display surfaces, the mapping in step S5 includes the following sub-steps: S521. Establish a local spherical coordinate system with the center of the sphere as the origin; As one embodiment, in step S521 of the present invention, a local spherical coordinate system is established with the center of the sphere as the origin for the spherical display surface. The reference direction of the spherical coordinate system (0° longitude direction) is usually set to the horizontal axis of symmetry of the spherical display surface (such as directly in front), and the equatorial plane is a horizontal plane passing through the center of the sphere.

[0103] S522. Calculate the latitude angle φ and longitude angle θ of the target position P in the local spherical coordinate system. The latitude angle φ is the angle between the line connecting the target position and the center of the sphere and the equatorial plane. The longitude angle θ is the angle between the projection of the target position on the equatorial plane and the reference direction. As one embodiment, in step S522 of the present invention, let the coordinates of the target position P in the spherical coordinate system be ( , , The center of the sphere is at C=( , , First, calculate the vector relative to the center of the sphere:

[0104] The latitude angle φ is calculated as follows:

[0105] Where |V| is the magnitude of vector V. The latitude angle φ ranges from -90° to +90°, φ=0° indicates that it is on the equatorial plane, φ=+90° indicates that it is directly above the sphere (North Pole), and φ=-90° indicates that it is directly below the sphere (South Pole).

[0106] The longitude angle θ is calculated as follows:

[0107] in arctan 2 represents the arctangent function in the four quadrants. The longitude angle θ ranges from 0° to 360°, representing the angle between the projection of the target position onto the equatorial plane and the reference direction (usually the positive Z direction).

[0108] S523. Normalize the latitude and longitude angles to UV coordinates:

[0109] Normalized UV coordinates and These are dimensionless values, ranging from 0 to 1. =0 indicates the direction of longitude 0°. =1 indicates the 360° longitude direction (coinciding with 0°); =0 indicates that the sphere is directly below (South Pole, φ=-90°). =0.5 represents the equator (φ=0°). =1 indicates the top of the sphere (North Pole, φ=+90°).

[0110] S524. Map the normalized UV coordinates to the screen pixel coordinates of the spherical or hemispherical display surface:

[0111] Where screen_width and screen_height are the rendering resolutions corresponding to the target display surface.

[0112] For example, suppose the center of the spherical display surface is at C=(0, 0, 0), and the radius is R=5m. The target position is P=(3, 2, 4). The vector relative to the center of the sphere is V=(3, 2, 4), and |V|=√(9+4+16)=√29≈5.385.

[0113] Latitude angle φ = asin(2 / 5.385) ≈ asin(0.371) ≈ 21.8°. Longitude angle θ = atan2(4, 3) ≈ 53.13°.

[0114] Normalized UV coordinates: =53.13 / 360≈0.148, =(90+21.8) / 180≈0.621.

[0115] If the rendering resolution of the spherical display surface is 3840×2160, then the pixel coordinates are: pixelX =0.148×3840≈568, pixelY =0.621×2160≈1341.

[0116] Example 8 Step S514 is followed by a perspective distortion correction sub-step: S515. Obtain the distortion parameters of the projection device corresponding to the target display surface, wherein the distortion parameters include radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2; As one embodiment, in step S515 of the present invention, for a display surface formed by a projection device (such as a projector), the projection lens will introduce optical distortion, causing geometric deformation of the projected image and affecting the positioning accuracy of the interactive indicator. The distortion parameters are obtained through the projector calibration process.

[0117] The distortion coefficients are automatically calculated using a calibration algorithm (such as the Zhang Zhengyou calibration method), eliminating the need for manual input. For ultra-short-throw projectors, the typical value range for k1 is -0.5 to -0.2 (barrel distortion, negative), for k2 it is -0.1 to 0.1, and for k3 it is a higher-order correction term (usually small, which can be set to 0). p1 and p2 are the tangential distortion coefficients, typically ranging from -0.05 to 0.05. For most commercial projectors, tangential distortion is much smaller than radial distortion and can be ignored when high precision is not required.

[0118] S516. Apply the distortion correction mapping function to the normalized UV coordinates:

[0119]

[0120] in, r ² = + ; The aforementioned polynomial model (Brown-Conrady model) is suitable for conventional wide-angle and ultra-short-throw projection lenses. For fisheye lenses (such as fisheye projectors used in spherical screens), distortion becomes more severe, requiring the use of specialized fisheye distortion models such as equidistant projection and equisolid angle projection for correction. In these cases, the polynomial model is no longer applicable.

[0121] S517, with the corrected coordinates and Alternative and Perform the mapping in step S514.

[0122] Suppose the distortion coefficients obtained from the calibration of a certain ultra-short throw projector are: k1=-0.35, k2=0.08, k3=0, p1=0.01, p2=-0.005. The normalized coordinates of a certain aiming point are... =0.5, =0.3 (When the image center is taken as the origin, the actual range is -1 to 1. Here, we assume it to be 0.5 and 0.3, which is the direction of deviation from the center to the upper right).

[0123] r²=0.5²+0.3²=0.25+0.09=0.34.

[0124] Corrected U coordinates: =0.5×(1-0.35×0.34+0.08×0.1156)+0.01×(0.34+2×0.25)+2×(-0.005)×0.5×0.3=0.5×(1-0.119+0.00925)+0.01×(0.34+0.5)-0.0015=0.5×0.89025+0.0084-0.0015≈0.445+0.0069≈0.452.

[0125] Corrected V-coordinates: =0.3×(1-0.119+0.00925)+2×0.01×0.5×0.3+(-0.005)×(0.34+2×0.09)=0.3×0.89025+0.003-0.005×0.52≈0.267+0.003-0.0026≈0.267.

[0126] After correction, the rendering position of the point was moved from (0.5, 0.3) to approximately (0.452, 0.267), compensating for the barrel distortion of the projection lens.

[0127] Example 9 The rendering in step S6 employs at least one of the following methods: The interactive indicator is overlaid and rendered at the screen pixel coordinates of the target display surface using a two-dimensional user interface. A three-dimensional crosshair model is placed at the three-dimensional spatial coordinates of the target location for rendering using a three-dimensional spatial rendering method.

[0128] As one embodiment, rendering is performed using at least one of the following methods: (i) The interactive indicator is overlaid and rendered at the screen pixel coordinates of the target display surface in a two-dimensional user interface manner.

[0129] As one embodiment, the two-dimensional user interface (UI) rendering method in this invention renders the interactive indicator as a 2D UI element, overlaid on top of the 3D scene. Specifically, in a game engine (such as Unreal Engine or Unity), a UI widget is created, which contains only a crosshair image. Each frame, the position of this UI widget is updated based on calculated screen pixel coordinates to ensure the crosshair is always displayed in the correct position. The advantages of this method are its simplicity, low performance overhead, and consistently clear crosshair unaffected by scene depth, making it suitable for scenarios requiring high aiming accuracy, such as competitive shooting games.

[0130] (ii) Using a three-dimensional spatial rendering method, a three-dimensional crosshair model is placed at the three-dimensional spatial coordinates of the target location for rendering.

[0131] As one embodiment, the 3D spatial rendering method in this invention creates an independent 3D model in the 3D scene as an interactive indicator. This model is placed at the intersection of the ray and the display surface (slightly offset towards the user by 0.05m to 0.1m to avoid depth conflicts caused by overlap with the display surface). Typical forms of the crosshair model include a floating crosshair, a floating light spot, and a three-dimensional crosshair (such as a triangular pyramid or tetrahedron). The size of the crosshair model is dynamically adjusted according to the distance from the user to ensure visual consistency when viewed on the display surface at different distances.

[0132] This invention can support two rendering methods simultaneously, which can be dynamically switched according to the scene or selected by the developer.

[0133] For example, in a sci-fi immersive shooting game, the developers chose a 3D spatial rendering method. When the player aims at an enemy on the display surface, the system places a glowing blue 3D crosshair model at the intersection of the ray and the display surface (3D coordinates (1.5, 2.0, 3.0)). This crosshair is slightly offset by 0.05m towards the player (to (1.5, 2.0, 2.95)) to avoid overlapping with the display surface and causing flickering. When the player rotates the motion-sensor gun to change the aiming direction, the crosshair smoothly moves in 3D space following the intersection point, presenting a visual effect of floating in the air, enhancing the game's immersion.

[0134] Example 10 The method also includes a cross-display plane transition step S7, as follows: S71. A transition area is preset at the junction of adjacent display surfaces. For a planar display surface, the width of the transition area is a preset proportional range of the width of the display surface; for a curved or spherical display surface, the width of the transition area is a preset proportional range of the arc dimension of the display surface in the direction perpendicular to the junction line at the junction. As one embodiment, in step S71 of the present invention, a transition region is provided at the junction of adjacent display surfaces to enable smooth switching of interactive indicators from one display surface to another. The width of the transition region is set to 3% to 10% of the display surface size, with a typical value of 5%. For flat display surfaces, the width of the transition region is 5% of the display surface width; for curved display surfaces, the width of the transition region is 5% of the curved surface size of the display surfaces at the junction along the perpendicular direction of the junction line.

[0135] S72. Determine whether the interactive indicator has entered the transition area. If it has not entered, render the interactive indicator with full transparency only on the current display surface. As one embodiment, in step S72 of the present invention, the system calculates the UV coordinates of the interactive indicator on the current display surface every frame and determines whether it is located within the transition area (i.e., the distance from the boundary line is less than the width W of the transition area). If it does not enter the transition area, the interactive indicator is rendered only on the current display surface with a transparency α=1 (completely opaque).

[0136] S73. If the interactive indicator enters the transition area, the same direction vector is used to calculate the intersection with the current display surface and the adjacent display surface, respectively, to obtain two independent world coordinates of the intersection point, which are then mapped to the screen pixel coordinates of the corresponding display surface. When the distance between the intersection point and the corresponding boundary line is less than a preset threshold, the interactive indicator is rendered simultaneously on the current display surface and the adjacent display surface. The transparency of the interactive indicator on the current display surface is determined according to the distance d from the boundary line and the width W of the transition area in the following manner:

[0137] The transparency of interactive indicator marks on adjacent display surfaces is determined as follows:

[0138] As one embodiment, in step S73 of the present invention, when the interactive indicator enters the transition area, the cross-display surface transition logic is triggered. At this time, the same ray (the same direction vector originating from the same fixed reference position) is subjected to intersection calculations with the current display surface and the adjacent display surface, respectively, resulting in two independent intersection points—one on the current display surface and the other on the adjacent display surface. Both intersection points are located on the same ray, but on different display surfaces.

[0139] The world coordinates of the two intersection points are mapped to the screen pixel coordinates of their respective display surfaces. Then, interactive indicator icons are rendered simultaneously on both display surfaces, with their transparency dynamically adjusted based on their distance from the boundary line. Transparency of interactive indicator icons on the current display surface:

[0140] Transparency of interactive indicator marks on adjacent display surfaces:

[0141] Where d is the distance from the interactive indicator to the boundary line (measured on the current display surface), and W is the width of the transition area.

[0142] When d=0 (it is exactly on the boundary line). =1, =0, the crosshair is fully displayed on the current screen.

[0143] When d=W / 2 (middle of the transition region). =0.5, =0.5, the crosshair displays half transparency on each of the two display surfaces.

[0144] When d=W (it is about to completely enter the adjacent display surface) =0, =1, the crosshair is fully displayed on the adjacent display surface.

[0145] S74, when the interactive indicator fully enters the adjacent display surface, that is... When =0, rendering stops on the current display surface, and the interactive indicator is rendered only on the adjacent display surface with full transparency.

[0146] As one embodiment, in step S74 of the present invention, when the transparency of the interactive indicator is... When the value drops to 0, it indicates that the crosshair has completely entered the adjacent display surface. At this point, rendering on the current display surface stops, and interactive indicator markers are rendered only on the adjacent display surface with α=1 (completely opaque), completing a smooth transition across display surfaces.

[0147] For example, in a four-view projection system, the boundary line between the front and right display surfaces extends vertically. The transition area width W = 0.2m (5% of the 4m width of the front display surface). The user's aiming direction gradually rotates to the right from directly in front.

[0148] When the crosshair is centered on the front display plane, d>W. =1, only the front view displays the crosshair.

[0149] When the crosshair moves to a distance of d=0.1m from the boundary line (entering the transition zone). =1-0.1 / 0.2=0.5, α_adjacent=0.1 / 0.2=0.5, and a semi-transparent crosshair is displayed on each of the two display surfaces.

[0150] When the crosshair moves to a distance of d=0.05m from the boundary line, =0.75, =0.25, the crosshair on the front display is brighter, and the crosshair on the right display is darker.

[0151] When the crosshair crosses the boundary line and enters the right display plane at d=0.15m (measured from the front, it has crossed the boundary line, but measured from the right side, it is 0.15m), =0.25, =0.75, the crosshair on the right side of the display is brighter.

[0152] When the crosshair is fully inside the right-side display plane (d>W) =0, stopping rendering on the front display plane and displaying the complete crosshair only on the right display plane. Throughout the process, the crosshair smoothly transitions between the two display planes, and the user will not perceive any jumps or flickers.

[0153] Example 11 The multiple display surfaces are arranged in a surrounding manner around the user, including at least one of the following arrangements: The four display surfaces are located in front of, behind, to the left and to the right of the user, respectively; Five display surfaces, including a display surface located above the user in addition to the four display surfaces; The system has six display surfaces, including five additional surfaces below the user, creating a complete surround effect.

[0154] As one example, the specific details are as follows: (a) Four display surfaces, located in front of, behind, to the left and to the right of the user.

[0155] As one embodiment, the present invention provides a panoramic view of the user in the horizontal direction using four display surfaces. The size of each display surface can be customized according to the actual site conditions, with typical dimensions being 3 to 5 meters in width and 2.5 to 3 meters in height. The four display surfaces are located in front of the user (positive Z direction), behind them (negative Z direction), to their left (negative X direction), and to their right (positive X direction), respectively, forming a 360° surround view in the horizontal direction.

[0156] (ii) Five display surfaces, including a display surface located above the user in addition to the four display surfaces.

[0157] In one embodiment, this invention adds a top display surface to the four horizontal display surfaces, forming a panoramic view encompassing the user's horizontal and overhead space. The top display surface is planar or nearly planar in shape, and its size covers the field of view above the user's head, typically 4 meters × 4 meters.

[0158] (iii) Six display surfaces: In addition to the five display surfaces, there is also a display surface located below the user, forming a complete surround for the user.

[0159] As one embodiment, the six display surfaces in this invention form a complete spatial enclosure for the user (i.e., the six-sided configuration of the CAVE system). The bottom display surface can be implemented using a translucent grid floor or embedded LED floor tiles, which can both support the user standing upright and display virtual environment content.

[0160] Example 12 The plurality of display surfaces are formed by at least one of the following devices: front projection projection device, rear projection projection device, and LED display screen.

[0161] Specifically: (a) Front-projection equipment. The projection equipment is located in front of the display surface (i.e., on the same side as the user), and the projection light path projects onto the display surface from the front. The advantages of front-projection mode are simple installation and low cost, making it suitable for most immersive experience venues. The preferred projection equipment is an ultra-short-throw laser projector with a throw ratio of less than 0.4, typically 0.25 to 0.35, placed at the bottom edge of the corresponding display surface to reduce user obstruction of the projection light path.

[0162] (ii) Rear projection equipment. The projection equipment is located behind the display surface (i.e., on the opposite side from the user), and the user is located between the projection equipment and the display surface. The projection light path is projected from behind the display surface onto the screen, and the user watches in front of the screen without being affected by the projection equipment blocking the light path. Rear projection mode is suitable for commercial venues with high traffic, avoiding the safety hazards of projection light shining directly into people's eyes.

[0163] (III) LED Display Screen. Composed of multiple LED modules, forming a large-area display wall. The pixel pitch of each LED module can be selected according to viewing distance and resolution requirements, typically ranging from P2 to P5. LED displays are self-illuminating, requiring no projection light path or projection screen, offering high brightness (up to 1000 cd / m² or more) and high refresh rate (up to 3840Hz), making them suitable for scenarios with high ambient light conditions and high refresh rate requirements.

[0164] Example 13 The display surface also includes a transparent display layer disposed in front of the projection surface projected by the projection device, the transparency of the transparent display layer being adjustable within a preset range; The rendering described in step S6 includes two-layer rendering: The three-dimensional scene content is rendered on the projection surface as the main display layer; At least a portion of the interactive indicator and two-dimensional enhancement information are rendered on the transparent display layer as an auxiliary display layer; The display coordinates of the main display layer and the auxiliary display layer are aligned to ensure spatial consistency.

[0165] In one embodiment, the transparent display layer of this invention is positioned in front of the main display layer (projection screen or LED screen), typically at a distance of 0.5 meters to 2 meters, for overlaying and displaying enhanced information. The transparent display layer can be implemented using a transparent OLED display, a transparent LCD screen, or a transparent glass projection film, etc. The transparency of the transparent display layer is typically adjusted between 50% and 85%, allowing for the display of enhanced information without completely obscuring the content of the main display layer behind it.

[0166] The specific method of dual-layer rendering is as follows: The main display layer (projection plane) renders the 3D scene content. It contains the 3D graphics of the game scene, including virtual environments, characters, and objects, but does not include the crosshair itself.

[0167] At least a portion of the interactive indicator and two-dimensional enhancement information are rendered on the transparent display layer as an auxiliary display layer. The transparent display layer carries two-dimensional enhancement information such as the crosshair, HUD elements (ammunition quantity, health, score, etc.), minimap, and mission prompts.

[0168] The display coordinates of the main display layer and the auxiliary display layer are aligned to ensure spatial consistency. The alignment method is as follows: First, several feature points are set on both the main display layer and the transparent display layer; then, the positions of these feature points in three-dimensional space are measured using a stereo vision or infrared tracking system; finally, the coordinate transformation matrix between the two display surfaces is calculated based on the spatial correspondence of the feature points. After alignment, the system projects the three-dimensional intersection points obtained from ray intersection onto the main display layer and the transparent display layer respectively, ensuring that the displayed content on both layers precisely overlaps in the user's line of sight.

[0169] Regarding the rendering layer of interactive indicators: Under normal aiming conditions, the crosshair itself is rendered only on the transparent display layer, and the main display layer does not render the crosshair to avoid ghosting issues caused by displaying two layers simultaneously. Only during special interactions such as hit detection can special effects (such as halos or bullet holes) be briefly rendered at the corresponding position on the main display layer, during which time the crosshair on the transparent display layer remains continuously displayed.

[0170] Example 14 The method also includes a multi-sensory device parallel processing step S8, as follows: S81. Obtain the attitude angles independently output by multiple motion-sensing input modules. Each motion-sensing input module has a unique communication identifier. Different motion-sensing input modules are distinguished and rendering channels are allocated independently based on the communication identifier. As one embodiment, in step S81 of the present invention, the system supports the simultaneous operation of multiple motion-sensing input modules. Each motion-sensing input module (such as a motion-sensing gun, motion-sensing gloves, etc.) integrates an inertial sensor and a wireless communication module, and has a unique communication identifier (such as a MAC address or a custom ID). The system distinguishes different modules through this identifier and independently allocates a data processing thread and a rendering channel to each module.

[0171] S82. For each motion input module, execute steps S2 to S6 independently to calculate the target display surface and display coordinates of the corresponding interactive indicator; As one embodiment, in step S82 of the present invention, the system processes the data of each motion-sensing input module in parallel. The attitude angle of each module is independently constructed with direction vectors, ray intersection is calculated, target display surface is selected, coordinate mapping is performed, and rendering position is calculated, without interference between them.

[0172] S83. On the target display surface, render the corresponding interactive indicator for each motion input module independently. The interactive indicator of different motion input modules are distinguished by color coding or style differences.

[0173] As one embodiment, in step S83 of the present invention, each user's interaction indicator has unique visual features for differentiation. Typical differentiation methods include: different colors (e.g., red for player 1, blue for player 2, green for player 3), different shapes (cross, circle, triangle, rhombus, etc.), different sizes, or combinations of the above. Users can customize the crosshair style, and the system automatically assigns the most different visual feature combinations to adjacent users.

[0174] Example 15 When the interactive indicators corresponding to multiple motion-sensing input modules are located on the same display surface, a conflict resolution sub-step is also included: S84. Calculate the pixel distance of each interactive indicator on the display surface; As one embodiment, in step S84 of the present invention, the system calculates the pixel distance between each interactive indicator on the same display surface in real time. Let the pixel coordinates of the two crosshairs be (pixelX1, pixelY1) and (pixelX2, pixelY2), then the pixel distance is:

[0175] S85. When the pixel distance is less than a preset threshold, the following at least one strategy shall be applied: (a) Maintain differentiation through color coding and style differences; (b) Enlarge the size of one of the interactive indicator icons to form a nested structure; (c) Display the combined crosshair and mark the identification information of overlapping users; (d) Adjust the transparency of each interactive indicator according to the preset priority; (e) Offset each interactive indicator outward radially by different pixels.

[0176] As one embodiment, in step S85 of the present invention, the typical value of the preset threshold is 20 to 100 pixels, or 0.5% to 2% of the display surface width. When the pixel distance is less than this threshold, conflict handling is triggered. The handling strategy includes at least one of the following: (a) Maintain differentiation through color coding and style differences. Even when positions overlap, crosshairs of different colors can still be distinguished.

[0177] (b) Enlarge the size of one of the interactive indicators to create a nested structure. Enlarge the outer ring of one of the crosshairs by 20% to 50% so that the two crosshairs form concentric circles or a nested structure, while still allowing the user to identify the center position of each crosshair.

[0178] (c) Display the combined crosshair and label the overlapping user's identification information. The system displays a special combined crosshair (such as a two-color ring or a numbered multi-lobed shape) and labels the overlapping user's number or name abbreviation next to the crosshair.

[0179] (d) Adjust the transparency of each interactive indicator according to the preset priority. For example, the local player has the highest priority, so their crosshair is fully displayed, while the crosshairs of other players are displayed semi-transparently.

[0180] (e) Offset each interactive indicator radially outward by different pixels. Offset each crosshair outward by 2 to 10 pixels in the direction from the center of the display surface to the crosshair position to achieve visual separation.

[0181] Example 16 The motion-sensing device is at least one of the following forms: A motion-sensing control device in the form of a firearm, wherein the inertial sensor is disposed inside the firearm-shaped device; A motion-sensing control device in the form of a melee weapon (simulated toy model, non-aggressive), wherein the melee weapon (simulated toy model, non-aggressive) includes at least one of swords, shields, and clubs; A handheld motion-sensing control device, in which a single user holds multiple of the aforementioned motion-sensing devices; A motion-sensing input module deployed on different parts of the user's body, including at least two of the following: wrist, ankle, and waist.

[0182] Specifically, the motion-sensing device is at least one of the following forms: (a) A motion-sensing control device in the form of a firearm, wherein the inertial sensor is disposed inside the firearm-shaped device.

[0183] As one embodiment, the motion-sensing control device in this invention, shaped like a firearm, mimics the shape and grip of a real firearm and integrates an inertial measurement unit. The user holds the motion-sensing gun and aims it as if operating a real firearm, with the gun's direction mapped in real-time to the position of an interactive indicator.

[0184] (ii) A motion-sensing control device in the form of a melee weapon (simulated toy model, non-aggressive), wherein the melee weapon (simulated toy model, non-aggressive) includes at least one of swords, shields, and clubs.

[0185] As one embodiment, the motion-sensing control device for melee weapons (simulated toy models, non-aggressive) in this invention includes swords (for slashing attacks), shields (for blocking and defense), clubs (for striking), etc. The system determines the direction vector based on the posture angle and calculates the intersection of the direction vector with multiple display surfaces to determine the melee interaction area and collision detection position.

[0186] (iii) Handheld motion control device, wherein a single user holds multiple of the aforementioned motion control devices.

[0187] As one embodiment, in this invention, a single user can simultaneously hold multiple motion-sensing devices. For example, one user can hold a gun-shaped motion-sensing device for shooting and aiming, while another user can hold a melee weapon (simulating a toy model, non-aggressive)-shaped motion-sensing device for melee blocking or attacking. The system independently calculates the direction vector and renders interactive indicator icons for each motion-sensing device.

[0188] (iv) A motion-sensing input module deployed on different parts of the user’s body, wherein the different parts of the body include at least two of the wrist, ankle and waist.

[0189] In one embodiment, multiple motion-sensing input modules are deployed on different parts of the user's body, with each module independently outputting a posture angle. The system independently calculates the direction vector based on the posture angle of each module and renders the corresponding interactive indicator, thus achieving full-body motion-sensing interaction.

[0190] Example 17 The method also includes motion platform linkage steps: U1. Obtain the motion attitude of the motion platform, including the platform's position in the world coordinate system. and attitude quaternions ; As one embodiment, in step U1 of the present invention, the motion platform is a movable base, such as a six-degree-of-freedom platform, a hydraulic platform, or an electromechanical servo platform. The platform controller outputs the motion posture of the platform in real time, including the three-dimensional position of the platform center in the world coordinate system. (Unit: m) and attitude quaternion describing the platform's orientation .

[0191] U2. Update the real-time position of the fixed reference position in the world coordinate system according to the motion posture of the platform:

[0192] in, The fixed reference position is the fixed position in the platform's local coordinate system. Let be the rotation matrix constructed from the platform attitude quaternions; It is a 3×1 vector with units of m; It is a 3×3 rotation matrix, dimensionless; l is a 3×1 vector with units of m; It is a 3×1 vector with units of m.

[0193] U3. Acquire the relative posture of the motion sensing device with respect to the motion platform. Q gun _ relative ; As one embodiment, in step U3 of the present invention, the inertial measurement unit inside the motion sensor gun adopts an independent inertial reference frame. During the static calibration phase, the initial attitude of the motion sensor gun relative to the platform base is recorded. During the dynamic operation phase, the attitude of the motion sensor gun relative to the inertial reference frame is converted to its attitude relative to the platform base through coordinate transformation, resulting in the relative attitude Q_gun_relative (quaternion representation).

[0194] U4, based on the platform attitude and the relative attitude Q gun _ relative The orientation vector of the motion-sensing device in the world coordinate system is calculated using the following formula:

[0195] in, This is the initial orientation vector of the motion sensing device in the local coordinate system; U5, with P ref _ world Starting point dir world Given the direction vector, execute steps S3 to S6.

[0196] As one embodiment, in step U5 of the present invention, the updated reference point position and direction vector are substituted into the processes of steps S3 to S6 to complete the aiming interaction. Since the reference point is updated in real time with the platform's movement, the user can still maintain accurate aiming during platform movement.

[0197] Example 18 It also includes the linkage steps for wearable augmented reality display devices: V1. Obtain the position and orientation of wearable augmented reality display devices in space through a tracking system; As one embodiment, in step V1 of the present invention, the wearable augmented reality display device is AR glasses or a lightweight head-mounted display, installed on the user's head. The real-time position and orientation of the AR glasses in space are obtained through an external tracking system (UWB positioning system).

[0198] V2. Transform the spatial direction vector into the local coordinate system of the wearable augmented reality display device; As one embodiment, in step V2 of the present invention, the spatial direction vector in the world coordinate system is converted into a direction vector in the local coordinate system of the AR glasses through coordinate transformation, so that the aiming information can be correctly displayed in the field of view of the AR glasses.

[0199] V3. Render aiming assistance information on the display screen of the wearable augmented reality display device according to the converted direction vector.

[0200] As one embodiment, in step V3 of the present invention, personalized aiming assistance information, such as aiming guide lines, ballistic prediction, and target markers, is rendered on the display screen of the AR glasses. When the user turns their head, the content displayed on the AR glasses changes accordingly, but the aiming guide lines always remain consistent with the actual direction of the motion-sensing gun.

[0201] Example 19 The inertial sensor is an inertial measurement unit, which integrates a three-axis gyroscope and a three-axis accelerometer.

[0202] As one embodiment, the inertial sensor in this invention is preferably an inertial measurement unit (IMU), specifically an InvenSense MPU6050 or MPU9250 chip. The MPU6050 integrates a three-axis gyroscope and a three-axis accelerometer, while the MPU9250 additionally integrates a three-axis magnetometer. The three-axis gyroscope is used to measure the angular velocity of the motion-sensing device around the three coordinate axes, and the three-axis accelerometer is used to measure the acceleration of the motion-sensing device in the three coordinate axis directions. Through data fusion algorithms, the attitude angle of the motion-sensing device can be accurately calculated.

[0203] Example 20 The motion-sensing device communicates wirelessly with the plurality of display surfaces. As one embodiment, the motion-sensing device in this invention is a handheld or worn device. The user can control the position of the interactive indicator by changing the direction of the motion-sensing device in the air, without needing to touch any display surface. This is fundamentally different from traditional touch interaction (such as touchscreens, laser pointers, etc.)—users can operate from several meters away from the display surface, making the interaction more natural and free.

[0204] Example 21 An immersive multi-display interactive aiming system includes: The motion input module includes an inertial sensor for outputting attitude angles; The orientation calculation module is used to determine a spatial orientation vector based on the attitude angle, wherein the spatial orientation vector is constructed from a fixed reference position in space. The intersection positioning module is used to take the spatial direction vector as a ray and perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray. Based on the distance, the display surface where the closest intersection point is located is selected as the target display surface, and the intersection point is used as the target position of the interactive indicator in three-dimensional space. A coordinate mapping module is used to map the target position to display coordinates on the target display surface; A rendering module is used to render the interactive indicator on the target display surface at the display coordinates.

[0205] The system also includes a cross-display-plane transition module for: A transition area is preset at the junction of adjacent display surfaces; When the interactive indicator enters the transition area, the same direction vector is intersected with the current display surface and the adjacent display surface respectively to obtain two independent intersection points, which are mapped to the screen pixel coordinates of the corresponding display surface. The interactive indicator is rendered on both the current display surface and the adjacent display surface. The transparency of the interactive indicator on the current display surface decreases dynamically according to the distance from the boundary line, and the transparency of the interactive indicator on the adjacent display surface increases dynamically according to the distance from the boundary line. When the interactive indicator fully enters the adjacent display surface, rendering stops on the current display surface and only occurs on the adjacent display surface.

[0206] The rendering module supports at least one of the following rendering methods: Two-dimensional user interface rendering method: The interactive indicator is overlaid and rendered at the screen pixel coordinates of the target display surface; 3D spatial rendering method: Place a 3D crosshair model at the 3D spatial coordinates of the target location for rendering.

[0207] It also includes a parallel processing module for: The attitude angles output independently by multiple motion-sensing input modules are obtained. Each motion-sensing input module has a unique communication identifier. Different motion-sensing input modules are distinguished and rendering channels are allocated independently based on the communication identifier. Each motion-sensing input module independently calculates its corresponding target display surface and display coordinates for interactive indicators. On the target display surface, a corresponding interactive indicator is rendered independently for each motion-sensing input module. The interactive indicator of different motion-sensing input modules is distinguished by color coding or style differences.

[0208] It also includes a conflict handling module, which performs at least one of the following processing steps when the interaction indicators corresponding to multiple motion-sensing input modules are located on the same display surface and the pixel distance is less than a preset threshold: (a) Maintain differentiation through color coding and style differences; (b) Enlarge the size of one of the interactive indicator icons to form a nested structure; (c) Display the combined crosshair and mark the identification information of overlapping users; (d) Adjust the transparency of each interactive indicator according to the preset priority; (e) Offset each interactive indicator outward radially by different pixels.

[0209] The multiple display surfaces are arranged in a surrounding manner around the user, including at least one of the following arrangements: The four display surfaces are located in front of, behind, to the left and to the right of the user, respectively; Five display surfaces, including a display surface located above the user in addition to the four display surfaces; The system has six display surfaces, including five additional surfaces below the user, creating a complete surround effect.

[0210] The plurality of display surfaces are formed by at least one of the following devices: front projection projection device, rear projection projection device, and LED display screen.

[0211] The display surface also includes a transparent display layer disposed in front of the projection surface projected by the projection device; The rendering module supports dual-layer rendering: rendering three-dimensional scene content on the projection surface as the main display layer, and rendering at least a portion of the interactive indicator and two-dimensional enhancement information on the transparent display layer as an auxiliary display layer; The display coordinates of the main display layer and the auxiliary display layer are aligned to ensure spatial consistency.

[0212] The motion-sensing device is at least one of the following forms: A motion-sensing control device in the form of a firearm; A motion-sensing control device in the form of a melee weapon (simulated toy model, non-aggressive), wherein the melee weapon (simulated toy model, non-aggressive) includes at least one of swords, shields, and clubs; A handheld motion-sensing control device, in which a single user holds multiple of the aforementioned motion-sensing devices; Motion-sensing input modules deployed on different parts of the user's body.

[0213] The system also includes a motion platform, which is linked to the motion sensing device via bidirectional signals. Forward control flow: The posture change of the motion sensing device serves as the input signal that triggers the motion platform's movement; Reverse feedback stream: The motion signal of the motion platform is synchronously sent to the rendering engine to adjust the perspective of the virtual scene, and at the same time, tactile feedback is transmitted to the user through the force feedback device of the somatosensory device.

[0214] Example 22 This invention provides an immersive aiming and interactive system with a four-sided projection front projection mode, which is a preferred embodiment of the invention and is suitable for small and medium-sized immersive experience scenarios such as verification stores.

[0215] In the specific implementation process, the four-sided projection front projection mode uses four planar projection screens to form a surround display environment. The four display surfaces are located in front of, behind, to the left and right of the user, respectively, forming a panoramic surround of the user in the horizontal direction.

[0216] The dimensions of each display panel can be customized according to the actual site conditions, with typical dimensions ranging from 3 to 5 meters in width and 2.5 to 3 meters in height. The display screen uses diffuse reflection projection screen material to achieve uniform brightness distribution and a wide viewing angle.

[0217] The system employs multi-projection surface synchronous display technology to achieve multi-viewport rendering, typically implemented using Unreal Engine 5's nDisplay multi-projection system. The nDisplay system can divide the entire image into multiple sub-images, each output to its corresponding projection device, achieving synchronized display across multiple surfaces. The projection device is an ultra-short-throw laser projector with a projection ratio of less than 0.4, typically 0.25 to 0.35, placed at the bottom edge of the corresponding display surface to minimize user obstruction of the projection light path. The projector uses a laser light source with a brightness of no less than 5000 lumens, ensuring a clear image even in venues with adequate ambient lighting.

[0218] The motion-sensing input module adopts a gun-shaped motion-sensing control device, integrating an inertial measurement unit (IMU). The IMU uses the MPU6050 chip, which integrates a three-axis gyroscope and a three-axis accelerometer. The gyroscope's range is ±250° / s to ±2000° / s, and the accelerometer's range is ±2g to ±16g, with a configurable sampling rate up to 8kHz. The microcontroller uses Espressif Systems' ESP32 chip, which features a dual-core processor with a clock speed of up to 240MHz, rich communication interfaces, and low power consumption, making it suitable for battery-powered portable devices.

[0219] The attitude data processing flow is as follows: First, raw gyroscope angular velocity data and accelerometer data are acquired from the inertial measurement unit (IMU). Then, complementary filtering or Mahony algorithm is used for data fusion, fusing the gyroscope and accelerometer data into a quaternion representation of the attitude. Next, the quaternion is converted into Euler angles to obtain the yaw and pitch angles. Finally, the attitude angle data is transmitted to the host computer via Bluetooth or low-latency wireless communication. It is recommended that the attitude angle sampling frequency be set to 100Hz to 200Hz to ensure real-time response without excessive data transmission burden.

[0220] By adopting the above technical solution, through the combination of a gun-shaped motion-sensing device and an inertial measurement unit, users can hold the motion-sensing gun and aim it as if operating a real gun. The direction of the motion-sensing gun is mapped in real time to the direction of the interactive indicator in three-dimensional space, realizing a natural motion-sensing aiming interactive experience.

[0221] In some embodiments of this application, the spatial direction vector is constructed as follows.

[0222] In practice, the specific location of the starting point (i.e., the fixed reference position) of the direction vector is determined based on the system configuration and application scenario. Typical setup methods include the following: Chest / Clavicle Midline Position: For a standing immersive shooting experience, a fixed reference position is typically set in the user's chest area, specifically 5cm to 10cm in front of the midpoint of the clavicle. This corresponds to approximately (0, 1.5m, 0) coordinates in 3D space (using the projection of the user's midpoints on the ground as the origin of the world coordinate system, with the Y-axis pointing vertically upwards and the Z-axis pointing directly forward). This position is close to the user's body midline and at eye level, providing the most intuitive aiming mapping.

[0223] Motion-sensing device mounting base location: For scenarios where the motion-sensing gun is mounted on a fixed bracket or motion platform, the fixed reference position is set as the center of the motion-sensing device's rotation axis or the universal joint fulcrum. This position remains fixed in space, and the motion-sensing device rotates around this point in pitch and yaw motion.

[0224] User head position: For scenarios requiring more precise viewpoint matching, a fixed reference position can be set at the midpoint between the user's eyes, at a height of approximately 1.6m to 1.7m. This position coincides with the user's visual focus, resulting in the highest aiming accuracy, but requires additional head tracking equipment.

[0225] Preset points for the venue: For large multiplayer venues, fixed reference positions can be preset at designated points for each game position according to the venue layout. All players' reference points are uniformly marked, which facilitates system management and calibration.

[0226] In this embodiment, the chest position is used as the default fixed reference position, and the coordinates of this position in three-dimensional space remain unchanged, which is different from the follow-up scheme of virtual reality head-mounted display solutions that use the head position as the reference point.

[0227] The direction vector is constructed based on the yaw and pitch angles, and the calculation formula is as follows: dirX = cos(pitch) × cos(yaw) dirY = sin(pitch) dirZ = cos(pitch) × sin(yaw) Where yaw represents the yaw angle, pitch represents the pitch angle, and dirX, dirY, and dirZ are the three components of the normalized direction vector. The direction vector originates from the starting point and points towards the user's desired aiming direction.

[0228] Coordinate system and angle definitions: The angles and coordinates used in the above formulas follow the following definitions: Coordinate system: A right-handed coordinate system is used, with the Y-axis pointing vertically upward, the Z-axis pointing forward, and the X-axis pointing horizontally to the right. Yaw is the angle between the projection of the direction vector onto the XZ plane (horizontal plane) and the Z-axis, positive to the right; pitch is the angle between the direction vector and the XZ horizontal plane, positive upward. All angles are defined relative to the world coordinate system (or site coordinate system), not the motion sensing device's own coordinate system.

[0229] Angle range: The yaw angle ranges from -180° to +180° (or -π to +π radians), covering the entire circumference of the horizontal plane; the pitch angle ranges from -90° to +90° (or -π / 2 to +π / 2 radians), covering the vertical range from directly below to directly above.

[0230] The quaternion to Euler angle conversion sequence is as follows: Attitude angles are obtained from quaternions output by the inertial sensor, using a ZYX internal rotation sequence (i.e., first rotate around the Z-axis (yaw), then around the Y-axis (pitch), and finally around the X-axis (roll). In actual implementation, to avoid the gimbal lock problem of Euler angles, the internal calculation prioritizes direct vector rotation using quaternions, only converting to Euler angles when angle values ​​need to be output or displayed. The formula for the direct rotation direction vector using quaternions is: dir_world = q × dir_local × q - ¹ Where dir_local is the initial direction vector in the local coordinate system (usually (0, 0, 1), i.e., directly in front of the Z-axis), and q is the attitude quaternion. - ¹ is its conjugate (the inverse of a unit quaternion is equal to its conjugate).

[0231] Gimbal Lock Handling: When the pitch angle approaches ±90°, Euler angle representation will exhibit gimbal lock, meaning that yaw and roll angles lose their independent meaning. In the system of this invention, this problem is avoided through the following methods: The calculation of the internal direction vector always uses quaternion rotation and does not rely on the intermediate representation of Euler angles; Euler angle formulas are only used for auxiliary calculations and output debugging within the normal pitch angle range (e.g., -70° to +70°); When |pitch|>70°, the calculation path is completely switched to quaternion to ensure numerical stability at extreme angles.

[0232] Extended Pitch Angle Range Handling: When the pitch angle exceeds ±90° (e.g., in a five- or six-sided system, the user's upward tilt exceeds 90° or downward tilt exceeds 90°), the direction vector calculation adopts the following strategy: At this point, the physical meaning of the pitch angle changes from "tilt-up angle" to "pitch-down angle," requiring a sign flip of yaw to ensure directional continuity. That is, when pitch > 90°, it is equivalent to pitch' = 180° - pitch, and yaw' = yaw + 180°. This problem does not exist in quaternion representation because quaternions naturally cover the entire attitude space.

[0233] By employing the above technical solution, the starting point of the direction vector remains fixed, and the direction of the vector is changed only by variations in the yaw and pitch angles. This achieves a posture-driven method with a completely fixed reference point position and orientation, resulting in lower system complexity and better stability compared to the camera-following mode of virtual reality headsets. The quaternion-first calculation strategy ensures numerical stability and computational accuracy across the entire angular range.

[0234] In some embodiments of this application, the rendering method of the interactive indicator is as follows.

[0235] In practice, the rendering of interactive indicators utilizes the game engine's multi-viewport rendering mechanism. For a four-sided projection system, each display surface corresponds to an independent rendering viewport, and the viewport's resolution is consistent with the pixel resolution of the display surface. During rendering, the rendering instructions for the interactive indicators are sent to the corresponding viewport based on the target display surface's number.

[0236] The shape of the interactive indicator can be customized according to application requirements, typically a crosshair or a dot crosshair. The crosshair's color, size, style, and other parameters can be dynamically adjusted at runtime. The crosshair is rendered as a user interface (widget) component overlaid on the game screen, ensuring it is always visible and unaffected by depth testing.

[0237] When the interaction indicator is located in the center of the current display surface, the crosshair is rendered only on that display surface. When the interaction indicator moves to the edge of the display surface, it is handled according to the cross-display surface transition method.

[0238] By adopting the above technical solution and utilizing the multi-viewport rendering mechanism of the game engine, the interactive indicator marks are accurately rendered on multiple display surfaces, and the crosshair style can be flexibly customized to meet the visual needs of different applications.

[0239] In some preferred embodiments of this application, a method for implementing interactive indicators in a three-dimensional spatial rendering mode is provided. Unlike the aforementioned two-dimensional UI component rendering mode, the three-dimensional spatial rendering mode directly places a 3D crosshair model in the three-dimensional scene. This model updates in real time with the position of the intersection point of the direction vector and the display surface, thereby presenting an aiming effect with a sense of spatial depth.

[0240] In practice, the 3D spatial rendering method is implemented as follows: 3D Crosshair Model Creation: Create a standalone 3D model in the 3D scene as an interactive indicator. The shape of the crosshair model can be designed according to application requirements; typical shapes include: Suspended cross ring: A cross-shaped or circular mesh model located in three-dimensional space, with a hollow area in the center; Floating light spot: A self-illuminating small sphere or flat sprite that always faces the camera (Billboard rendering). 3D crosshair: A three-dimensional crosshair model with a sense of volume, such as a triangular pyramid or tetrahedron, with its tip pointing in the aiming direction. The size of the crosshair model is dynamically adjusted according to the display distance to ensure visual consistency when viewed on a display surface at different distances.

[0241] Position update logic: Each frame, the position of the 3D crosshair model in the world coordinate system is updated based on the coordinates of the intersection point between the ray and the display surface. The crosshair model is placed in 3D space at the intersection point, and can be slightly offset towards the user (e.g., 0.05 meters to 0.1 meters) to avoid Z-fighting (depth conflict) issues caused by overlapping with the display surface.

[0242] Orientation and Attitude: There are two strategies for orienting a 3D crosshair model: Oriented towards a reference point: The crosshair model always faces a fixed reference position (i.e., the user's direction), ensuring that the user can see the front of the crosshair from any angle. In this way, the projection of the crosshair onto the display surface is a graphic that is always facing the user.

[0243] Along the direction vector: The axis of the crosshair model coincides with the direction vector, simulating the effect of a real laser pointer or the flight direction of a bullet. In this way, when the crosshair is located on the side display surface, the user sees the side or angle of the crosshair, providing a stronger sense of spatial depth.

[0244] Hybrid rendering strategy: The system can simultaneously support both 2D UI rendering and 3D spatial rendering, dynamically switching according to the scene. For example, a 2D UI crosshair is used for long-range aiming to ensure accuracy; a 3D crosshair is switched to enhance immersion during close-quarters combat or cutscenes. Both methods share the same set of ray intersection calculation results, differing only in the rendering path.

[0245] By adopting the above technical solution, the three-dimensional spatial rendering method provides a stronger sense of spatial depth and immersion for interactive signs and indicators, complementing the two-dimensional UI rendering method and enabling the system to adapt to application scenarios with different styles and precision requirements.

[0246] In some preferred embodiments of this application, a method for seamless transition across display surfaces is provided.

[0247] In practical implementation, a transition area is preset at the boundary between adjacent display surfaces. The width of the transition area is set to 3% to 10% of the display surface width, with a typical value of 5%. The geometric definition of the transition area in three-dimensional space is as follows: For two adjacent planar display surfaces A and B, let the equation of the plane containing the boundary line between the two planes be the transition plane. The transition region is the area in the local coordinate system of the two display surfaces that is less than the width of the transition region from the boundary line.

[0248] When the interactive indicator enters the transition area, the cross-display-plane transition logic is triggered. During the transition, the interactive indicator is rendered simultaneously on the current display plane and the adjacent display plane, and the rendering parameters are dynamically adjusted according to the distance from the boundary line. The transparency of the crosshair on the current display surface = 1 - (distance from the boundary / width of the transition area) The transparency of the crosshair on adjacent display surfaces = distance from the boundary / width of the transition area When the interactive indicator fully enters the adjacent display surface, that is, when the transparency on the current display surface drops to zero, the crosshair stops being rendered on the current display surface and is rendered with full transparency only on the adjacent display surface.

[0249] Cross-display surface coordinate conversion is based on the physical geometric relationship of the display surfaces in three-dimensional space. The core principle is to keep the world coordinates in three-dimensional space unchanged and project them onto the two display surfaces respectively. The specific conversion method is as follows: Let P_A (world coordinates) be the intersection point of the direction vector with display surface A, and P_B (world coordinates) be the intersection point with display surface B. During the transition process, the system simultaneously calculates the intersection points of the ray with the two adjacent display surfaces: The same ray, two intersection points: Starting from a fixed reference point, the same direction vector intersects with display surfaces A and B respectively, resulting in two independent intersection points, P_A and P_B. These two intersection points lie on the same ray, but on different display surfaces.

[0250] World coordinate consistency: Because the two intersection points share the same ray direction and the same starting point, they are always on the same straight line in three-dimensional space, ensuring that the crosshairs on the two display surfaces point to the same aiming direction in physical space. When the user observes from the center position, the two crosshairs coincide in the line of sight, and there will be no visual misalignment.

[0251] UV coordinates are calculated independently: For display surface A, P_A is projected onto the local coordinate system of A to obtain the UV coordinates (u_A, v_A): u_A = (P_A - P_origin_A) · U_vec_A / width_A v_A = (P_A - P_origin_A) · V_vec_A / height_A Where P_origin_A is the lower left corner origin of display surface A, U_vec_A and V_vec_A are the horizontal and vertical unit vectors of A, respectively, and width_A and height_A are the width and height of A.

[0252] Similarly, for the display surface B, P_B is projected onto the local coordinate system of B to obtain the UV coordinates (u_B, v_B).

[0253] Pixel coordinate transformation: Multiply each UV coordinate by the pixel resolution of the corresponding display surface to obtain the final rendered pixel position. pixel_A = (u_A × width_pixels_A, v_A × height_pixels_A) pixel_B = (u_B × width_pixels_B, v_B × height_pixels_B) Positional continuity during the transition: When the direction vector moves from display surface A to display surface B, P_A moves along display surface A towards the boundary line, while P_B enters display surface B from the boundary line. The two crosshairs move smoothly on their respective display surfaces, coinciding in spatial position at the boundary line, achieving a seamless transition.

[0254] By adopting the above technical solution, through the setting of transition areas and dynamic adjustment of transparency, a smooth transition of interactive indicators between different display surfaces is achieved, avoiding the crosshair jump and flickering problems common in existing two-dimensional multi-screen solutions, and improving the user's visual experience and operational continuity.

[0255] Through the above technical solution, this invention achieves for the first time real-time aiming and interaction based on the posture information of a motion-sensing device in a multi-display immersive environment, without the need to wear a head-mounted display, eliminating the dizziness caused by the feeling of being enclosed, and supporting flexible adaptation to various display forms, thus having broad application prospects.

[0256] Example 23 In some preferred embodiments of this application, an immersive aiming and interactive system in rear projection mode is provided, which is suitable for scenarios such as large venues or high-end experience flagship stores.

[0257] In practice, the system architecture of rear projection mode is basically the same as that of front projection mode, the difference being the installation position of the projection device. In rear projection mode, the projection device is located behind the projection screen, and the user is located between the projection device and the screen. The projection light path is projected from behind the screen onto the screen surface, and the user views the screen from the front without being affected by the projection device blocking the light path.

[0258] Rear projection screens utilize transmissive semi-transparent screens or specialized rear projection screen materials. The screen's transmittance and scattering characteristics are optimized to provide a wide viewing angle while ensuring the brightness of the projected image. The installation distance between the projection device and the display screen is determined based on the projector's throw ratio and the image size. For ultra-short-throw projectors, the projection distance can be controlled within 0.5 meters to 1 meter.

[0259] Because the user is isolated from the projection light path in rear projection mode, this solution is particularly suitable for commercial venues with high foot traffic, as users' activities in front of the display screen will not interfere with the projection light path. At the same time, rear projection mode avoids the safety hazard of projection light shining directly into people's eyes, thus improving user safety.

[0260] In rear projection mode, in the geometric model for calculating the intersection of a ray and the display surface, the normal vector of the display surface points towards the user's side (i.e., opposite to the projection direction). This requires special clarification: in front projection mode, both the user and the projector are on the same side of the display surface, so the normal vector naturally points towards the user's side; however, in rear projection mode, the user and the projector are on opposite sides of the display surface. To ensure the correctness of the ray-plane intersection calculation (i.e., the ray originating from the user's reference point intersects the display surface in a positive direction), the normal vector of the display surface must uniformly point towards the user's side, not towards the projection device side. Specifically: A unified definition of the normal vector: Regardless of whether it is a front projection or a rear projection mode, the plane normal vector of each display surface points into the immersive space (i.e., towards the central area where the user is located). In the plane equation Ax + By + Cz + D = 0, (A, B, C) is the unit normal vector, which always points from the display surface towards the user.

[0261] Impact of Rear Projection Mode: Rear projection mode only affects the projection optical path, the installation position of the projection equipment, and the display screen material; it does not affect the geometric calculation model for ray-plane intersection. When the system obtains the geometric parameters of each display surface during the calibration phase, it has already defined them according to the unified rule of "normal vector pointing towards the user side."

[0262] Furthermore, the geometric calibration parameters of the display screen need to be adjusted accordingly in rear projection mode, as the position and orientation of the display screen may differ from those in front projection mode. Additionally, rear projection mode provides more space in front of the display screen, which is beneficial for achieving larger display areas and more complex site layouts.

[0263] By adopting the above technical solution, the rear projection mode maintains the same aiming and interactive functions as the front projection mode, while providing better safety and site adaptability, making it suitable for large immersive experience venues.

[0264] Example 24 In some preferred embodiments of this application, an immersive aiming and interactive system with five-sided and six-sided surround projection is provided, which is suitable for scenarios such as flagship stores or high-end experience centers and can provide a more complete panoramic immersive experience.

[0265] In practice, the five-projection system adds a top display surface (ceiling projection) above the user to the aforementioned four-projection system, forming five display surfaces that surround the user's horizontal and overhead space. The geometry of the top display surface is planar or near-planar, and its size covers the user's field of vision above their head.

[0266] The six-sided projection system further adds a bottom display surface (ground projection) located below the user, forming six display surfaces that completely surround the user's space. The bottom display surface can be implemented using a translucent grid floor or embedded LED floor tiles, which can both support the user standing and display virtual environment content.

[0267] For five-sided and six-sided systems, the algorithm for finding the intersection of rays and display surfaces needs to be extended accordingly. In a four-sided system, the pitch angle range of the ray direction vector is typically -60° to +60°, basically covering the user's field of view when looking straight ahead and rotating slightly up and down. In a five-sided system, the upper limit of the pitch angle needs to be extended to cover the top display surface, with the pitch angle range extended to -60° to +120°; in a six-sided system, the pitch angle range is further extended to -120° to +120°, covering the entire space from directly above to directly below.

[0268] For ray intersection calculations within the extended pitch angle range, first determine the pitch angle interval of the ray and identify the set of target display surfaces that may intersect: when the pitch angle is between -30° and +30°, it mainly intersects with the four side display surfaces; when the pitch angle is greater than +30°, it may intersect with the top surface; when the pitch angle is less than -30°, it may intersect with the bottom surface.

[0269] For cases where the pitch angle is close to ±90°, i.e., the ray direction is nearly vertically upward or downward, the intersection point calculation between the ray and the planar display surface still uses the aforementioned ray-plane intersection formula, but special attention needs to be paid to the stability of the numerical calculation. When the angle between the ray and the plane normal vector is close to 90°, i.e., the ray direction is nearly parallel to the plane, the denominator term tends to zero, and a minimum angle threshold needs to be set. When the angle is less than the threshold, it is determined that they do not intersect.

[0270] Using the above technical solutions, the five-sided and six-sided surround projection systems provide a complete panoramic immersive experience. The extension of the ray intersection algorithm ensures accurate aiming and positioning throughout the entire space, making it suitable for flagship experience scenarios with higher requirements for immersion.

[0271] Example 25 In some preferred embodiments of this application, immersive aiming and interactive systems for spherical and hemispherical screens are provided, suitable for scenarios such as dome-style cinema experiences or high-end theme parks.

[0272] In practical implementation, the geometry of the spherical screen display surface is a sphere or a portion of a sphere. The equation of a sphere can be expressed as (x - xc). 2 + (y - yc) 2 + (z - zc) 2 = R 2 Where (xc, yc, zc) are the coordinates of the sphere's center, and R is the radius of the sphere. For a hemispherical screen, the applicable region of the spherical equation is limited to z ≥ zc (or z ≤ zc).

[0273] The solution for the intersection of a ray and a sphere differs from that for a plane. Let the ray originate at P0 and its direction vector be dirRay. Substitute these into the equations of the sphere: dot(P0 - C, P0 - C) - 2 × dot(dirRay, P0 - C) × t - dot(dirRay,dirRay) × t 2 = R 2 Where C represents the coordinates of the sphere's center. This is a quadratic equation in t, which may have 0, 1, or 2 real solutions. When a real solution exists, the intersection point with the smallest value of t greater than zero is taken as the target intersection point.

[0274] For any point P on the sphere, its spherical coordinates can be determined in the following way: establish a local spherical coordinate system with the center of the sphere as the origin, the latitude angle φ is defined as the angle between the line connecting point P to the center of the sphere and the equatorial plane, ranging from -90° to +90°; the longitude angle θ is defined as the angle between the projection of point P onto the equatorial plane and the reference direction, ranging from 0° to 360°.

[0275] The mapping method for spherical UV coordinates is as follows: U = θ / 360° V = (90° + φ) / 180° The normalized UV coordinate range is [0,1] × [0,1], which is mapped to the pixel resolution of the spherical display surface.

[0276] Projection equipment for spherical screens typically uses fisheye lens projectors or multiple ordinary projectors seamlessly stitched together. Fisheye lens projections suffer from severe barrel distortion, requiring anti-distortion processing during the rendering stage to ensure the geometric accuracy of the projected image on the spherical surface.

[0277] For hemispherical screen solutions, the display surface only covers a 180° spherical area above the user, and the tilt angle is limited to 0° to 90°, eliminating the need for intersection calculations in the lower area. The ray intersection algorithm for hemispherical screen systems is simpler and suitable for scenarios with constraints on cost and complexity.

[0278] Using the above technical solutions, spherical and hemispherical screen systems provide a higher degree of immersive surround feeling. The ray-spherical intersection algorithm ensures that the user's aiming in any direction can be accurately mapped to the spherical display position, which is suitable for application scenarios that have the ultimate pursuit of immersive experience.

[0279] Example 26 In some preferred embodiments of this application, an immersive aiming interaction system for LED displays is provided, suitable for high-brightness environments or e-sports experience scenarios requiring high refresh rates.

[0280] In practice, LED displays are composed of multiple LED modules spliced ​​together to form a large-area display wall. The pixel pitch of each LED module can be selected according to the viewing distance and resolution requirements, with typical values ​​ranging from P2 to P5 (i.e., the center-to-center distance between adjacent LED beads is 2mm to 5mm). The LED display wall can be arranged in a planar layout to form a multi-faceted surround structure, or it can be arranged in an arc shape to form a curved display.

[0281] The fundamental difference between LED displays and projection systems lies in their active light emission characteristics, eliminating the need for a projection optical path. The geometric position of each display surface is determined by the physical installation position of the LED modules, eliminating the need for projection calibration and distortion correction. The intersection calculation between rays and LED display surfaces employs the same ray-plane intersection algorithm as for planar projection screens.

[0282] For curved LED display surfaces, the display surface is composed of multiple planar LED modules installed along an arc trajectory. Each module still has a planar geometry, but the overall effect is a curved surface. The intersection calculation of the curved display surface can be performed by discretizing the entire curved surface into multiple planar regions, performing ray-plane intersection calculations on each region, and selecting the closest intersection point; or the curved surface can be regarded as a parametric surface and the ray-surface intersection algorithm can be used for direct calculation.

[0283] LED displays typically have a refresh rate of 3840Hz, far exceeding the 60Hz of traditional projection systems, enabling smoother motion graphics and making them particularly suitable for the fast-paced aiming scenes in shooting games. Furthermore, LED displays can achieve brightness levels exceeding 1000 cd / m², ensuring image visibility even under high ambient light conditions.

[0284] By adopting the above technical solution, the LED display system eliminates the calibration process of projection equipment and projection screen, reducing system complexity and maintenance costs, while providing high brightness and high refresh rate display effects, making it suitable for commercial venues with high traffic and high sunlight conditions.

[0285] Example 27 In some preferred embodiments of this application, a pseudo-augmented reality aiming and interaction system with a projection surface overlaid with a transparent display layer is provided, which is suitable for mixed reality experience scenarios that require the overlay of augmented information.

[0286] In practice, the system consists of two display layers: the main display layer is the aforementioned projection screen or LED screen, used to present immersive 3D scene content; the transparent display layer is set in front of the main display layer, typically at a distance of 0.5 meters to 2 meters, used to overlay display enhancement information.

[0287] Transparent display layers can be achieved using transparent OLED displays, transparent LCD screens, or transparent glass projection films. The transparency of a transparent display layer is typically between 50% and 85%, allowing it to display enhanced information without completely obscuring the content of the main display layer behind it or the user's view of the real environment.

[0288] The method for aligning the coordinates of the transparent display layer and the main display layer is as follows: First, several feature points are set on both the main display layer and the transparent display layer. Then, the positions of these feature points in three-dimensional space are measured using a stereo vision or infrared tracking system. Finally, based on the spatial correspondence of the feature points, the coordinate transformation matrix between the two display surfaces is calculated. After calibration, when performing coordinate mapping, the system first maps the intersection coordinates to the pixel positions of the main display surface, and then transforms them to the pixel positions of the transparent display surface using the coordinate transformation matrix, ensuring that the content coordinates on the two display surfaces are aligned.

[0289] The interactive indicators are rendered on two separate display surfaces, but they are not displayed as two independent crosshairs simultaneously. Instead, they are assigned to different layers based on the type of interactive content. Main display layer (projection screen / LED screen): Renders 3D game scene content, hit effects, explosion effects, aiming guide lines within the scene (such as bullet trajectories), and other visual elements belonging to the virtual world. The main display layer does not render the crosshair itself.

[0290] Transparent Display Layer: This layer renders the crosshair itself, HUD elements (ammunition quantity, health, score, weapon status (simulated toy model, non-aggressive), minimap, mission prompts, and other two-dimensional enhanced information. This information belongs to the user interface layer, overlaid in front of the user's field of view, and is separated from the virtual scene.

[0291] The crosshairs on both display surfaces point to the same point in three-dimensional space. The system first calculates the aiming intersection point in three-dimensional space using ray intersection, and then projects this intersection point onto the main display layer and the transparent display layer respectively to obtain their respective pixel coordinates. When the user observes from behind the transparent display layer, the crosshair on the transparent display layer and the target point on the main display layer precisely coincide in the line of sight, forming a visual spatial alignment effect.

[0292] When visual feedback of the crosshair needs to be displayed on the main display layer (such as the red flashing of the crosshair when it hits an enemy), the system uses the following strategy: Only at the moment of hit detection or special interaction, a brief effect (such as a halo or bullet hole) is rendered at the corresponding position on the main display layer. During this time, the crosshair on the transparent display layer remains continuously displayed. In normal aiming mode, the crosshair itself is only rendered on the transparent display layer, avoiding ghosting issues caused by displaying both layers simultaneously.

[0293] The choice of display layer is determined by the content type. The system has a built-in rendering layer routing table: content with spatial depth goes through the main display layer, while UI information goes through the transparent display layer. Developers can also customize the layer allocation strategy according to application needs.

[0294] By adopting the above technical solution, the transparent display layer solution provides a clear two-dimensional augmented information display channel while maintaining an immersive three-dimensional scene. The coordinate alignment technology ensures the spatial consistency of multi-layer display content, and the pseudo-augmented reality effect enhances the readability of information and the intuitiveness of interaction.

[0295] Example 28 In some preferred embodiments of this application, an immersive aiming interaction system linked to a wearable augmented reality display device is provided, which is suitable for high-end experience scenarios that require personalized perspective enhancement.

[0296] In practice, wearable augmented reality display devices are AR glasses or lightweight head-mounted displays, installed on the user's head without obstructing the user's view of the immersive display environment. The AR glasses have built-in microdisplays, inertial measurement units, and wireless communication modules, enabling them to independently render personalized augmented content.

[0297] In the system architecture, the immersive multi-display surface serves as a public display environment, where all users share the same virtual scene content; the AR glasses serve as personal display terminals, displaying only the wearer's personal perspective enhancement information, such as aiming guide lines, ballistic predictions, and target markers.

[0298] The method for synchronizing the coordinates of AR glasses and motion-sensing gun is as follows: First, the position and orientation of the AR glasses in space are obtained through a tracking system; then, the pointing vector of the motion-sensing gun is converted into the local coordinate system of the AR glasses; finally, aiming assistance information is rendered on the display screen of the AR glasses according to the converted pointing vector.

[0299] Because the relative position of the AR glasses and the user's head is fixed, the content displayed on the AR glasses changes as the user turns their head, but the aiming guide line corresponding to the motion-sensing gun always aligns with the actual direction of the motion-sensing gun, unaffected by the user's head movement. This feature allows users to maintain an immersive environment while precisely aiming using the motion-sensing gun.

[0300] By adopting the above technical solution, the wearable augmented reality display device linkage system provides personalized perspective enhancement functions for each user while maintaining a shared immersive environment for multiple users, thus expanding the application scenarios and interaction possibilities of the system.

[0301] Example 29 In some preferred embodiments of this application, an immersive aiming interaction system linked to a motion platform is provided, which is suitable for application scenarios such as simulated driving, simulated flight, or motion experience.

[0302] In practical implementation, the motion platform serves as a movable base, such as a six-degree-of-freedom platform, a hydraulic platform, or an electromechanical servo platform. The movement of the motion platform can affect the perspective and interactive effects in the virtual environment. For example, in a flight simulation experience, the platform's pitch and roll are synchronized with the attitude of the virtual aircraft.

[0303] I. Linkage Logic between Motion Platform and Motion Sensing Device The interaction between the motion platform and the motion sensing device is a two-way signal interaction, and the specific signal flow is as follows: Forward control flow (motion sensing → platform): Changes in the user's posture while operating the motion sensing device (such as aiming direction or firing trigger) are captured by the system and used as one of the input signals to trigger platform movement. For example, when the user turns quickly, the platform generates corresponding yaw motion feedback; when the user fires, the platform generates a slight recoil vibration. The signal path of the forward control flow is: motion sensing device → host computer → platform controller → platform driver → platform movement.

[0304] Reverse Feedback Stream (Platform → Motion / Visual): When the platform actively moves according to the game's storyline or simulated content (such as a simulated aircraft rolling or a vehicle bumping), the platform's motion signal is synchronously sent to the rendering engine to adjust the virtual scene's perspective and environment. Simultaneously, tactile feedback is transmitted to the user through the vibration motors or force feedback devices of the motion sensing equipment. The signal path of the reverse feedback stream is: Game Engine → Platform Controller → Platform Motion + Rendering Perspective Update + Motion Feedback.

[0305] Synchronization mechanism: Platform movement, scene rendering and motion feedback are synchronized through a unified timestamp, with synchronization accuracy controlled within 16ms (corresponding to a 60Hz refresh rate) to avoid dizziness caused by asynchrony between vision and motion.

[0306] II. Definition of Fixed Reference Points in Dynamic Platform Scenarios In motion platform scenarios, because the motion-sensing device moves along with the platform, the definition of the originally "fixed in space" reference point needs to be adjusted accordingly. The specific strategy is as follows: Reference point follows platform movement: When the motion sensing device is mounted on the motion platform, the fixed reference position (starting point) of the direction vector is no longer a fixed point in the world coordinate system, but a point fixed relative to the platform base. This reference point is fixed at a certain position on the platform (such as the center of the motion sensing gun's axis of rotation, the front end of the seat armrest, etc.) and moves together with the platform's translation and rotation.

[0307] Reference point coordinate update: Let the position of the reference point on the platform base in the platform's local coordinate system be P_ref_local (fixed), the platform's pose in the world coordinate system be Q_platform (quaternion), and the position of the platform center in the world coordinate system be P_platform. Then the real-time position of the reference point in the world coordinate system is: P_ref_world = P_platform + R(Q_platform) · P_ref_local Where R(Q_platform) is a rotation matrix constructed from the platform attitude quaternions.

[0308] Calculation of the direction vector: Given that the attitude of the motion sensor gun relative to the platform is Q_gun_relative (calculated using the independent inertial reference frame of the inertial sensor), the direction vector of the motion sensor gun in the world coordinate system is: dir_world = R(Q_platform) × R(Q_gun_relative) · dir_local Where dir_local is the initial direction vector of the motion sensor in the local coordinate system (e.g., along the Z-axis directly in front).

[0309] Intersection calculation: Using P_ref_world as the starting point and dir_world as the direction vector, perform ray-plane intersection calculations with each display surface. Since the position of the display surface remains fixed in the world coordinate system, the real-time movement of the reference point ensures that the user's aiming direction and visual observation direction remain consistent throughout the platform's movement.

[0310] III. Independent Inertial Reference Frame for the Motion-Sensing Gun's Posture When mounting a motion-sensing gun on a motion platform, the impact of the platform's motion on the gun's attitude needs to be considered. Specifically, the inertial measurement unit inside the motion-sensing gun uses an independent inertial reference frame, ensuring its attitude output remains unchanged regardless of the motion platform's movement. The specific implementation method is as follows: First, during the static calibration phase, the initial attitude and position of the motion sensor gun relative to the platform base are recorded as a reference for subsequent calculations. Then, during the dynamic operation phase, the platform's real-time motion attitude (Eulerian angles or quaternions) is acquired through the platform's motion sensors, while the motion sensor gun's attitude relative to the inertial reference frame is acquired through its internal inertial measurement unit. Finally, through coordinate transformation, the motion sensor gun's attitude relative to the inertial reference frame is converted to its attitude relative to the platform base, eliminating the influence of platform motion and restoring the actual pointing of the user's hand.

[0311] The specific method for coordinate transformation is as follows: Let the attitude quaternion of the motion sensor gun relative to the inertial reference frame be Q_gun, and the motion attitude quaternion of the platform be Q_platform. Then the attitude quaternion of the motion sensor gun relative to the platform base is Q_relative = Q_platform. -1 × Q_gun, where Q -1 This represents the inverse operation of quaternions.

[0312] By adopting the above technical solution, the motion platform linkage system realizes two-way linkage between the motion sensing device and the motion platform. Through the dynamic reference point definition and real-time coordinate update under the platform coordinate system, the accuracy and stability of aiming interaction during platform movement are ensured, the interference of platform movement on motion sensing aiming is eliminated, and users can still accurately control the direction of interactive indicators during platform movement. This achieves the natural integration of motion sensing aiming and the motion platform, and enhances the realism and immersion of the simulation experience.

[0313] Example 30 In some preferred embodiments of this application, a system architecture that supports multi-user parallel aiming interaction is provided, which is suitable for multi-player confrontation or multi-user collaborative experience scenarios.

[0314] In practice, each user has an independent motion-sensing input module, and each module contains an inertial sensor and a unique communication identifier. The system independently allocates a rendering channel and a data processing thread to each motion-sensing input module to ensure parallel processing capabilities in multi-user scenarios.

[0315] In the system architecture, the direction calculation module independently calculates the corresponding spatial direction vector for each motion input module; the intersection positioning module processes the intersection calculation of all direction vectors with multiple display surfaces in parallel; and the rendering module assigns an independent rendering layer to the interaction indicator of each user, distinguishing the crosshairs of different users through color coding or style differences.

[0316] The display surface management strategy for multi-user scenarios is as follows: When the interaction indicators of multiple users are located on different display surfaces, they are rendered independently without interference; when the interaction indicators of multiple users are located on the same display surface, the following strategy is adopted to ensure differentiation and readability: Color and style differentiation: Each user's interaction indicator has a unique color code (e.g., red for player 1, blue for player 2, green for player 3, etc.) and style characteristics (e.g., different shapes such as cross, circle, triangle, etc.). Users can customize the crosshair color and style in the game, and the system automatically assigns the most different color combinations to adjacent users to ensure high recognizability.

[0317] Spatial isolation and overlap handling: Spatial isolation is triggered when the distance between the crosshairs of two or more users on the same display surface is less than a preset threshold (such as 50 pixels or 1% of the display surface width). Slight overlap (distance > 20 pixels): The crosshairs maintain their respective colors and styles and are displayed normally. The system automatically adjusts the rendering level (Z-order) to place the crosshair with the player ID on the upper layer or to indicate overlap by alternating flashing. Severe overlap (distance ≤ 20 pixels): A differentiated size enlargement strategy is adopted to enlarge the outer ring of one of the crosshairs by 20% to 50%, so that the two crosshairs form concentric circles or nested structures, and the user can still identify the center position of each crosshair. Complete overlap (distance < 5 pixels): The system displays a special composite crosshair (such as a two-color ring or a numbered multi-lobed shape) and marks the overlapping user's number or name abbreviation next to the crosshair, clearly informing the user that there is currently an aiming overlap.

[0318] Priority and Avoidance Mechanism: The system can set priorities according to the game rules. For example: Local players have the highest priority, and their crosshairs are always fully displayed, while other players' crosshairs are displayed semi-transparently when they overlap. Players in "dead" or "spectator" status will have their crosshair transparency automatically reduced (e.g., to 30%) to avoid interfering with active players; When multiple user crosshairs overlap near the center of the screen, the system achieves visual separation by slightly increasing the distance between each crosshair (3-5 pixels offset per user). The offset direction is radially outward to achieve maximum differentiation with minimal displacement.

[0319] User-configurable strategies: Multi-user crosshair conflict handling strategies can be configured by users or administrators in the system settings, including differentiation methods, threshold parameters, priority rules, etc., to adapt to different game types and scenario requirements.

[0320] By adopting the above technical solution, the multi-user parallel aiming interaction system makes full use of the natural parallel characteristics of attitude-driven systems. Each user's aiming interaction is completely independent and does not interfere with each other. The system does not need to perform complex multi-user coordination processing, which facilitates the realization of a large-scale immersive experience for multiple users in the same venue.

[0321] Example 31 In some preferred embodiments of this application, an immersive aiming interaction system with a curved display surface is provided, which is suitable for experience scenarios that require a wider field of view and a better sense of immersion.

[0322] In practical implementation, the curved display surface is constructed from curved screens or LED modules, with a typical radius of curvature ranging from 3 to 8 meters and an arc coverage range of 90° to 180°. Compared to flat display surfaces, curved display surfaces offer the following advantages: a wider horizontal viewing angle, reducing the difference in viewing angle between the edges and the center; a more natural sense of immersion, enhancing the overall experience; and a reduction in the number of display surfaces, thus lowering system complexity.

[0323] The method for calculating the intersection of a ray and an arc-shaped display surface is as follows: Discretize the arc-shaped display surface into multiple planar segments, and use the aforementioned ray-plane intersection algorithm for each segment; or, parameterize the arc-shaped display surface into an arc surface equation and directly perform the ray-arc surface intersection calculation.

[0324] The method for finding the intersection of the parametric arc surface equations will be illustrated using a cylindrical surface as an example. Let the axis of the cylindrical surface be parallel to the Z-axis, the center be located at (Xc, Yc), and the radius be R. The equation of the cylindrical surface is (x - Xc)² + (y - Yc)² = R². Substituting the ray equation P = P0 + t × dirRay into the cylindrical surface equation, we obtain a quadratic equation in t: [(dirRay.x)² + (dirRay.y)²] × t² + 2[(P0.x - Xc) × dirRay.x + (P0.y- Yc) × dirRay.y]× t + [(P0.x - Xc)² + (P0.y - Yc)² - R²] = 0 Solve the equation and take the smallest solution with a value greater than zero as the intersection point parameter.

[0325] The UV coordinate mapping of a curved display surface needs to consider the curvature effect. Let θ be the angle between the line connecting a point P on the curved surface and the center of the circle and the reference direction. Then the normalized UV coordinates are calculated as follows: U = θ / θ_max V = (Pz - Z_min) / (Z_max - Z_min) Where θ_max is the maximum angle of the arc-shaped display surface, and Z_min and Z_max are the minimum and maximum heights of the display surface in the vertical direction, respectively.

[0326] By adopting the above technical solutions, the curved display surface system provides a wider viewing angle coverage and a better sense of immersion. At the same time, the ray intersection algorithm and UV mapping method can correctly handle the surface geometry effect and ensure the accurate positioning of interactive indicators on the curved display surface.

[0327] Example 32 In some preferred embodiments of this application, a geometric calibration and distortion correction method for the projection system is provided to ensure the accurate positioning of interactive indicator signs.

[0328] In the specific implementation process, the geometric calibration of the projection system includes the following steps: Display surface calibration: Determine the geometric parameters of each display surface in three-dimensional space, including the normal vector of the plane equation, position parameters, width, and height. The geometric parameters can be obtained in several ways: Manual measurement and data entry: The size, position, and orientation angle of each display surface are measured using tools such as a tape measure and laser rangefinder, and the measurement data is manually entered into the system. This method is suitable for simple flat panel display systems, and the accuracy depends on the operator's measurement skill, typically at the centimeter level.

[0329] Automatic calculation using visual calibration algorithms: A camera or depth camera is used to capture the calibration pattern (such as a checkerboard or AprilTag marker) on the display surface. Computer vision algorithms then automatically calculate the position and normal vector of the display surface in space. Common methods include Zhang Zhengyou calibration and homography matrix decomposition. This method offers high accuracy (millimeter-level) and ease of operation, making it the mainstream solution for commercial systems.

[0330] CAD Model Import: For venues or display systems designed using 3D design software (such as AutoCAD, SketchUp, Revit, etc.), the geometric parameters of the display surfaces in the CAD model can be directly exported and imported into the system. This method offers the highest accuracy (depending on the accuracy of the CAD design) and is suitable for the conceptual design phase of new venues and precision installation scenarios.

[0331] Infrared tracking calibration: Using an infrared tracking system (such as OptiTrack, Vicon, etc.), the calibration tool is positioned at different locations on the display surface, and the plane equation of the display surface is fitted by the coordinates of the marker points on the tracking tool. This method offers the highest accuracy (sub-millimeter level) and is suitable for scientific research or military applications requiring extremely high precision.

[0332] Automatic calibration: The system projects a feature pattern onto the display surface via a projector. A camera then captures the pattern and automatically calculates the geometric parameters of the display surface, requiring no manual intervention. This automatic calibration solution is suitable for scenarios requiring periodic recalibration.

[0333] The calibration accuracy directly affects the accuracy of ray intersection calculations; a calibration accuracy better than 5 mm is recommended.

[0334] Projector intrinsic parameter calibration: This involves determining the internal parameters of each projector, including focal length, principal point position, and distortion coefficient. Calibration methods can include Zhang Zhengyou's calibration method or active calibration. Intrinsic parameter calibration is used to calculate the distortion mapping function of the projected image.

[0335] Distortion correction mapping: Based on the projector's intrinsic parameters and the projection screen geometry, calculate the mapping function from the coordinates of the projected image to the three-dimensional coordinates of the actual projection point. For ultra-short-throw projectors, the main distortion is barrel distortion, which can be corrected using a polynomial model. x_corrected = x × (1 + k1 × r² + k2 × r 4 + k3 × r 6 ) + p1 × (r² +2x²) + 2 × p2 × x × y y_corrected = y × (1 + k1 × r² + k2 × r 4 + k3 × r 6 ) + 2 × p1 × x× y + p2 × (r² + 2y²) Where: - x and y are normalized image coordinates (i.e., dimensionless coordinates obtained by dividing image coordinates by focal length, with the principal point of the image as the origin), in pixels per focal length; - r² = x² + y², the square of the normalized radial distance; - k1, k2, and k3 are radial distortion coefficients used to describe barrel or pincushion distortion. For ultra-short-throw projectors, the typical value range of k1 is -0.5 to -0.2 (barrel distortion, negative value), the typical value range of k2 is -0.1 to 0.1, and k3 is a higher-order correction term (usually small, can be set to 0); - p1 and p2 are tangential distortion coefficients used to describe tangential distortion caused by lens mounting tilt, with a typical value range of -0.05 to 0.05. For most commercial projectors, tangential distortion is much smaller than radial distortion and can be ignored when high precision requirements are not required.

[0336] The polynomial model described above is applicable to conventional wide-angle and ultra-short-throw projection lenses. For fisheye lenses (such as fisheye projectors used in spherical screens), distortion becomes more severe, requiring the use of specialized fisheye distortion models such as equidistant projection and equisolid angle projection for correction. In these cases, the polynomial model is no longer applicable.

[0337] The distortion coefficients are automatically calculated using a calibration algorithm (such as the Zhang Zhengyou calibration method), eliminating the need for manual input.

[0338] Distortion correction for interactive indicator positions needs to be performed after the ray intersection calculation is completed. The correction method is as follows: First, map the intersection point of the ray and the display surface to the projected image coordinates; then, apply the distortion correction mapping to obtain the corrected image coordinates; finally, convert the corrected image coordinates into the final rendering position.

[0339] Distortion correction in multi-projector blending areas is more complex. When interactive indicators are located in the overlapping area of ​​multiple projector screens, the rendering position on each projector's screen needs to be calculated separately, and corrections made independently based on the distortion parameters of each projector. Boundary processing of the blending area can employ a weighted blending method, determining the weight based on the distance between the intersection point and the boundaries of each projector's screen.

[0340] By adopting the above technical solution, the geometric calibration and distortion correction method ensures the accurate positioning of interactive indicators in the projection system. Through layered calibration and correction strategies, it can effectively compensate for various geometric distortions in the projection system and improve the user's aiming accuracy experience.

[0341] Example 33 In some preferred embodiments of this application, methods for system latency optimization and real-time performance guarantee are provided to ensure timely interactive responses.

[0342] In practical implementation, system latency consists of the following components: inertial sensor sampling latency (5ms), data transmission latency (2ms to 10ms, depending on the communication method), attitude fusion calculation latency (2ms), ray intersection calculation latency (0.5ms), and rendering latency (depending on the frame rate, typically 8ms to 16ms). End-to-end system latency is usually between 20ms and 40ms.

[0343] Delay optimization strategies include the following aspects: Sampling rate optimization: The sampling rate of inertial sensors needs to strike a balance between accuracy and latency. An excessively high sampling rate increases data transmission overhead and processing latency, while an excessively low sampling rate leads to sluggish response. A sampling rate of 100Hz to 200Hz is recommended, as it covers the frequency range of most human movements.

[0344] Communication protocol optimization: Employ low-latency wireless communication protocols, such as a proprietary 2.4GHz protocol or Bluetooth Low Energy 5.0. Communication latency should be controlled within 2ms. If necessary, predictive algorithms can be used to send attitude data in advance to further reduce perceived latency.

[0345] Rendering pipeline optimization: Utilizing the game engine's multi-viewport rendering capabilities, the rendering of interactive indicators is processed in parallel with scene rendering. The rendering of interactive indicators uses a user interface layer (UI Layer) approach, unaffected by scene complexity, ensuring continuous real-time updates.

[0346] Prediction and extrapolation: When rapid user movement is detected, linear prediction or Kalman filter prediction algorithms can be used to estimate the trend of posture change in advance and extrapolate the position of interactive indicators to compensate for the perception lag caused by rendering delay.

[0347] By employing the above technical solution and optimizing latency across the entire link, the system can achieve an end-to-end latency of less than 40ms, meeting the response speed requirements of real-time shooting games. Users can enjoy a sensitive and responsive aiming experience without perceptible operational lag.

[0348] Example 34 In some preferred embodiments of this application, a display surface adaptive switching and dynamic configuration method is provided to improve the system's flexibility and maintainability.

[0349] In practice, the adaptive switching mechanism for display surfaces allows the system to adjust the display surface configuration without restarting, adapting to different site conditions and usage requirements.

[0350] Display surface configuration parameters include: number of display surfaces (2 to 6 configurable), geometric parameters of each display surface (position, size, orientation), display surface type (planar, curved, spherical), and projection device parameters (intrinsic parameters, extrinsic parameters, distortion coefficient).

[0351] The configuration switching can be triggered in the following ways: manual switching, by selecting a preset configuration scheme through the administrator interface; automatic detection, by automatically identifying the current display surface layout through a site perception system (such as an infrared sensor array); and scene linkage, by automatically switching to the corresponding display surface configuration based on the currently running application scenario.

[0352] The implementation of dynamic configuration requires the system to have the following capabilities: real-time updates of display surface geometry parameters without reloading the entire scene; dynamic adjustment of the number and resolution of rendering viewports; and automatic re-execution of the geometry calibration process (if necessary).

[0353] By adopting the above technical solution, the adaptive switching and dynamic configuration method of the display surface enables the system to flexibly adapt to various deployment environments, reduces the difficulty of installation and debugging, and improves the versatility and maintainability of the product.

[0354] Example 35 In some preferred embodiments of this application, an immersive multi-display interactive system in the form of a melee weapon (simulated toy model, non-aggressive) is provided, which is suitable for game scenarios with combined shooting and melee interaction.

[0355] In practical implementation, the motion-sensing device adopts a motion-sensing control device in the form of a melee weapon (simulated toy model, non-aggressive), including but not limited to swords, shields, clubs, and other melee weapons (simulated toy models, non-aggressive). The melee weapon (simulated toy model, non-aggressive) motion-sensing device is equipped with an inertial sensor to output attitude angles. The system determines a direction vector based on the attitude angles and calculates the intersection of this direction vector with multiple display surfaces to determine the melee interaction area and collision detection location.

[0356] For motion-sensing devices such as those for swords, the starting point of the direction vector is located along the blade's extension direction. Based on the intersection calculation, the system further calculates the projected trajectory of the blade onto multiple display surfaces. When the user performs a slashing motion, the inertial sensor collects attitude angle changes at a high frequency, and the system updates the direction vector in real time, rendering interactive indicators and trajectory afterimages corresponding to the slashing trajectory on the display surfaces. Collision detection is based on the intersection of the direction vector and the bounding box of the virtual object; a collision is considered successful when the intersection point is within the bounding box of the virtual object.

[0357] For shield-type motion-sensing devices, the attitude angle is used to determine the orientation and range of the blocking area. The system determines whether blocking is effective based on the angle between the shield's normal vector and the incoming direction vector. When the angle is less than a preset threshold, a protective shield effect is rendered on the display surface.

[0358] In scenarios involving combined shooting and melee combat, a single user may hold multiple motion-sensing devices. For example, one user might hold a gun-shaped motion-sensing device for aiming, while another holds a melee weapon (simulating a toy model, non-aggressive)-shaped motion-sensing device for blocking or attacking. The system independently calculates direction vectors and renders interactive indicators for each motion-sensing device. Both interactive indicators are visible simultaneously in a multi-display environment without interfering with each other. The rendering priority of the shooting and melee interactive indicators is dynamically adjusted based on the current game state.

[0359] By adopting the above technical solution, the immersive interactive system for melee weapons (simulated toy models, non-aggressive) expands the form coverage of motion-sensing interaction, enables shooting games to have melee capabilities, and provides a technical foundation for subsequent pure melee fighting games.

[0360] Example 36 In some preferred embodiments of this application, a full-body sensory multi-part interaction system is provided, which is suitable for immersive interaction scenarios that require capturing the user's full-body movements.

[0361] In practice, multiple motion-sensing input modules are deployed on different parts of the same user's body, including but not limited to key areas such as the wrist, ankle, and waist. Each motion-sensing input module has an independent inertial sensor, and each module independently outputs an attitude angle. The system independently calculates the direction vector based on the attitude angle of each module and renders the corresponding interactive indicator on multiple display surfaces.

[0362] The full-body skeletal posture is generated using a kinematic chain solution method: the waist module provides the basic trunk posture, the wrist module provides the upper limb posture, and the ankle module provides the lower limb posture. Based on the relative positional relationships and posture angles of each module, the system generates the complete full-body skeletal posture through inverse kinematics (IK) and drives the corresponding virtual character in the virtual environment.

[0363] In a full-body motion-sensing interaction scenario, the wrist module controls hand interaction indicators (shooting aiming or melee attack), the ankle module controls the virtual character's movement direction and pace, and the waist module controls the virtual character's torso orientation and posture. The interaction indicators of each module are rendered separately in a multi-display environment, allowing users to achieve complex motion-sensing interactions such as shooting, weapon combat, and hand-to-hand combat through full-body movements.

[0364] Using the above technical solution, the full-body motion-sensing multi-part interaction system expands from a single handheld motion-sensing device to multi-part collaborative motion-sensing interaction, supporting comprehensive gameplay such as shooting, weapon combat, and melee combat. At the same time, the six-degree-of-freedom tracking that combines optical and inertial tracking ensures the accuracy and reliability of the interaction.

[0365] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An immersive multi-display-surface interactive aiming method for an immersive space containing multiple projection display surfaces, characterized in that, Specifically, the following steps are included: S1. Obtain the attitude angles output by the inertial sensor in the motion sensing device, wherein the attitude angles include yaw angle and pitch angle; S2. Construct a spatial direction vector based on the attitude angle. The spatial direction vector starts from a fixed reference position in space, and its direction is determined by the yaw angle and the pitch angle. S3. Using the spatial direction vector as a ray, perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray; S4. Select the display surface where the nearest intersection point is located as the target display surface according to the distance, and use the intersection point as the target location of the interactive indicator in three-dimensional space; S5. Map the target position to display coordinates on the target display surface; S6. Render the interactive indicator on the target display surface at the display coordinates.

2. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The construction of the spatial direction vector in step S2 includes the following sub-steps: S21. The raw data output by the inertial sensor is fused into an attitude quaternion q=(q0, q1, q2, q3); S22. Calculate the spatial direction vector based on the attitude quaternion, using at least one of the following methods: The attitude quaternion is converted into the yaw angle and the pitch angle according to the ZYX internal rotation order, and then the spatial direction vector is calculated using trigonometric formulas based on the yaw angle and the pitch angle. The spatial direction vector is calculated directly using quaternion rotation: dir world = q · dir local · q - ¹, where dir local Let q be the initial direction vector in the local coordinate system. - ¹ represents the conjugate of the attitude quaternion, dir world The spatial direction vector; S23. Normalize the spatial direction vector.

3. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The intersection calculation in step S3 includes the following sub-steps: S31. Obtain the geometric parameters of each display surface. The geometric parameters include at least the type, spatial position, size, and normal vector of the display surface. The type of the display surface includes at least one of planar display surface, curved display surface, spherical display surface, and hemispherical display surface. S32. Select the corresponding intersection algorithm according to the type of the display surface; S33. Perform the corresponding intersection algorithm on each display surface to obtain the coordinates of the intersection point of the spatial direction vector with each display surface and the intersection point parameter t, where t represents the distance parameter from the fixed reference position to the intersection point; S34. Select intersections where t>0 as valid intersections.

4. The immersive multi-display surface interactive aiming method as described in claim 3, characterized in that, The intersection algorithm described in step S32 includes at least one of the following: If the display surface is a planar display surface, then the intersection algorithm of a ray and a plane is used, and the calculation formula is as follows: in,( A , B , C () is the unit normal vector of the display surface. D The signed distance from the origin to the display surface. P 0=( , , ) is the fixed reference position. dir =( , , ) is the spatial direction vector; If the display surface is an arc surface, then the arc surface is discretized into multiple planar segments and the intersection calculation of the ray and the plane is performed respectively, or the intersection algorithm of the ray and the parameterized arc surface is used. If the display surface is a spherical or hemispherical display surface, then the intersection algorithm between the ray and the sphere is used, and the calculation formula is as follows: in,( , , ( ) represents the coordinates of the sphere's center. R Let t be the radius of the sphere. Solve the quadratic equation in one variable and take the solution with the smallest value of t that is greater than zero as the intersection point parameter.

5. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, For flat or curved display surfaces, the mapping in step S5 includes the following sub-steps: S511. Establish a local coordinate system on the target display surface, wherein the U-axis of the local coordinate system is parallel to the horizontal direction of the target display surface, and the V-axis is parallel to the vertical direction of the target display surface; S512. Calculate the offset of the target position in the local coordinate system, and project the offset along the U-axis and V-axis directions of the target display surface to obtain the projection values ​​u and v; S513, based on the width of the target display surface width and height height The projection values ​​are normalized to UV coordinates: S514. Map the normalized UV coordinates to the screen pixel coordinates of the target display surface: in screen_width and screen_height The rendering resolution corresponding to the target display surface.

6. The immersive multi-display surface interactive aiming method as described in claim 5, characterized in that, Step S514 is followed by a perspective distortion correction sub-step: S515. Obtain the distortion parameters of the projection device corresponding to the target display surface, wherein the distortion parameters include radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2; S516. Apply the distortion correction mapping function to the normalized UV coordinates: in, r ² = + ; S517, with the corrected coordinates and Alternative and Perform the mapping in step S514.

7. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, For spherical or hemispherical display surfaces, the mapping in step S5 includes the following sub-steps: S521. Establish a local spherical coordinate system with the center of the sphere as the origin; S522. Calculate the latitude angle φ and longitude angle θ of the target position P in the local spherical coordinate system. The latitude angle φ is the angle between the line connecting the target position and the center of the sphere and the equatorial plane. The longitude angle θ is the angle between the projection of the target position on the equatorial plane and the reference direction. S523. Normalize the latitude and longitude angles to UV coordinates: S524. Map the normalized UV coordinates to the screen pixel coordinates of the spherical or hemispherical display surface: in screen_width and screen_height The rendering resolution corresponding to the target display surface.

8. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The rendering in step S6 employs at least one of the following methods: The interactive indicator is overlaid and rendered at the screen pixel coordinates of the target display surface using a two-dimensional user interface. A three-dimensional crosshair model is placed at the three-dimensional spatial coordinates of the target location for rendering using a three-dimensional spatial rendering method.

9. The immersive multi-display interactive aiming method as described in claim 1, characterized in that, The method also includes a cross-display plane transition step S7, as follows: S71. A transition area is preset at the junction of adjacent display surfaces. For a planar display surface, the width of the transition area is a preset proportional range of the width of the display surface; for a curved or spherical display surface, the width of the transition area is a preset proportional range of the arc dimension of the display surface in the direction perpendicular to the junction line at the junction. S72. Determine whether the interactive indicator has entered the transition area. If it has not entered, render the interactive indicator with full transparency only on the current display surface. S73. If the interactive indicator enters the transition area, the same direction vector is used to calculate the intersection with the current display surface and the adjacent display surface, respectively, to obtain two independent world coordinates of the intersection point, which are then mapped to the screen pixel coordinates of the corresponding display surface. When the distance between the intersection point and the corresponding boundary line is less than a preset threshold, the interactive indicator is rendered simultaneously on the current display surface and the adjacent display surface. The transparency of the interactive indicator on the current display surface is determined according to the distance d from the boundary line and the width W of the transition area in the following manner: The transparency of interactive indicator marks on adjacent display surfaces is determined as follows: S74, when the interactive indicator fully enters the adjacent display surface, that is... When =0, rendering stops on the current display surface, and the interactive indicator is rendered only on the adjacent display surface with full transparency.

10. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The display surface also includes a transparent display layer disposed in front of the projection surface projected by the projection device, the transparency of the transparent display layer being adjustable within a preset range; The rendering described in step S6 includes two-layer rendering: The three-dimensional scene content is rendered on the projection surface as the main display layer; At least a portion of the interactive indicator and two-dimensional enhancement information are rendered on the transparent display layer as an auxiliary display layer; The display coordinates of the main display layer and the auxiliary display layer are aligned to ensure spatial consistency.

11. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The method also includes a multi-sensory device parallel processing step S8, as follows: S81. Obtain the attitude angles independently output by multiple motion-sensing input modules. Each motion-sensing input module has a unique communication identifier. Different motion-sensing input modules are distinguished and rendering channels are allocated independently based on the communication identifier. S82. For each motion input module, execute steps S2 to S6 independently to calculate the target display surface and display coordinates of the corresponding interactive indicator; S83. On the target display surface, render the corresponding interactive indicator for each motion input module independently. The interactive indicator of different motion input modules are distinguished by color coding or style differences.

12. The immersive multi-display surface interactive aiming method as described in claim 11, characterized in that, When the interactive indicators corresponding to multiple motion-sensing input modules are located on the same display surface, a conflict resolution sub-step is also included: S84. Calculate the pixel distance of each interactive indicator on the display surface; S85. When the pixel distance is less than a preset threshold, the following at least one strategy shall be applied: (a) Maintain differentiation through color coding and style differences; (b) Enlarge the size of one of the interactive indicator icons to form a nested structure; (c) Display the combined crosshair and mark the identification information of overlapping users; (d) Adjust the transparency of each interactive indicator according to the preset priority; (e) Offset each interactive indicator outward radially by different pixels.

13. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The method also includes motion platform linkage steps: U1. Obtain the motion attitude of the motion platform, including the platform's position in the world coordinate system. and attitude quaternions ; U2. Update the real-time position of the fixed reference position in the world coordinate system according to the motion posture of the platform: in, The fixed reference position is the fixed position in the platform's local coordinate system. Let be the rotation matrix constructed from the platform attitude quaternions; U3. Obtain the relative posture Q_gun_relative of the motion sensing device with respect to the motion platform; U4. Calculate the orientation vector of the motion sensing device in the world coordinate system based on the platform attitude and the relative attitude Q_gun_relative. The calculation formula is as follows: in, This is the initial orientation vector of the motion sensing device in the local coordinate system; U5, with P ref _ world Starting point dir world Given the direction vector, execute steps S3 to S6.

14. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, It also includes the linkage steps for wearable augmented reality display devices: V1. Obtain the position and orientation of wearable augmented reality display devices in space through a tracking system; V2. Transform the spatial direction vector into the local coordinate system of the wearable augmented reality display device; V3. Render aiming assistance information on the display screen of the wearable augmented reality display device according to the converted direction vector.

15. The immersive multi-display surface interactive aiming method as described in claim 1, characterized in that, The fixed reference position is set in at least one of the following positions: User's chest area; The mounting base location for the motion sensing device is specifically the center of the device's rotating shaft or the universal joint fulcrum. The midpoint between the user's eyes; The game's designated positions are preset in the game area.

16. An immersive multi-display interactive aiming system, characterized in that, include: The motion input module includes an inertial sensor for outputting attitude angles; The orientation calculation module is used to determine a spatial orientation vector based on the attitude angle, wherein the spatial orientation vector is constructed from a fixed reference position in space. The intersection positioning module is used to take the spatial direction vector as a ray and perform intersection calculations with multiple display surfaces in the immersive display environment to determine the intersection point corresponding to each display surface and the distance between the intersection point and the starting point of the ray. Based on the distance, the display surface where the closest intersection point is located is selected as the target display surface, and the intersection point is used as the target position of the interactive indicator in three-dimensional space. A coordinate mapping module is used to map the target position to display coordinates on the target display surface; A rendering module is used to render the interactive indicator on the target display surface at the display coordinates.

17. An immersive multi-display interactive aiming system as described in claim 16, characterized in that, The intersection positioning module includes: The display surface parameter storage unit is used to store the geometric parameters of each display surface. The geometric parameters include at least the type, spatial position, size and normal vector of the display surface. The type of the display surface includes at least one of the following: planar display surface, curved display surface, spherical display surface and hemispherical display surface. The intersection algorithm selection unit is used to select the corresponding intersection algorithm according to the type of the display surface; The intersection calculation unit is used to execute the corresponding intersection algorithm for each display surface to obtain the coordinates of the intersection point of the spatial direction vector with each display surface and the intersection point parameter t; The target display surface determination unit is used to filter intersections where t>0 and select the display surface containing the nearest intersection as the target display surface.

18. An immersive multi-display interactive aiming system as described in claim 16, characterized in that, The system also includes a cross-display-plane transition module for: A transition area is preset at the junction of adjacent display surfaces; When the interactive indicator enters the transition area, the same direction vector is intersected with the current display surface and the adjacent display surface respectively to obtain two independent intersection points, which are mapped to the screen pixel coordinates of the corresponding display surface. The interactive indicator is rendered on both the current display surface and the adjacent display surface. The transparency of the interactive indicator on the current display surface decreases dynamically according to the distance from the boundary line, and the transparency of the interactive indicator on the adjacent display surface increases dynamically according to the distance from the boundary line. When the interactive indicator fully enters the adjacent display surface, rendering stops on the current display surface and only occurs on the adjacent display surface.

19. An immersive multi-display interactive aiming system as described in claim 16, characterized in that, The rendering module supports at least one of the following rendering methods: Two-dimensional user interface rendering method: The interactive indicator is overlaid and rendered at the screen pixel coordinates of the target display surface; 3D spatial rendering method: Place a 3D crosshair model at the 3D spatial coordinates of the target location for rendering.

20. An immersive multi-display interactive aiming system as described in claim 16, characterized in that, It also includes a parallel processing module for: The attitude angles output independently by multiple motion-sensing input modules are obtained. Each motion-sensing input module has a unique communication identifier. Different motion-sensing input modules are distinguished and rendering channels are allocated independently based on the communication identifier. Each motion-sensing input module independently calculates its corresponding target display surface and display coordinates for interactive indicators. On the target display surface, a corresponding interactive indicator is rendered independently for each motion-sensing input module. The interactive indicator of different motion-sensing input modules is distinguished by color coding or style differences.

Images