3D dynamic positioning and rendering system and method based on binocular camera and IMU

Abstract

The present invention discloses a 3D dynamic positioning and rendering system and method based on a binocular camera and an IMU. The 3D dynamic positioning and rendering method based on a binocular camera and an IMU includes: S1. receiving the 3D coordinates of an Aruco marker set in an environment obtained by a binocular camera in a handheld interactive prop, and receiving IMU data obtained in real time by an IMU sensor in the handheld interactive prop, and fusing the IMU data with the 3D coordinates of the Aruco marker to obtain the posture of the handheld interactive prop; S2. receiving a facial image captured by a binocular camera, and using the posture of the handheld interactive prop to convert the human eye position in the facial image into a global coordinate system; wherein the Aruco marker set in the center of the naked eye 3D display is used as the origin of the global coordinate system; S3. calculating virtual camera parameters in real time according to the data converted from the human eye position to the global coordinate system and the parameters of the display, and rendering the virtual scene using the calculated virtual camera parameters using asynchronous rendering.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and human-computer interaction technology, and in particular to a 3D dynamic positioning and rendering system and method based on a binocular camera and an IMU. Background Art

[0002] With the booming development of virtual reality (VR), augmented reality (AR) and naked-eye 3D display technologies, the accurate rendering of dynamic scenes and the optimization and upgrading of user visual experience have become core issues that the industry is tackling.

[0003] Traditional naked-eye 3D display technologies, such as traditional naked-eye 3D advertising screens and fixed-viewpoint stereoscopic displays, all use gratings, cylindrical lenses or parallax barrier technology to present stereoscopic images within a specific viewing angle range. They are limited by the viewing angle range, and users can only feel the stereoscopic effect within a preset area. Once this range is exceeded, the 3D visual effect will be significantly reduced or even disappear completely, greatly limiting the user's freedom of movement.

[0004] At the same time, multi-camera systems such as CAVE virtual reality systems and multi-camera AR navigation systems deploy multiple cameras to capture user positions, combine visual algorithms to calculate human eye coordinates, and drive dynamic rendering of the picture. Although they can achieve large-scale user position tracking, the deployment cost is high and the accuracy and quantity of hardware equipment are extremely high. In addition, in complex lighting or occlusion environments, the tracking effect is easily affected, resulting in system failure.

[0005] In addition, glasses positioning solutions based on reflective markers, such as the OptiTrack optical motion capture system and VR laboratory positioning solutions, require users to wear glasses equipped with reflective markers, capture the positions of the markers through infrared cameras, and infer the coordinates of the human eye in space. Although these solutions can achieve high-precision positioning at the millimeter level, they force users to wear additional equipment, which seriously affects the comfort and convenience of use and makes it difficult to meet the market demand for unconstrained naked-eye interaction.

[0006] In summary, existing technologies have significant defects in terms of dynamic interactive freedom, multi-user support, deployment cost control, and environmental adaptability. There is an urgent need for a naked-eye 3D dynamic rendering solution that does not require complex deployment and can support the free movement of multiple users to fill the market gap and achieve a perfect balance between low cost and high freedom. Summary of the Invention

[0007] The purpose of the present invention is to address the defects of the existing technology and provide a 3D dynamic positioning and rendering system and method based on a binocular camera and an IMU.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0009] 3D dynamic positioning and rendering method based on binocular camera and IMU, including:

[0010] S1. Receive the 3D coordinates of the Aruco markers set in the environment obtained by the binocular camera in the handheld interactive prop, and receive the IMU data obtained in real time by the IMU sensor in the handheld interactive prop, and fuse the IMU data with the 3D coordinates of the Aruco markers to obtain the position and posture of the handheld interactive prop;

[0011] S2. Receive a facial image captured by a binocular camera and transform the eye position in the facial image into a global coordinate system using the handheld interactive prop; wherein the Aruco marker located at the center of the naked-eye 3D display is used as the origin of the global coordinate system;

[0012] S3. Calculate virtual camera parameters in real time based on the data converted from the human eye position to the global coordinate system and the parameters of the display, and render the virtual scene using the calculated virtual camera parameters using asynchronous rendering.

[0013] Furthermore, the step S1 is specifically as follows:

[0014] S11. Receive the Aruco markers set in the environment acquired in real time by the binocular camera in the handheld interactive prop, and calculate the 3D coordinates of the Aruco markers based on the binocular parallax;

[0015] S12. Receive IMU data acquired in real time by the IMU sensor in the handheld interactive prop;

[0016] S13. Fuse the IMU data with the 3D coordinates of the Aruco markers through an extended Kalman filter to predict and output the 6-DOF pose of the handheld interactive prop.

[0017] S14. Bind the 6-DOF pose of the handheld interactive prop to the global coordinate system according to the 6-DOF pose.

[0018] Furthermore, before step S11, the following steps are further included:

[0019] S10. The handheld interactive props are pre-calibrated using a calibration board to calibrate the intrinsic parameters of the binocular cameras and the extrinsic parameters between the binocular cameras and the IMU sensor.

[0020] Furthermore, the step S2 is specifically as follows:

[0021] S21 receives a user's facial image captured by a binocular camera in a handheld interactive prop;

[0022] S22. Use a lightweight eye detection model to locate the 2D coordinates of the user's eyes in the camera coordinate system in the user's facial image;

[0023] S23. Convert the 2D coordinates of the human eye into 3D coordinates in the camera coordinate system;

[0024] S24. Based on the 6-DOF position of the handheld interactive prop, convert the 3D coordinates of the human eye in the camera coordinate system to the global coordinate system.

[0025] Furthermore, in step S24, the 3D coordinates of the human eye in the camera coordinate system are converted to the global coordinate system, which is expressed as:

[0026] ;

[0027] in, Represents the coordinates of the human eye in the global coordinate system; Indicates the pose of the handheld interactive prop; Indicates the 3D coordinates of the human eye in the camera coordinate system.

[0028] Furthermore, the step S3 is specifically as follows:

[0029] S31. Obtain the parameters of the display, obtain the three-dimensional coordinates of the human eye in the global coordinate system, and calculate the perspective projection matrix of the virtual camera in real time based on the three-dimensional coordinates of the human eye in the global coordinate system and the display parameters;

[0030] S32. Obtain the eye position and convert it to the eye posture in the global coordinate system, and set the rotation and translation parameters of the virtual camera according to the eye posture;

[0031] S33. Render virtual scenes of different perspectives according to the perspective projection matrix, rotation parameters, and translation parameters of the virtual camera, and use an asynchronous rendering engine to synthesize the rendered virtual scenes of different perspectives to obtain a rendered multi-perspective image.

[0032] Furthermore, after step S33, the following steps are further included:

[0033] S34. Process the rendered multi-view image through the rasterization and distortion correction module, and output the processed multi-view image to the naked-eye 3D display.

[0034] Furthermore, the parameters of the display in step S31 include size, resolution, and lens curvature.

[0035] Furthermore, after step S3, the following steps are further included:

[0036] S4. Obtain the movement data of the handheld interactive prop and dynamically update the virtual camera parameters based on the movement data.

[0037] Correspondingly, a 3D dynamic positioning and rendering system based on a binocular camera and an IMU is also provided, including a handheld interactive prop, a naked-eye 3D display, and multiple Aruco markers; the handheld interactive prop is provided with a binocular camera, an IMU sensor, and a processor, one of the multiple Aruco markers is fixed at the center of the naked-eye 3D display, and the remaining Aruco markers are fixed in the environment; the processor is used to process the 3D dynamic positioning and rendering method based on the binocular camera and the IMU.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. Full-degree-of-freedom viewing angle support: Users can view glasses-free 3D images from any position and angle, with a viewing angle range of over 180°. When the user moves sideways, the system can dynamically adjust the virtual viewpoint to ensure that the glasses-free 3D effect is not weakened, significantly improving the user's visual experience and interactive freedom.

[0040] 2. Simple Calibration and Deployment: Using a handheld device and environmental reference points, the system pre-calibrates binocular and IMU parameters at the factory, eliminating the need for on-site calibration. Simply attach Aruco markers (a minimum of four required) and the system automatically constructs a spatial map. Compared to multi-camera systems, the calibration process is 95% simplified, allowing non-experts to complete deployment in minutes. Furthermore, support for dynamic expansion and occlusion compensation of Aruco markers improves system robustness by 80%.

[0041] 3. Multi-user support: This solution does not require users to wear special equipment, supports multiple users to move freely in the same scene, and provides each user with clear and accurate naked-eye 3D images, greatly expanding the application scenarios and user groups of the technology.

[0042] 4. Efficient Resource Utilization: Sensor fusion technology, combined with an extended Kalman filter, achieves tight coupling between binocular vision and IMU data, ensuring high-precision and real-time device pose estimation. Simultaneously, the use of a lightweight deep learning model for eye detection and feature extraction effectively reduces computing resource consumption and improves system efficiency and responsiveness.

[0043] 5. Strong environmental adaptability: This technology can operate stably under complex lighting conditions and partially occluded scenes, avoiding tracking failure or cumulative errors caused by environmental changes, and significantly enhancing the reliability and practicality of the system.

[0044] 6. Cost-effectiveness: Compared to traditional multi-camera systems and positioning solutions based on reflective landmarks, this invention significantly reduces hardware deployment costs and complexity. Eliminating the need to install multiple high-precision cameras or equip each user with dedicated equipment makes naked-eye 3D dynamic rendering technology more cost-effective and competitive, facilitating large-scale promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a structural diagram of a 3D dynamic positioning and rendering system based on a binocular camera and an IMU provided in Example 1;

[0046] Figure 2 This is a flow chart of the 3D dynamic positioning and rendering method based on binocular cameras and IMU provided in Example 2. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0048] The purpose of the present invention is to address the defects of the existing technology and provide a 3D dynamic positioning and rendering system and method based on a binocular camera and an IMU.

[0049] Example 1

[0050] This embodiment provides a 3D dynamic positioning and rendering system based on binocular cameras and IMU, such as Figure 1 As shown, it includes a handheld interactive prop 1, a naked-eye 3D display (not shown in the figure), and multiple Aruco markers 2.

[0051] The handheld interactive prop 1 is equipped with a binocular camera (two wide-angle fisheye lenses) 11, an IMU sensor (not shown in the figure), and a processor (not shown in the figure). The binocular camera 11 and the IMU sensor are both communicatively connected to the processor, and the binocular camera 11 of the handheld interactive prop 1 can realize the front and rear functions, thereby realizing the acquisition of Aruco landmarks on the user's face and the environment.

[0052] Fix an Aruco marker (ID=0) at the center of the naked-eye 3D display and use it as the origin of the global coordinate system. Stick multiple (e.g., three) Aruco markers (ID≥1) within the user's activity area (e.g., on the ceiling) to form a spatial positioning reference network with at least four Aruco markers.

[0053] The system of this embodiment pre-calibrates the binocular camera 11 and IMU sensor carried by the handheld interactive prop 1 through existing calibration tools (such as Kalibr, etc.), and then obtains the internal parameters of the binocular camera 11 and the external parameters between the binocular camera 11 and the IMU sensor. Usually, the calibration process is completed when the prop is produced and no additional operation is required by the user.

[0054] This embodiment realizes a dynamic naked-eye 3D effect through a prepared environment and based on the processor of the handheld interactive prop 1.

[0055] Example 2

[0056] This embodiment provides a 3D dynamic positioning and rendering method based on a binocular camera and an IMU. The method is based on the 3D dynamic positioning and rendering system based on a binocular camera and an IMU in the first embodiment. Figure 2 As shown, the 3D dynamic positioning and rendering method includes:

[0057] S1. Receive the 3D coordinates of the Aruco markers set in the environment obtained by the binocular camera in the handheld interactive prop, and receive the IMU data obtained in real time by the IMU sensor in the handheld interactive prop, and fuse the IMU data with the 3D coordinates of the Aruco markers to obtain the position and posture of the handheld interactive prop;

[0058] S2. Receive a facial image captured by a binocular camera and transform the eye position in the facial image into a global coordinate system using the handheld interactive prop; wherein the Aruco marker located at the center of the naked-eye 3D display is used as the origin of the global coordinate system;

[0059] S3. Calculate virtual camera parameters in real time based on the data converted from the human eye position to the global coordinate system and the parameters of the display, and render the virtual scene using the calculated virtual camera parameters using asynchronous rendering.

[0060] When the user uses the handheld interactive prop, turn on the power of the handheld interactive prop and start the binocular camera and IMU sensor on the handheld interactive prop.

[0061] Move and rotate the handheld interactive props in the user's activity area to ensure that the binocular camera can observe each attached Aruco marker. Wait for about one minute to complete the external parameter calibration between the handheld interactive props and the Aruco markers, establish the mapping relationship between the physical space and the virtual space, and then achieve a dynamic naked-eye 3D effect through the following method.

[0062] In step S1, the 3D coordinates of the Aruco marker set in the environment obtained by the binocular camera in the handheld interactive prop are received, and the IMU data obtained in real time by the IMU sensor in the handheld interactive prop are received, and the IMU data is fused with the 3D coordinates of the Aruco marker to obtain the position and posture of the handheld interactive prop; specifically, the following steps are performed:

[0063] S11. Receive Aruco markers set in the environment acquired in real time by the binocular camera in the handheld interactive prop, and calculate the 3D coordinates of the Aruco markers based on binocular parallax.

[0064] The handheld device's binocular cameras capture real-time images of Aruco markers in the detection environment (e.g., ceiling). These images are processed to extract the Aruco marker's feature points. Using the principle of binocular parallax, the 3D coordinates of the Aruco markers in the camera coordinate system are calculated based on the intrinsic and extrinsic parameters of the left and right cameras (the binocular cameras) and the coordinates of the matched feature points. Triangulation is then used to construct a sparse point cloud of the environment, providing geometric constraints for subsequent positioning.

[0065] S12. Receive IMU data acquired in real time by the IMU sensor in the handheld interactive prop.

[0066] The IMU sensor acquires high-frequency IMU attitude data from the handheld interactive prop in real time, including angular velocity and acceleration data. It then pre-integrates the raw IMU data to estimate the relative motion of the handheld interactive prop between two consecutive moments. Pre-integration converts the high-frequency IMU data into changes in displacement, velocity, and attitude, providing motion information for subsequent pose estimation.

[0067] Based on the pre-integration results, a kinematic model is used to predict the current pose of the handheld interactive prop. This kinematic model is typically based on the rigid body motion assumption, taking into account the rotation and translation of the device, and predicting the pose change of the handheld interactive prop by integrating IMU data.

[0068] S13. The IMU data is fused with the 3D coordinates of the Aruco markers through an extended Kalman filter to predict and output the 6-DOF pose of the handheld interactive prop.

[0069] A state vector is predefined, which includes information such as the position, velocity, posture, and bias of the handheld interactive prop. The state vector is predicted using the IMU pre-integration results and motion model to predict the position and posture of the handheld interactive prop at the current moment and the uncertainty of the state.

[0070] The 3D coordinate information of the Aruco marker in the camera coordinate system obtained by the binocular camera is used as the observation value and compared with the predicted state of the handheld interactive prop. By calculating the difference between the observation value and the predicted value, the Kalman filter (EKF) gain is used to update the state vector to obtain a more accurate handheld interactive prop pose estimation. At the same time, the state uncertainty is updated to reflect the degree of improvement of the state estimation due to the observation information.

[0071] After the above fusion steps, the 6-DOF pose of the handheld interactive prop in the camera coordinate system is obtained. .

[0072] S14. Bind the 6-DOF pose of the handheld interactive prop to the global coordinate system according to the 6-DOF pose.

[0073] Position the handheld interactive prop Convert to the global coordinate system centered on the Aruco marker with ID = 0. This usually requires using the known relative position relationship between Aruco markers and the geometric relationship between the handheld interactive props and the Aruco markers to perform coordinate transformation.

[0074] This embodiment maps the pose information of the handheld interactive prop to the global coordinate system, providing an accurate spatial reference for subsequent eye position detection and dynamic adjustment of the virtual space camera. This allows the system to track the movement of the device in the global coordinate system and adjust the virtual camera parameters in real time based on the device's pose changes, achieving dynamic naked-eye 3D rendering.

[0075] In step S2, a facial image captured by a binocular camera is received, and the position of the human eyes in the facial image is converted to a global coordinate system using the posture of the handheld interactive prop; wherein the Aruco marker set at the center of the naked-eye 3D display is used as the origin of the global coordinate system; specifically, the process includes:

[0076] S21. Receive a user's facial image captured by a binocular camera in a handheld interactive prop.

[0077] The user's facial image is captured in real time using a binocular camera mounted on the front of the handheld interactive prop. To ensure detection accuracy, the camera needs to have a high resolution and frame rate. Also, during the acquisition process, the camera should be kept clear of obstructions and the user's face should be within the camera's field of view.

[0078] Preprocessing the captured facial images, including grayscale conversion and denoising, improves the efficiency and accuracy of subsequent detection. Grayscale conversion reduces computational complexity, while denoising removes noise from the image. Common denoising methods include Gaussian filtering.

[0079] S22. Use a lightweight eye detection model to locate the 2D coordinates of the eyes in the user's facial image in the camera coordinate system.

[0080] The pre-processed images are processed using a lightweight deep learning network to detect the location of the pupil centers of both eyes. This lightweight deep learning network, based on a convolutional neural network (CNN), has been trained on a large amount of facial image data and can accurately locate the position of the human eye in an image.

[0081] This embodiment optimizes the deep learning model to meet the computing power and real-time requirements of handheld interactive props. For example, model compression techniques such as pruning and quantization are used to reduce model size and increase inference speed. Furthermore, hardware acceleration capabilities of mobile devices (such as GPU acceleration) can be utilized to further improve detection efficiency.

[0082] The deep learning model outputs the 2D coordinates of the pupil centers of both eyes in the binocular camera coordinate system. These 2D coordinates are typically expressed in pixels and represent the location of the pupil center on the image plane. To improve positioning accuracy, sub-pixel positioning technology can be used. By interpolating the pixel values ​​around the pupil center, a more precise pupil center location is obtained, improving the accuracy of subsequent 3D coordinate conversion.

[0083] S23. Convert the 2D coordinates of the human eye into 3D coordinates in the camera coordinate system.

[0084] The three-dimensional coordinates of the human eye in the binocular camera coordinate system are calculated using the binocular parallax principle and depth estimation model. Based on the difference (parallax) in the two-dimensional coordinates of the human eye in the left and right camera images, combined with the camera's internal parameters (such as focal length, pixel size, etc.), the distance from the human eye to the camera (depth information) is calculated. Then, combining the depth information and two-dimensional coordinates, the human eye position is converted into three-dimensional coordinates in the camera coordinate system through a geometric projection model. .

[0085] To improve the accuracy and robustness of depth estimation, a deep learning-based depth estimation model can be used. For example, a convolutional neural network can be used to process binocular images and directly predict depth maps, thereby obtaining more accurate depth information. Furthermore, stereo matching algorithms (such as semi-global block matching (SGBM)) can be combined to further optimize depth estimation results and reduce errors.

[0086] This embodiment also builds a geometric model of the user's head to account for the impact of head shape and movement on eye position. This model can include three-dimensional structural features of the head, such as eye socket location and head contour. By estimating head posture (such as pitch, yaw, and roll angles) and combining this with the geometric model, it can compensate for perspective shifts caused by head rotation.

[0087] Based on the head pose estimation results, the 3D coordinates of the human eye in the camera coordinate system are adjusted. For example, when the head rotates, the eye position is converted to the coordinate system corresponding to the head pose through geometric transformations (such as rotation matrix multiplication), ensuring accurate calculation of virtual camera parameters.

[0088] S24. Based on the 6-DOF posture of the handheld interactive prop, convert the 3D coordinates of the human eye in the camera coordinate system to the global coordinate system.

[0089] Get the pose of the handheld interactive prop , including position and posture information.

[0090] The three-dimensional coordinates of the human eye in the camera coordinate system Converted to the global coordinate system, it is expressed as:

[0091] ;

[0092] in, Represents the coordinates of the human eye in the global coordinate system; Indicates the pose of the handheld interactive prop; Indicates the 3D coordinates of the human eye in the camera coordinate system.

[0093] During the coordinate conversion process, there may be a certain amount of accumulated errors, such as measurement errors and model errors. To improve the accuracy of the conversion, error correction algorithms, such as Kalman filtering, can be used to correct the converted coordinates in real time. At the same time, the position information of Aruco landmarks in the global coordinate system is used to verify and adjust the conversion results to ensure the accuracy of the human eye position in the global coordinate system.

[0094] In step S3, virtual camera parameters are calculated in real time based on the data of the human eye position converted to the global coordinate system and the parameters of the display, and the calculated virtual camera parameters are used to render the virtual scene using asynchronous rendering; specifically, the following steps are performed:

[0095] S31. Obtain the parameters of the display, and obtain the three-dimensional coordinates of the human eye in the global coordinate system after converting the human eye position into the global coordinate system, and calculate the perspective projection matrix of the virtual camera in real time based on the three-dimensional coordinates of the human eye in the global coordinate system and the display parameters.

[0096] Get the three-dimensional coordinates of the human eye in the global coordinate system =(X eye , Y eye , Z eye ), which is the key input data for calculating the virtual camera parameters.

[0097] Obtain the physical parameters of the display, including its size (width and height), resolution (number of pixels), and lens curvature. These parameters determine the imaging characteristics of the virtual camera. For example, the display's size and resolution determine the virtual camera's viewport size and pixel density, while the lens curvature affects light refraction and imaging quality.

[0098] Based on the human eye position and display parameters, the perspective projection matrix of the virtual camera is calculated using the perspective projection principle. The calculation of the perspective projection matrix usually involves a mathematical transformation that converts 3D world coordinates into 2D screen coordinates. The formula is as follows:

[0099] x screen =(f x ×X eye ) / Z eye +c x

[0100] y screen =(f y ×Y eye ) / Z eye +c y

[0101] Among them, f x and f y is the focal length parameter of the virtual camera, which is usually related to the physical size and resolution of the display; c x and c y The coordinates of the Aruco marker with ID=0 are located at the center of the display; x screen and y screen It is the two-dimensional coordinate of the human eye in the screen coordinate system.

[0102] S32. Obtain the eye position and convert it to the eye posture in the global coordinate system, and set the rotation and translation parameters of the virtual camera according to the eye posture;

[0103] Get the pose of the human eye in the global coordinate system, including position (X eye , Y eye , Z eye ) and orientation (usually represented by a rotation matrix or quaternion). This determines the viewing direction and position of the virtual camera.

[0104] According to the human eye position, the rotation and translation parameters of the virtual camera are initially set. The initial rotation parameters can be determined by the direction of the human eye, and the initial translation parameters can be determined by the position of the human eye.

[0105] To ensure consistency between the virtual camera's image and the human eye's perspective, an optimization algorithm is used to adjust the virtual camera's rotation and translation parameters. For example, numerical optimization methods such as gradient descent can be used to iteratively optimize the parameters, with the objective function being to minimize the difference between the virtual camera's rendered image and the image actually observed by the human eye.

[0106] S33. Render virtual scenes of different perspectives according to the perspective projection matrix, rotation parameters, and translation parameters of the virtual camera, and use an asynchronous rendering engine to synthesize the rendered virtual scenes of different perspectives to obtain a rendered multi-perspective image.

[0107] Obtain the virtual scene data that needs to be rendered, including 3D models, textures, lighting information, etc.

[0108] Using an asynchronous rendering engine, the virtual scene is rendered from different perspectives based on the virtual camera parameters. Asynchronous rendering allows rendering operations to be performed without blocking the main thread, improving the real-time performance and responsiveness of the system.

[0109] Based on the rendering results of different perspectives, multi-perspective images are synthesized. These images contain information about the virtual scene from different angles and are the basis for achieving naked-eye 3D effects.

[0110] S34. Process the rendered multi-view image through the rasterization and distortion correction module, and output the processed multi-view image to the naked-eye 3D display.

[0111] Converting 3D graphics data in a multi-view image into 2D pixel data. This process involves converting the vertex coordinates of the 3D model into screen coordinates, and then performing operations such as filling and shading to generate the final pixel image.

[0112] Because display devices (naked-eye 3D displays) may exhibit optical distortion, such as barrel distortion or pincushion distortion, the rendered image must be corrected for this distortion. Distortion correction algorithms, such as polynomial distortion models, are applied to the image to geometrically transform the image and compensate for optical distortion, ensuring image accuracy and visual quality.

[0113] The rasterized and distortion-corrected multi-view images are output to the naked-eye 3D display, and the corresponding format conversion and adaptation are performed according to the characteristics and requirements of the display.

[0114] This embodiment also includes: S4. Acquiring movement data of the handheld interactive prop, and dynamically updating virtual camera parameters according to the movement data.

[0115] The binocular camera continuously captures Aruco landmarks (ID ≥ 1) in the environment in real time, acquiring image sequences at a high frame rate (e.g., 30 frames per second or higher) to ensure that changes in perspective caused by user movement can be detected in a timely manner.

[0116] The IMU sensor continuously acquires the angular velocity and acceleration data of the handheld interactive prop at a higher frequency (such as 100Hz or higher). This high-frequency data can promptly reflect the small movements of the handheld interactive prop, such as rotation and translation.

[0117] The binocular vision positioning results and IMU data are continuously fused through an extended Kalman filter. Every time new data is obtained, the handheld interactive prop's pose estimation is updated to ensure the real-time and accuracy of the handheld interactive prop's pose information.

[0118] The binocular camera in front of the handheld interactive prop continuously captures the user's facial image stream. Each frame is pre-processed and eye detected, tracking changes in eye position in real time.

[0119] Based on the new image frames and device pose information, the 3D coordinates of the human eye in the camera coordinate system are dynamically updated using the binocular parallax principle and depth estimation model, and then converted to the global coordinate system. This allows for timely reflection of changes in the eye position and provides a basis for adjusting virtual camera parameters.

[0120] An event-triggered mechanism is used to immediately initiate the virtual camera parameter update process when user movement is detected (via changes in the pose of a handheld interactive prop or eye position). A timed polling mechanism is also implemented to check for parameter updates at a fixed interval (e.g., every 16 milliseconds, corresponding to a 60 frames per second refresh rate). This ensures stable rendering even when the user is stationary.

[0121] Based on the incremental changes in the position of the human eye and the position of the handheld interactive prop, the perspective projection matrix, rotation parameters, and translation parameters of the virtual camera are incrementally adjusted. This adjustment method can reduce the amount of calculation and improve the system's response speed.

[0122] Real-time evaluation of the complexity of virtual scenes, including the number of 3D models in the scene, the level of detail, the number of textures, etc. For example, complexity can be measured by counting the number of polygons in the scene or the total number of pixels in the texture map.

[0123] Dynamically adjust the frequency and accuracy of virtual camera parameter updates based on scene complexity. For complex scenes, reduce the update frequency appropriately but increase the accuracy of each update to ensure rendering quality; for simple scenes, increase the update frequency to achieve smoother visual effects.

[0124] Distribute rendering tasks to the asynchronous rendering queue. Set priorities based on the urgency of the tasks. For example, set rendering tasks related to the user's current perspective to a high priority to ensure that these tasks are processed first.

[0125] By using multithreading technology, different threads can handle different rendering tasks. For example, one thread can render the image from the user's current perspective, while another thread can pre-render the image for the next perspective the user might move to, improving rendering efficiency.

[0126] Enable the vertical synchronization function of the display device to ensure that the rendered image is synchronized with the refresh rate of the display device. This can avoid screen tearing and improve the stability of the display effect.

[0127] Double or triple buffering is used to manage the render buffer. When a frame is rendered, it is immediately switched from the back buffer to the foreground display buffer for display, while the next frame's rendering begins. By properly managing the buffers, a seamless transition between rendering and display is achieved.

[0128] Record user interaction data during use, such as the user's movement trajectory, the viewing position where the user stays for a long time, preferences for different rendering effects, etc. This data can be collected through the sensors of the handheld interactive props and the software interface.

[0129] In this embodiment, when the user moves, the system continuously repeats the above positioning, detection, conversion and rendering processes, dynamically updates the virtual camera parameters, thereby achieving a continuous and smooth naked-eye 3D visual effect, providing the user with a more natural and immersive interactive experience.

[0130] The method described in this embodiment utilizes a handheld device and environmental reference point calibration mode. By pre-calibrating binocular and IMU parameters at the factory, the need for on-site calibration by the user is eliminated. Simply attach any Aruco markers (a minimum of four is required), and the system automatically constructs a spatial map. Compared to multi-camera systems, the calibration process is simplified by 95%, allowing non-professionals to complete deployment in minutes. Furthermore, support for dynamic expansion and occlusion compensation of Aruco markers improves system robustness by 80%. Table 1 below compares the time consumption of this method with traditional multi-camera solutions.

[0131] Table 1 Time comparison

[0132]

[0133] As shown in Table 1, this method saves non-experts at least 91% of their time. It eliminates the need for multi-camera calibration, which relies on expensive high-precision calibration plates. This solution is factory-calibrated, eliminating the need for recalibration, saving time and money.

[0134] This method supports dynamic expansion and occlusion compensation of Aruco markers. Traditional multi-view solutions require at least 4 cameras to ensure that any object in space can be observed by at least 3 cameras at the same time, so as to calculate the spatial three-dimensional point of the object through multi-view geometry algorithms. However, due to occlusion or blind spots in the camera's viewing angle, it is usually necessary to increase the number of cameras or repeatedly adjust the camera's viewing angle. This process requires repeated external parameter calibration of the camera, which increases the time consumption of deployment. This method, however, requires tedious camera optimal viewing angle adjustment and camera external parameter calibration processes. It only requires sticking Aruco markers on the ceiling of the occluded area. While improving the robustness of the system, it effectively controls the cost. Usually, the price of Aruco markers is negligible relative to the camera.

[0135] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A 3D dynamic positioning and rendering method based on a binocular camera and an IMU, characterized in that: include: S1. Receive the 3D coordinates of the Aruco markers set in the environment obtained by the binocular camera in the handheld interactive prop, and receive the IMU data obtained in real time by the IMU sensor in the handheld interactive prop, and fuse the IMU data with the 3D coordinates of the Aruco markers to obtain the position and posture of the handheld interactive prop; S2. Receive a facial image captured by a binocular camera and transform the eye position in the facial image into a global coordinate system using the handheld interactive prop; wherein the Aruco marker located at the center of the naked-eye 3D display is used as the origin of the global coordinate system; S3. Calculate virtual camera parameters in real time based on the data converted from the human eye position to the global coordinate system and the parameters of the display, and render the virtual scene using the calculated virtual camera parameters using asynchronous rendering.

2. The 3D dynamic positioning and rendering method based on binocular camera and IMU according to claim 1 is characterized in that: The step S1 is specifically as follows: S11. Receive the Aruco markers set in the environment acquired in real time by the binocular camera in the handheld interactive prop, and calculate the 3D coordinates of the Aruco markers based on the binocular parallax; S12. Receive IMU data acquired in real time by the IMU sensor in the handheld interactive prop; S13. Fuse the IMU data with the 3D coordinates of the Aruco markers through an extended Kalman filter to predict and output the 6-DOF pose of the handheld interactive prop. S14. Bind the 6-DOF pose of the handheld interactive prop to the global coordinate system according to the 6-DOF pose.

3. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 2, characterized in that: Before step S11, the following steps are also included: S10. The handheld interactive props are pre-calibrated using a calibration board to calibrate the intrinsic parameters of the binocular cameras and the extrinsic parameters between the binocular cameras and the IMU sensor.

4. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 2, characterized in that: The step S2 is specifically as follows: S21 receives a user's facial image captured by a binocular camera in a handheld interactive prop; S22. Use a lightweight eye detection model to locate the 2D coordinates of the user's eyes in the camera coordinate system in the user's facial image; S23. Convert the 2D coordinates of the human eye into 3D coordinates in the camera coordinate system; S24. Based on the 6-DOF position of the handheld interactive prop, convert the 3D coordinates of the human eye in the camera coordinate system to the global coordinate system.

5. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 4, characterized in that: In step S24, the 3D coordinates of the human eye in the camera coordinate system are converted to the global coordinate system, which is expressed as: ； in, Represents the coordinates of the human eye in the global coordinate system; Indicates the pose of the handheld interactive prop; Indicates the 3D coordinates of the human eye in the camera coordinate system.

6. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 4, characterized in that: The step S3 is specifically as follows: S31. Obtain the parameters of the display, obtain the three-dimensional coordinates of the human eye in the global coordinate system, and calculate the perspective projection matrix of the virtual camera in real time based on the three-dimensional coordinates of the human eye in the global coordinate system and the display parameters; S32. Obtain the eye position and convert it to the eye posture in the global coordinate system, and set the rotation and translation parameters of the virtual camera according to the eye posture; S33. Render virtual scenes of different perspectives according to the perspective projection matrix, rotation parameters, and translation parameters of the virtual camera, and use an asynchronous rendering engine to synthesize the rendered virtual scenes of different perspectives to obtain a rendered multi-perspective image.

7. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 6, characterized in that: After step S33, the following steps are further included: S34. Process the rendered multi-view image through the rasterization and distortion correction module, and output the processed multi-view image to the naked-eye 3D display.

8. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 6, characterized in that: The display parameters in step S31 include size, resolution, and lens curvature.

9. The 3D dynamic positioning and rendering method based on binocular cameras and IMU according to claim 1, characterized in that: After step S3, the following steps are also included: S4. Obtain the movement data of the handheld interactive prop and dynamically update the virtual camera parameters based on the movement data.

10. 3D dynamic positioning and rendering system based on binocular camera and IMU, characterized by: The invention comprises a handheld interactive prop, a naked-eye 3D display, and multiple Aruco markers; the handheld interactive prop is provided with a binocular camera, an IMU sensor, and a processor; one of the multiple Aruco markers is fixed to the center of the naked-eye 3D display, and the remaining Aruco markers are fixed to the environment; the processor is used to process the 3D dynamic positioning and rendering method based on the binocular camera and the IMU according to any one of claims 1 to 9.

Images