Virtual scene display system and method based on rotating mechanism

Abstract

The invention provides a virtual scene display system and method based on a rotating mechanism. Relates to the technical field of virtual reality and augmented reality display, and comprises a rotation driving assembly which comprises a bearing platform and an absolute value angle encoder, and the bearing platform is used for bearing audiences and rotating around a vertical axis; the attitude acquisition unit is used for outputting rotating attitude parameters in real time through the angle encoder; the audience perception-cooperative control unit comprises a plurality of depth cameras, and the depth cameras are used for collecting audience number and orientation data. The virtual scene display system and method based on the rotating mechanism have the advantages of high precision, low delay, easy expansion, multi-person immersion and friendly operation and maintenance, and are obviously superior to the existing fixed screen surface or one-way rotating display technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual reality and augmented reality display, in particular to a virtual scene display system and method based on a rotating mechanism. BACKGROUND

[0002] The existing virtual scene display devices can be divided into several categories, such as head-mounted displays, dome / cylinder screen projection systems, 360° panoramic LED screens, and "rotating platform + single screen" devices. Head-mounted displays provide immersive stereoscopic images for a single user through posture sensors, but they are not suitable for multiple users. Dome / cylinder screen projection systems can serve a group of viewers by projecting a super-wide field of view image on a fixed screen, but they are expensive and have strict site requirements. Panoramic LED or flexible OLED cylindrical screens surround the viewers with a physical screen, but the viewing point is fixed and the content production cost is high. Some interactive exhibits slowly rotate the viewers or physical objects, and render multi-angle visual effects on a single large screen, but this solution lacks precise posture feedback and real-time image linkage.

[0003] The above solutions have multiple deficiencies in application. Head-mounted displays can cause dizziness and are difficult to maintain in public places. Large screens and panoramic screens support multiple viewers, but the fixed viewing point causes perspective distortion and lack of displacement. The rotating platform solution can cause visual misalignment due to the asynchronous rendering and mechanical movement, and fast rotation can cause safety and wear problems. In addition, large projection / LED systems are expensive to build and calibrate, and mechanical platforms are generally bulky and have low modularity, making it difficult to meet the urgent needs of low-cost, easy-to-deploy, and comfortable multi-user interaction in travel and exhibition scenarios. SUMMARY

[0004] To overcome the deficiencies of the prior art, the present application provides a virtual scene display system and method based on a rotating mechanism, which solves the problem of how to realize multi-user synchronous sharing of immersive dome display through coupling of high-precision posture acquisition of the rotating platform and adaptive sector rendering of the audience distribution.

[0005] To achieve the above purpose, the present application realizes the following technical solutions: a virtual scene display system and method based on a rotating mechanism, comprising: A rotating drive assembly, including a bearing platform and an absolute value angle encoder, the bearing platform being used to bear the audience and rotate around a vertical axis; A posture acquisition unit for real-time output of the rotating posture parameters through the angle encoder; The audience perception-collaborative control unit includes several depth cameras for collecting audience quantity and position data, and divides the annular display unit into several sectors according to the audience quantity and position data, and generates sector mapping parameters and display parameters. The real-time rendering unit generates video signals corresponding to each sector based on the rotation attitude parameters and the sector mapping parameters. The annular display unit receives the video signals and independently drives each sector according to the display parameters, synchronously presents a 360° virtual scene, and enables multiple audiences to simultaneously obtain an immersive viewing experience.

[0006] Preferably, the absolute value angle encoder has an angle resolution of not greater than 0.05° and an encoding frequency of not less than 1000 Hz, to ensure high-precision attitude feedback.

[0007] Preferably, the attitude acquisition unit uses extended Kalman filtering to eliminate mechanical jitter and completes attitude calculation within 5 ms.

[0008] Preferably, the audience perception-collaborative control unit uses a density clustering algorithm to group audiences based on the point cloud obtained by the depth cameras, and dynamically determines the number of sectors of the annular display unit based on the average position of the audience groups, wherein the number of sectors N is in the range of 2≤N≤12.

[0009] Preferably, the real-time rendering unit calculates a perspective projection matrix and generates a video signal according to the following model formula: wherein, is a projection matrix of the i-th sector, is a camera intrinsic parameter matrix, is a rotation matrix around a vertical axis, is a current platform rotation angle, is a center azimuth angle of the i-th sector.

[0010] Preferably, the annular display unit adaptively adjusts the luminance according to the following model formula:

[0011] wherein, is a reference luminance, is a luminance adjustment coefficient, is the number of audiences in the i-th sector, is the current total number of sectors.

[0012] ​​Preferably, the virtual scene display system based on the rotating mechanism further comprises a network communication unit, which realizes real-time transmission of video signals and display parameters between the real-time rendering unit and the annular display unit through 5G.

[0013] A virtual scene display method based on a rotating mechanism, comprising: S1. Obtain the rotation angle of the platform around the vertical axis through the absolute value angle encoder installed on the bearing platform, and output the denoised attitude parameter by using the extended Kalman filter.

[0014] S2. Collect the audience point cloud data through the depth camera, perform density clustering on the point cloud data, and obtain the audience direction and the number of audiences.

[0015] S3. According to the audience direction, divide the annular display unit into N sectors, and generate the brightness adjustment parameter according to the number of audiences in each sector, so that the sector with more audiences obtains higher brightness, so as to improve the viewing balance.

[0016] S4. Call the three-dimensional graphics engine, perform corresponding perspective transformation for each sector according to the attitude parameter, and generate a video signal, so that the end-to-end time delay of the system is not greater than 20 ms.

[0017] S5. Synchronize the video signal and the brightness parameter to the annular display unit, output the virtual scene picture, so that multiple audiences can obtain 360° immersive viewing experience without wearing head-mounted devices.

[0018] The application provides a virtual scene display system and method based on a rotating mechanism. The virtual scene display system and method based on the rotating mechanism seamlessly couples the mechanical platform movement and the virtual picture through the four-level closed loop of "high-precision rotation attitude acquisition + audience distribution adaptive sector division + real-time multi-view rendering + partition brightness balance". On the one hand, the combination of the absolute value angle encoder and the extended Kalman filter controls the rotation error within 0.05°, and the end-to-end time delay is less than or equal to 20 ms, which basically eliminates the perspective distortion and dizziness; on the other hand, the depth camera array can real-time perceive the audience direction, and dynamically match the display content and brightness with 2-12 variable sectors, which not only avoids the poor experience of "clear center and dark edge" in the large screen scheme, but also enables multiple audiences to obtain 360° immersive field of view without wearing head-mounted devices, so as to realize the core goal of "multi-person sharing + comfortable viewing".

[0019] The method scheme corresponding to the above hardware capability is as follows: first, real-time acquisition and denoising of the posture, then automatic determination of the sector number and corresponding brightness according to the audience clustering result, then independent perspective transformation output of the synchronous video according to each sector in the rendering stage, and finally partition driving of the annular screen. The process simplifies deployment and debugging, the brightness adaptive mechanism guarantees the viewing balance under different crowd densities, and the 5G low latency transmission takes into account both large bandwidth and high reliability. Overall, the technical scheme has the advantages of high precision, low delay, easy expansion, multi-person immersion and friendly operation and maintenance, and is significantly superior to the existing fixed screen or one-way rotating display technology. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of the implementation of the application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0022] As shown in Figure 1 , the embodiment of the application provides a virtual scene display system and method based on a rotating mechanism, which comprises a rotating driving assembly, including a bearing platform and an absolute value angle encoder. The bearing platform is used to bear the audience and rotate around a vertical shaft. The angle resolution of the absolute value angle encoder is not greater than 0.05°, and the encoding frequency is not less than 1000Hz, so as to ensure high-precision posture feedback.

[0023] The specific implementation is as follows: An aluminum alloy honeycomb bearing platform with a diameter of 2000mm is adopted. The platform has a self-weight of 85kg and a designed static load capacity of 600kg, and can simultaneously accommodate 6 to 8 adults. The platform is rigidly connected with a permanent magnet synchronous servo motor and a planetary reducer below through an integrated rotary table seat. The motor has a rated power of 2.2kW and a rated output torque of 35N·m, and the reduction ratio is 30:1. The platform can realize uniform speed or variable speed rotation at 0 to 30rpm, and the peak angular acceleration is 60° / s².

[0024] A Heidenhain absolute value grating encoder is selected for posture detection. The single-turn resolution is 21 bits, the corresponding angle resolution is 0.00026°, and the refresh frequency is 1200Hz. The encoder adopts taper sleeve positioning, and the coaxial error is controlled within 0.02mm, and the radial runout is not more than 10µm. The signal is transmitted through the built-in optical fiber slip ring, realizing 360° continuous rotation without cable twisting.

[0025] The main controller reads the encoder angle data at a frequency of 1000 Hz and runs the extended Kalman filter, and the end-to-end pose calculation delay is 3.8 ms on average. The 15 rpm uniform speed test shows that the static angle mean square error is 0.018°, the dynamic error is 0.032°, and the maximum instantaneous error is 0.041°. The system runs continuously for 24 hours without data frame loss.

[0026] Fault protection logic: automatically slow down to 5 rpm and alarm when the dynamic error is greater than 0.05° for more than 10 seconds; brake for more than 100 ms when the communication heartbeat is lost, and the braking distance is 12.5 mm.

[0027] In the demonstration environment of simultaneously driving 4 paths of 4K 60fps graphics rendering, the rotation driving assembly always keeps the picture synchronized with the actual viewing angle, without tearing or motion sickness, verifying the reliability of high-precision pose feedback and long-time stable operation The pose acquisition unit is configured to output the rotation pose parameter in real time through the angle encoder. The pose acquisition unit adopts the extended Kalman filter to eliminate mechanical jitter, and completes the pose calculation within 5 ms.

[0028] The audience perception-collaborative control unit includes a plurality of depth cameras, the depth cameras are configured to collect audience quantity and position data, and the audience perception-collaborative control unit is configured to divide the annular display unit into a plurality of sectors according to the audience quantity and position data, and generate sector mapping parameters and display parameters.

[0029] The audience perception-collaborative control unit uses a density clustering algorithm to group the point cloud obtained by the depth camera, and dynamically determines the number of sectors of the annular display unit based on the average position of the audience grouping, and the number of sectors N ranges from 2 to 12.

[0030] The specific implementation is as follows: Twelve depth cameras are installed at equal intervals on the outer edge of the annular display unit, with a horizontal field of view of 87° and a vertical field of view of 58°. Each camera is installed at a height of 2.3 m and inclined inward at an angle of 15°, covering the audience standing area to a radius of 6 m from the center platform. The camera supplies power and transmits data through gigabit Ethernet, with a depth frame resolution of 1280×720 and a frame rate of 60 Hz. All raw depth streams are converged to an edge server, and the end-to-end processing delay of the server is kept within 35 ms.

[0031] The server first converts the 12-channel depth map into point cloud data, and performs downsampling with a 0.1 m voxel size to reduce the computational load. Then, a density-based clustering algorithm is called, with a Euclidean distance threshold of 0.4 m and a minimum cluster point count of 5 pixels, to identify the three-dimensional point cloud clusters of standing audience members in real time. The algorithm outputs the centroid coordinates of each cluster and converts them into an azimuth angle φ and a radius r relative to the central stage, while also counting the number of depth points n included in the cluster as an estimate of the number of audience members. The entire clustering-conversion process is run every 100 ms.

[0032] The system determines the total number of audience clusters M recognized at the current time, and first merges clusters with a neighboring azimuth difference of less than 12° according to the "nearest merge" principle to obtain the final number of audience clusters N, ensuring that N is between 2 and 12. Then, N sectors are assigned to the annular display unit in a clockwise direction, with the sector center line coinciding with the azimuth of each audience cluster, and the single-sector angle is calculated by dividing 360° by N. If the number of audience members corresponding to a sector is greater than the global average audience number, the system increases the visibility of that sector through brightness enhancement and gamma fine-tuning, with the increase linearly mapped to a range of 20%. All sector mapping parameters and brightness parameters are packaged and sent to the real-time rendering unit at each update period.

[0033] For example, there are 30 audience members present at the same time, and the clustering algorithm identifies 8 audience clusters. The system dynamically divides the 8 sectors, with each sector angle being 45°. The 3rd sector has 6 audience members, which is higher than the average of 3.75, and the brightness is increased by 15%. The entire calculation and broadcast link delay is maintained within 50 ms, ensuring that the sector division and brightness adjustment are almost synchronized with the actual distribution of the audience, thereby achieving balanced optimization of the multi-audience immersive viewing experience.

[0034] The real-time rendering unit generates video signals corresponding to each sector based on the rotation attitude parameters and sector mapping parameters. The real-time rendering unit calculates the perspective projection matrix and generates the video signal according to the following model formula: where, is the projection matrix of the i-th sector, is the camera intrinsic matrix, is the rotation matrix around the vertical axis, is the current platform rotation angle, is the azimuth angle of the center of the i-th sector.

[0035] The annular display unit receives the video signals and independently drives each sector according to the display parameters, synchronously presenting a 360° virtual scene to enable multiple audience members to simultaneously experience an immersive viewing experience. The annular display unit adaptively adjusts the brightness according to the following model formula:

[0036] wherein, is the reference brightness, is the brightness adjustment coefficient, is the number of viewers in the i-th sector, is the total number of sectors. Embodiment one

[0037] The specific implementation is as follows: Application scenario: regular exhibition mode of digital museum.

[0038] A set of 8-sector ring screen system is deployed in the second floor round hall of Shenzhen Futian Digital Museum. The bearing platform has a diameter of 2000mm and a thickness of 120mm, is made of aluminum honeycomb plate plus carbon fiber shell, has a self weight of 85kg, and can stand 8 viewers at the same time. The rotation drive is powered by a 2.2kW permanent magnet synchronous servo motor cooperating with a 30:1 planetary reducer, has a maximum speed of 10rpm, an angular acceleration of 60° / s², and a positioning error controlled within 0.04°. The attitude data is output at a frequency of 1200Hz, and after Kalman filter processing, the end-to-end delay is 3.8ms.

[0039] The real-time rendering unit is composed of 2 dual-channel Xeon servers, each of which is installed with 4 RTX4090 graphics cards, and the total graphics computing power is about 950TFLOPS. The servers are interconnected with the attitude master control and display nodes through dual 10G Ethernet. The rendering pipeline is based on UnrealEngine5: the latest attitude angle and the center direction of 8 sectors are read every frame, the camera matrix is dynamically adjusted, and 8 4K60fps pictures are output; the system end-to-end delay is stably controlled within 18ms.

[0040] The ring screen adopts 1.5mm pitch curved LED module, with a total length of 30m and a height of 3.6m; the resolution of each sector is 3840x2160. The ring screen eaves are installed with 12 Intel RealSense D435 depth cameras at equal intervals, which collect point clouds at 60Hz. The edge server performs real-time clustering of viewer clusters based on the DBSCAN algorithm, with an average calculation time of 12ms. The system re-evaluates the viewer distribution every 200ms, and when the number of viewers in a sector is more than 30% higher than the average of the whole field, the brightness of the sector is increased by 15%, and the brightness of the sparse sector is reduced by 5%, with the overall power consumption controlled within 700W.

[0041] After 30 days of open operation, the platform has rotated for 700 hours, and the daily average reception is 3500 people. The satisfaction survey shows that 97% of the viewers think that the immersion is strong and there is no obvious dizziness feedback. Except for daily screen wiping and weekly encoder bolt tightening, there is no additional maintenance cost.

[0042] A virtual scene display method based on a rotating mechanism, comprising: S1. Obtain the rotation angle of the platform around the vertical axis through the absolute value angle encoder installed on the bearing platform, and output the denoised attitude parameter using the extended Kalman filter.

[0043] S2. Collect the audience point cloud data through the depth camera, perform density clustering on the point cloud data, and obtain the audience orientation and audience number.

[0044] S3. According to the audience orientation, divide the annular display unit into N sectors, and generate the brightness adjustment parameter according to the audience number of each sector, so that the sector with more audience number obtains higher brightness, to improve the viewing balance.

[0045] S4. Call the three-dimensional graphics engine, perform the corresponding perspective transformation for each sector according to the attitude parameter, and generate the video signal, to ensure that the end-to-end delay of the system is not more than 20ms.

[0046] S5. Synchronize the video signal and brightness parameter to the annular display unit, output the virtual scene picture, so that multiple audiences can obtain 360° immersive viewing experience without wearing head-mounted devices. Embodiment Two

[0047] The difference between this embodiment and embodiment one is that the application scenario of this embodiment is the high-load impact mode of a theme park.

[0048] A certain immersive park in Shanghai Pudong creates a 600m² science fiction shuttle project on the first floor underground, with 12-sector ultra-high-definition ring screens as the core. The bearing platform is 3000mm in diameter, with a titanium alloy frame and tempered glass steps, a platform mass of 210kg, a design static load of 1000kg, and can accommodate 15 audiences. The drive uses a 5kW water-cooled direct-drive motor, with a zero-to-25rpm acceleration time of 12s. The absolute value grating encoder refreshes at a frequency of 1400Hz, with a resolution of 0.00026°, and forms a redundant attitude chain with a dual-inertial measurement unit. If the difference between the two data exceeds 0.05°, the speed will be reduced.

[0049] The rendering cluster is composed of 4 servers equipped with NVIDIA A100, with a total of 160GB of video memory, which can output 12 8K 90fps pictures in real time. The attitude and sector mapping parameters are broadcasted through a 25GbE switch, and the end-to-end delay from the rendering node to the display end is 15ms. The engine uses distributed depth buffer technology to eliminate sector joint flicker.

[0050] The ring screen uses 1.2mm pitch Micro-LED modules with over 300 million pixels. The system uses adaptive brightness: when the single sector crowd density exceeds 4 people / m², the brightness is increased to 120% of the peak, and when the density is less than 1 person / m², it is reduced to 80%, and the overall power consumption is controlled within 20kW. To match the dynamic floor, the vibration sensor provides real-time acceleration feedback to the rendering engine, dynamically adjusting the camera's near cropping plane to avoid picture jitter.

[0051] During the 72-hour continuous pressure test, the maximum synchronization error between platform rotation and picture was 0.018°, and the brightness difference between the 12 sectors was kept within 3%. During the peak operation period, 1450 people were received per hour, and the audience could still enjoy a continuous visual effect. Night maintenance only requires replacing the UPS module and cleaning the camera lens, and the total time is 30 minutes to resume operation. Example Three

[0052] Unlike the previous examples, the application scenario of this example is an immersive display mode for a new car showroom.

[0053] A 6-sector ring screen system is deployed in a multimedia showroom at the headquarters of a certain automobile brand in Beijing, which is used for new car model release and media experience. The site is a circular space with a diameter of 14m, and the central rotating platform has a diameter of 2500mm. The platform is made of carbon fiber honeycomb composite material, weighs 120kg, and can accommodate 10 people including enterprise executives, design team and host. The platform is driven by a 3kW water-cooled direct-drive servo motor, with a reduction ratio of 25:1, a maximum speed of 12rpm, and an angular acceleration of 45° / s². The attitude data is output at a frequency of 1200Hz, and the extended Kalman filter backend has an end-to-end delay of 4ms.

[0054] The ring screen uses 1.6mm pitch curved LED modules, with a total length of 22m and a height of 3m, and consists of 6 sectors, each with a resolution of 3840x1920, which can fully present vehicle close-up, internal perspective and dynamic road conditions. Eight Intel RealSense L515 laser depth cameras are installed on the top to collect audience point clouds at 60Hz. The edge server uses density clustering to identify audience clusters and estimate the number of people in real time; when the number of media people in a single sector exceeds 20% of the average of the entire venue, the system increases the brightness of that sector by 10% and reduces the brightness of other sectors by 5% to guide the focus of photography and videography.

[0055] The real-time rendering cluster consists of 3 dual Xeon servers, each equipped with 2 RTXA6000 graphics cards, which can output 6 channels of 5K 75fps pictures simultaneously. The platform attitude angle and sector mapping parameters are broadcast through a 10GbE switch, and the rendering to the display end has an end-to-end delay of 22ms. To demonstrate the color accuracy of the vehicle, the system performs color lookup table correction at the end of each frame rendering, keeping the LED screen ΔE95 within 1.5.

[0056] To match the vehicle physical entry, the platform can accelerate from 0 rpm to the display speed of 8 rpm within 60 s and keep stable; the on-site laser scanning instrument ensures that the minimum safety distance between the vehicle contour and the ring screen is 400 mm. During the peak of the press conference, 300 media and guests are received in a single session, and the system runs continuously for 5 hours without any frame drop or synchronization exception. The visitor questionnaire shows that 92% of the media representatives believe that the ring screen display effectively enhances the immersion and visual impact of the vehicle highlights.

[0057] Daily maintenance only needs to wipe the LED screen every 2 days, check the camera lens cleaning and calibrate the platform encoder zero position once, the average monthly downtime maintenance time is less than 40 minutes, which meets the operation requirements of high frequency product release of enterprises.

[0058] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A virtual scene presentation system based on a rotating mechanism, characterized in that, include: A rotation drive assembly includes a support platform and an absolute angle encoder, the support platform being used to support spectators and rotate about a vertical axis; An attitude acquisition unit is used to output rotational attitude parameters in real time through the angle encoder; The audience perception-cooperative control unit includes several depth cameras, which are used to collect audience number and orientation data. The audience perception-cooperative control unit divides the ring display unit into several sectors based on the audience number and orientation data, and generates sector mapping parameters and display parameters. The real-time rendering unit generates video signals corresponding to each sector based on the rotational attitude parameters and the sector mapping parameters. The ring-shaped display unit is used to receive the video signal and drive each sector independently according to the display parameters to synchronously present a 360° virtual scene.

2. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The absolute angle encoder has an angular resolution of no more than 0.05° and an encoding frequency of no less than 1000Hz.

3. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The attitude acquisition unit uses an extended Kalman filter to eliminate mechanical jitter and completes attitude calculation within 5ms.

4. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The audience perception-cooperative control unit uses a density clustering algorithm to group the point cloud acquired by the depth camera, and dynamically determines the number of sectors of the ring display unit based on the average orientation of the audience groups. The number of sectors N is in the range of 2≤N≤12.

5. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The real-time rendering unit calculates the perspective projection matrix and generates a video signal according to the following model formula: in, For sector i Projection matrix, For the camera intrinsic parameter matrix, Let be the rotation matrix about the vertical axis. The current platform rotation angle, For the first Azimuth of the sector center.

6. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The annular display unit adaptively adjusts the luminance by the following model formula: : ； wherein, is the reference luminance, is the luminance adjustment coefficient, is the number of viewers in the i-th sector, is the total number of sectors.

7. The virtual scene presentation system based on a rotating mechanism according to claim 1, characterized in that: The virtual scene display system based on the rotating mechanism also includes a network communication unit, which uses 5G to realize the real-time transmission of video signals and display parameters between the real-time rendering unit and the ring display unit.

8. A virtual scene display method based on a rotating mechanism, characterized in that, include: S1. Obtain the rotation angle of the platform around the vertical axis by an absolute angle encoder installed on the carrier platform, and output the denoised attitude parameters using an extended Kalman filter; S2. Collect audience point cloud data using a depth camera, and perform density clustering on the point cloud data to obtain the audience location and number of audience members; S3. Based on the audience's orientation, divide the ring-shaped display unit into N sectors, and generate brightness adjustment parameters according to the number of audience members in each sector; S4. Call the 3D graphics engine to perform corresponding perspective transformations for each sector based on the posture parameters and generate video signals; S5. The video signal and the brightness parameters are sent synchronously to the circular display unit to output the virtual scene image.

Images