Oversized scene interaction system based on multi-screen integrated deduction

Through the multi-host combination configuration and the connected screen deformation correction algorithm, the problems of insufficient graphics card performance and image fragmentation in the integrated interpretation of multi-screen are solved, high-resolution seamless splicing and smooth rendering are achieved, and interactive experience and system stability are enhanced. Reduces hardware costs.

CN119071553APending Publication Date: 2024-12-03SHANGHAI FENGYUZHU CULTURE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411195704.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing multi-screen integrated interpretation technology deals with super-large scenes and ultra-high resolutions, and the graphics card performance is insufficient, the frame rate is low, the picture is split and the interactive experience is limited.

Method used

Using a combination configuration of multiple hosts and multiple screens, a connected screen deformation correction algorithm is introduced, and seamless splicing is achieved through distributed rendering and multi-machine frame synchronization modules, combining proxy models and batch processing tools to optimize rendering performance and support multiple interactive methods. .

Benefits of technology

It breaks through the hardware bottleneck of single-player multi-graphics cards, realizes high-resolution seamless splicing, maintains the smoothness of more than 30 frames per second, enhances the interactive experience and system stability, reduces hardware costs, and is suitable for large-scale applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071553A_ABST
    Figure CN119071553A_ABST
Patent Text Reader

Abstract

The invention provides a super-large scene interaction system based on multi-screen integrated deduction, and relates to the technical field of multi-screen large-space display. The oversized scene interaction hardware system based on multi-screen integrated deduction comprises a control system, the control system is connected with a playing and rendering system, the playing and rendering system is connected with a display system, and the display system is connected with a sound system. Through the designed software module, seamless splicing among a plurality of screens can be realized, perspective correction is carried out, the picture integrity and correctness under a main visual angle are ensured, the system can process a three-dimensional digital scene in an ultra-large range, the frame rate is not lower than 30FPS on the premise of ensuring high resolution, the bottleneck in the prior art is broken through, and the system is suitable for popularization and application. The interactive terminal for synchronously rendering the main screen is provided, multiple operation modes are supported, the interactive experience of audiences is improved, the dependence on a high-performance display card is reduced through hardware configuration and software optimization, and the cost performance of the system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multi-screen large-space display, and specifically to an ultra-large scene interactive system based on multi-screen integrated interpretation. Background Art

[0002] Multi-screen integrated interpretation refers to combining multiple screens into a unified display system, and displaying digital content on these screens as a whole through technical means such as synchronous rendering and perspective correction. The purpose is to achieve high-resolution, seamless picture presentation in a super-large space, so that the audience can experience digital content in an immersive environment. The main purpose of multi-screen integrated interpretation is to meet the growing demand for super-large scenes and ultra-high-resolution displays in the display industry. This interpretation method can solve the limitations of a single screen size and resolution, and achieve a larger range and higher-precision picture display by combining multiple screens and multiple rendering hosts. At the same time, combined with the use of interactive terminals, it enhances the audience's sense of participation and immersive experience.

[0003] The existing technology usually combines multiple display devices (such as LED screens, projectors, etc.) and uses specialized hardware and software systems to achieve image synchronization and splicing. These systems usually rely on high-performance graphics processing units (GPUs) for rendering, and achieve synchronous operation between multiple devices through networks or hardware interfaces. However, when dealing with ultra-large scenes and ultra-high resolutions, the existing technology often faces problems such as insufficient graphics card performance, low frame rate, and image fragmentation. Especially when dealing with scenes with ultra-large resolutions, very high-performance graphics cards are required, which is expensive. And when processing massive three-dimensional data, problems such as low frame rate and image freeze often occur. At the same time, the image synchronization between multiple hosts is often fragmented or delayed, affecting the audience experience. Therefore, the interactive terminals in the existing technology often have limited functions and it is difficult to provide a diversified and efficient interactive experience. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a super-large scene interactive system based on multi-screen integrated interpretation, which solves the problems of strong hardware dependence, difficult synchronization and splicing, rendering jams and single interactive form in the prior art.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a hardware system for ultra-large scene interaction based on multi-screen integrated interpretation, including a control system, the control system is connected to a playback rendering system, the playback rendering system is connected to a display system, and the display system is connected to an audio system; The control system includes a console, a network communication module and a rendering node. The playback rendering system includes a multi-node rendering host, perspective deformation correction, network communication, multi-level frame synchronization, and output to a display system. The display system includes screen layout, picture stitching, and CAVE space display. The sound system includes an audio processing module and impact output.

[0006] Preferably, the control system uses touch-screen interactive operation as the core management unit of the system, communicates with other rendering nodes through the network, receives peripheral input, and passes instructions to other modules. The control system has the function of selecting the main node to ensure the orderly operation of the entire system.

[0007] Preferably, the playback rendering system is composed of a multi-node high-performance graphics rendering workstation, each node serves as a relatively independent processing unit, the system adopts a distributed design, all nodes communicate through the network and perform synchronous rendering, and each node can become a master node to process external input and synchronize other nodes.

[0008] Preferably, the display system is designed as a multi-screen layout to form a CAVE space, including three wall screens on the left, middle and right, a ceiling screen and a ground screen. Each screen is configured with one or more hosts as needed to achieve high-resolution display. By stitching the pictures together, it is ensured that the overall picture presents the correct perspective relationship when viewed from the main perspective.

[0009] Preferably, the sound system is synchronized with the display system to provide spatial sound effects and enhance the user's immersive experience. Through audio processing, the system can synchronously render the sound effects in the three-dimensional scene to be consistent with the visual content.

[0010] A software system for super-large scene interaction based on multi-screen integrated interpretation, comprising a connected screen deformation correction module, wherein the connected screen deformation correction module is connected to a multi-machine frame synchronization module; The multi-machine frame synchronization module is connected to a performance optimization module, and the performance optimization module is connected to a main screen synchronization rendering module Preferably, the connected screen deformation correction module is used to solve the problem of combining the display contents of multiple hosts into an integral picture, especially in the CAVE space. The module performs secondary perspective deformation calculation for each screen to ensure that when viewed from the main perspective, the display contents of all screens form a complete whole and the perspective relationship is correct.

[0011] Preferably, in order to prevent screen fragmentation and asynchrony when multiple hosts are running simultaneously, the system provides a multi-machine frame synchronization module, which ensures that each rendering node renders the same frame content at the same time, thereby avoiding the problem of display asynchrony.

[0012] Preferably, the system in the performance optimization module optimizes the scheduling of multiple threads of Game, Draw, and GPU to ensure smooth rendering performance in high-resolution and high-data volume scenarios. The performance optimization module also provides a function of batch generating proxy models to reduce the number of DrawCalls and alleviate the rendering burden.

[0013] Preferably, the main screen synchronous rendering module supports real-time synchronous rendering of a tablet or all-in-one terminal, which is consistent with the three-dimensional scene content of the main screen. Users can interact with the three-dimensional scene in a variety of interactive ways, and the main screen synchronously responds to the operation results.

[0014] The present invention provides a large-scale scene interactive system based on multi-screen integrated interpretation. It has the following beneficial effects: The present invention provides a large-scale scene interactive system based on multi-screen integrated interpretation. The inventive technology adopts a combination configuration of multiple hosts and multiple screens, breaking through the hardware bottleneck of traditional single-machine multi-graphics cards, so that the system can support ultra-high-resolution display effects. In order to achieve seamless splicing of multiple screens, the system of the present invention introduces an integrated screen deformation correction algorithm, which can automatically correct the splicing errors between multiple screens, eliminate the common seams and image distortion problems in traditional splicing solutions, and ensure the continuity of the picture and the consistency of the visual effect. In the face of ultra-large-scale three-dimensional scenes, the system dynamically loads and renders key areas in the scene through a self-developed streaming mechanism, avoiding the full loading of the entire scene, greatly reducing the burden on computing and graphics cards. By introducing proxy models and batch processing tools, the system can optimize the processing process of a large number of detailed models without losing graphics quality. This mechanism effectively improves the rendering efficiency, so that even in highly complex scenes, it can maintain a smoothness of more than 30 frames per second. The system is equipped with a main screen synchronous interactive terminal, which can realize multiple interactive methods including gesture recognition, touch, and remote control, which allows users to visit the process The user can freely select and switch the interactive mode, which greatly enhances the flexibility and fun of the interaction. Through refined picture splicing and seamless scene transition, combined with the richness of the interactive mode, the user can get a more immersive experience, which has significant application value for exhibition halls, virtual reality environments, and education and training scenarios. By distributing tasks to multiple hosts for parallel processing, the system effectively reduces the dependence on a single high-performance graphics card. Compared with the traditional solution, the present invention has a significant cost advantage in hardware configuration and is particularly suitable for large-scale application deployment. Its multi-host architecture enables the system to quickly switch to other nodes when facing hardware failure or single-node crash, ensuring the stable operation of the system. This design greatly improves the reliability and risk resistance of the system, making it suitable for application scenarios with extremely high stability requirements. The present invention greatly improves the performance and user experience of the ultra-large scene interactive system through innovations in hardware configuration, rendering technology, interactive mode and system architecture. Its efficient ultra-large resolution display capability, smooth real-time rendering, flexible and diverse interactive modes and economical and practical hardware configuration give it a significant competitive advantage in the exhibition industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the hardware system framework flow of the present invention; Figure 2 It is a schematic diagram of the software system framework flow of the present invention; Figure 3 It is a spatial plane schematic diagram of the present invention; Figure 4 It is a schematic diagram of the spatial elevation of the present invention; Figure 5It is a schematic diagram of the overall picture effect of multi-screen integrated synchronous rendering after software-level calculation and processing of the present invention; Figure 6 A schematic diagram of a three-dimensional scene effect in which a visitor operates a tablet or an all-in-one terminal and is rendered synchronously with a large screen in the present invention; Figure 7 A schematic diagram of the hardware system topology of the present invention; Figure 8 It is a schematic diagram of the internal structure of the hardware system of the present invention; Fig. 9 It is a schematic diagram of the basic principle of perspective calculation of the present invention; Fig.10 It is a schematic diagram of perspective projection transformation of the present invention; Fig.11 A schematic diagram of calculating the viewing frustum of each screen of the present invention; Fig.12 It is a schematic diagram of the projection of the visual cone of the present invention onto a plane; Fig.13 It is a schematic diagram for explaining parameters of the present invention; Fig.14 A schematic diagram of the perspective distortion correction calculation output effect of the present invention; Fig.15 It is a schematic diagram of the multi-host frame synchronization principle of the present invention; Fig.16 A schematic diagram of the rendering pipeline optimization logic of the present invention; Fig.17 This is a schematic diagram of the basic call principle of WebRTC of the present invention; Fig.18 This is a schematic diagram of the effect of the spatial real-shot global scene content after the integrated deformation correction of the present invention; Fig.19 It is a schematic diagram of the effect of the local fine three-dimensional scene picture content in real space shooting after the integrated deformation correction of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] like Figure 1-19 As shown, an embodiment of the present invention provides a hardware system for ultra-large scene interaction based on multi-screen integrated interpretation, including a control system, the control system is connected to a playback rendering system, the playback rendering system is connected to a display system, and the display system is connected to an audio system; The control system includes a console, a network communication module and a rendering node. The playback rendering system includes a multi-node rendering host, perspective deformation correction, network communication, multi-level frame synchronization, and output to the display system. The display system includes screen layout, picture stitching, and CAVE space display. The sound system includes an audio processing module and impact output.

[0018] The control system uses touch-screen interactive operation as the core management unit of the system, and communicates with other rendering nodes through the network, receives peripheral input, and passes instructions to other modules. The control system has the function of selecting the master node to ensure the orderly operation of the entire system. The playback rendering system consists of multi-node high-performance graphics rendering workstations. Each node is a relatively independent processing unit. The system adopts a distributed design. All nodes communicate through the network for synchronous rendering. Each node can become a master node to process external input and synchronize other nodes. The display system is designed as a multi-screen layout to form a CAVE space, including three wall screens on the left, middle and right, a ceiling screen and a ground screen. Each screen is configured with one or more hosts as needed to achieve high-resolution display. Through picture stitching, it is ensured that the overall picture presents the correct perspective relationship when viewed from the main perspective. The sound system is synchronized with the display system to provide spatial sound effects and enhance the user's immersive experience. Through audio processing, the system can synchronously render the sound effects in the three-dimensional scene, which is consistent with the visual content.

[0019] A software system for super-large scene interaction based on multi-screen integrated interpretation, including a connected screen deformation correction module, which is connected to a multi-machine frame synchronization module; The multi-machine frame synchronization module is connected to the performance optimization module, and the performance optimization module is connected to the main screen synchronization rendering module.

[0020] The connected screen deformation correction module is used to solve the problem of combining the display contents of multiple hosts into one whole picture, especially in the CAVE space. This module performs secondary perspective deformation calculation for each screen to ensure that the display contents of all screens form a complete whole and the perspective relationship is correct when viewed from the main perspective. In order to prevent the problem of screen fragmentation and asynchrony when multiple hosts are running at the same time, the system provides a multi-machine frame synchronization module, which ensures that each rendering node renders the same frame content at the same time, thereby avoiding the problem of display asynchrony. In the performance optimization module, the system optimizes the scheduling of multiple threads of Game, Draw, and GPU to ensure smooth rendering performance in high-resolution and high-data volume scenes. The performance optimization module also provides the function of batch generation of proxy models to reduce the number of DrawCalls and reduce the rendering burden. The main screen synchronization rendering module supports real-time synchronous rendering of tablets or all-in-one terminals, which is consistent with the three-dimensional scene content of the main screen. Users can interact with the three-dimensional scene in a variety of interactive ways, and the main screen responds to the operation results synchronously.

[0021] The core technology of the present invention is to achieve ultra-high resolution (super 8K) based on the ultra-large screen LED combination or projector combination in the exhibition hall to form a unified and complete picture of various types of screens, and to achieve efficient real-time rendering of massive 3D data through diversified input of interactive terminals, providing an efficient large-scene and high-resolution interactive interpretation system, specifically: The present invention forms a unified and complete ultra-high resolution combined screen by deforming and correcting the contents of multiple hosts and multiple screens, and supports multiple screen types such as L-type (a combination of two screens on the front wall and the ground), CAVE type (a combination of three walls on the left, middle and right plus five screens on the ceiling and the ground), and bowl screen (a multi-projection fusion hyperbolic bowl screen) to form a unified and complete screen, so that digital scenes and content can be rendered and performed synchronously across screens in real time to achieve ultra-large screen resolution screens. The present invention can perform synchronous and smooth real-time rendering of ultra-large-scale digital content scenes (such as supporting thousands of square kilometers of three-dimensional digital fine scenes), with a frame rate of not less than 30fps, breaking through the technical bottleneck of real-time and smooth rendering of ultra-large scenes at ultra-large display resolutions, allowing the smooth interpretation of massive city-level fine model scene data, and providing core underlying support for interactive interpretation. The present invention independently developed a main screen synchronous interactive terminal, allowing the main screen three-dimensional scene to be smoothly and synchronously rendered in real time on a low-configuration rendering terminal, and supporting a variety of operation input methods, enriching the interactive form of the exhibition hall, and greatly mobilizing the enthusiasm of visitors to participate in the interaction.

[0022] The present invention proposes a software and hardware combination solution based on multi-screen multi-host synchronization.

[0023] At the hardware level, the space is rationally configured and designed, and the large space is divided into regions. Take the CAVE space as an example: with the stands as the main perspective, the wall is split into three screens on the left, middle and right. The ceiling is one screen and the ground is another screen. It can be divided into five single-screen images to form a box-shaped surrounded CAVE space screen. In order to solve the problem of ultra-large resolution display, each screen can be configured with one or two hosts according to the resolution of the display. Through the combination of multiple hosts, the overall picture of the original space is rationally divided to enable ordinary hosts to render pictures of reasonable sizes. Through efficient topological network communication, a picture stitching system is created.

[0024] At the software level, there are four major technological breakthroughs.

[0025] 1. Solve the problem of combining the display contents of multiple hosts into one overall picture, and ensure that the perspective relationship and picture viewing angle of the real-time rendering of the 3D scene when viewed from the main stand are correct. Provide an integrated screen deformation correction software module, which performs picture perspective deformation correction calculation based on the main view. It is necessary to perform secondary processing and deformation calculation on the 3D scene content of each screen in the display space after conventional rendering. Take the CAVE space as an example: perform spatial perspective deformation calculation on the center, left, right, ceiling, and ground of the screen respectively. After the calculation, all the picture contents form a complete and unified overall picture when viewed from the main stand, and the perspective relationship of the 3D digital scene content is correct.

[0026] 2. Solve the problem of image fragmentation and asynchrony caused by the simultaneous operation of multiple hosts, and provide a multi-machine frame synchronization software module. Since each host is a unit node in the large system, the system has a high demand for synchronization during the overall operation, so a synchronization benchmark is required. One host is required as a rendering server, and all other separate nodes are client nodes. The client reception uses the server node as the rendering benchmark and performs frame image synchronization response to achieve a frame synchronization effect.

[0027] 3. To solve the bottleneck caused by the rendering of super-large scenes, a set of optimized scene rendering performance combination solutions is provided, so that the screen can run smoothly from the whole area to the local details. This combination solution has three main aspects: 1. Since the screen has been cut in advance, each host needs to render only a local picture, and the pressure on the rendering side has been significantly reduced compared to the whole, but this is far from enough, and further optimization of the scene calculation side is needed. 2. In addition to using conventional LOD technology, this solution divides the scene into cells of different levels. According to the size of the scene itself, the scene can be divided into large, medium and small levels of combined cell streaming according to the granularity. The large level is divided into mountains, terrain, etc., the middle level is buildings, vegetation, etc., and the small level is people, roles, sketches, traffic, etc. The small level objects only need to be rendered and displayed at a very close field of view, followed by the middle level, and so on. The streaming mechanism forms a selective rendering to reduce the pressure on the computing and rendering sides. 3. Due to the huge size of the scenes and the problems of multiple textures and multiple material balls caused by the use of early technology in many scenes, a set of convenient and fast batch processing software tools for generating proxy models is provided. All original data assets can be filtered and optimized, and proxy models with reasonable effects can be generated by batch processing. For example, a single multi-dimensional sub-model may have dozens or even hundreds of texture maps, which can be quickly identified and batch processed into a high-resolution texture map and a material ball to reduce the number of Draw Calls and reduce the burden and pressure on the computing side.

[0028] 4. On the interactive side, innovative upgrades have been made to provide a main screen synchronous rendering software module. Different from the traditional touch screen click input terminal, the main screen synchronous rendering module supports real-time synchronous rendering of the main screen (i.e., rendering server node) three-dimensional scene content on conventionally configured tablet terminals or all-in-one terminals, and customized interactive input combinations on the terminal, including but not limited to buttons, gestures, single-point, multi-point, dragging, sliding and other operations on three-dimensional digital scenes, and the main screen can also synchronously respond to the interactive results, providing visitors with greater choices and experience in terms of freedom.

[0029] Space CAD schematic diagram Figure 3 and Figure 4 The plan and elevation shown below can divide the entire space into five screen box-enclosed spaces of the CAVE. Visitors can experience the viewing area in the stands. The wall screen is divided into three LED left, middle and right screens. Each screen can form a 90° angle, and the top and ground are seamlessly spliced ​​with the wall screen. Then, after software-level calculation and processing, the multi-screen integrated synchronous rendering forms an overall picture, such as Figure 5 shown.

[0030] Visitors can use tablets or all-in-one terminals to operate the 3D scenes that are rendered synchronously with the large screen, and quickly select, measure, drag, and move scenes. They can also customize the path to fly through the scene, roam, and other functions. Figure 6 After testing and experiments, the fine 3D scene data of a city was integrated and processed according to the current method, and the frame rate of loading and running the fine 3D digital city model of more than 500 square kilometers was not less than 30FPS.

[0031] System gameplay process (1) Wake up the sandbox standby interface.

[0032] (2) Go to the tablet homepage and select playback or partition demonstration.

[0033] (3) Select Play Video to play the content.

[0034] (4) The progress of the playback content can be manually controlled during playback.

[0035] (5) Select the scene content demonstration and control or close the relevant demonstration according to the interface prompts.

[0036] (6) Operate the 3D scene by clicking, selecting, measuring, and other methods to interact with the large-screen content in real time.

[0037] (7) Return to the standby interface after completion.

[0038] Hardware System The hardware system is divided into four modules: control system, playback and rendering system, display system, and audio system. The control system can support touch-based interactive operations, the playback and rendering system is a multi-node rendering host, the display system is a large screen with sliced ​​images, and the audio system adds rich sound effects to the spatial experience. The systems are organized in an orderly manner, and the corresponding hardware parameters are reasonably configured according to the actual space requirements, such as Figure 7 The hardware system topology diagram and Figure 8 The internal structure of the hardware system is shown.

[0039] Deformation correction calculation The viewing frustum that needs to be rendered on each screen is calculated based on the viewing point, thereby obtaining the deformation parameters that need to be calculated for each primitive in each screen.

[0040] The coordinates entered into the program are all 3D, but what is finally displayed on the screen can only be 2D. So a process is needed to map the 3D coordinates to our 2D viewing plane. Only in this way can we view it through the monitor. Set the near plane, far plane, and viewing angle to get a frustum. And we often use the near plane as the viewing plane, such as Fig. 9. Cut off the vertices outside the view cone and keep only those inside. If the view cone is cut directly, the tilt of the plane will also lead to too much calculation, so it is mapped to the unit cube (CVV), at this time, all are vertical planes, and in the mapping process, the relationship between the vertex and the view cone remains unchanged, and those outside the view cone are still outside the view cone. When cutting, only one coordinate needs to be determined, which greatly reduces the amount of calculation, such as Fig.10 , when mapped to CVV, the proportions between vertices change, and objects often deform, so when mapped to the screen coordinate system, the proportions must be restored. In this example, the total viewing frustum needs to be cut into five small viewing frustums for independent calculations, such as Fig.11 .

[0041] The perspective projection process is divided into 4 parts.

[0042] 1. Project the vertices in the frustum to the viewing plane 2. Map the view cone to the CVV (regular observation volume) for clipping 3. Perform perspective division.

[0043] 4. Perform viewport transformation, restore its proportions, and finally map it to the screen coordinate system.

[0044] like Fig.12 As shown: The parameters are np: near section, fp: far section, p': vertex after projection, p: object vertex, N: distance from near section to observation point, F: eye: observation point.

[0045] According to the similar triangle properties of p and p', we can get:

[0046] Similarly, on the z-axis and y-axis we can get:

[0047] So we get the projected coordinates:

[0048] At this time, the z value of all points is -N. If this is used as the final coordinate, subsequent operations such as depth testing cannot be performed, so the z value of the original vertex in space needs to be retained. The final coordinate can be determined as:

[0049] We transform the form of z of p' and finally get:

[0050] Since x'=-Nx / z and y'=-Ny / z are both on the viewing plane, we only need to transform the viewing plane to the unit section of CVV. Let the coordinates finally transformed into CVV be x'', y'', and the height of the projection plane be H. Since the length, width and height of CVV ​​are all 2, we can get from geometric transformation: y' / y''= H / 2, and we can get from matrix and homogeneous transformation: y''=-(y / z)*cot(fovy / 2) x''=-(x / z)*cot(fovy / 2)*(1 / aspect) When finding the matrix, perform the homogeneous coordinate inverse processing and multiply by -z. Therefore, the matrix M can be obtained as follows:

[0051] The parameter description is as follows Fig.13 As shown. Therefore, each viewing frustum can finally obtain the following rendering result single frame, as shown Fig.14 shown.

[0052] Principle of multi-machine frame synchronization calculation: The system adopts a distributed design concept, that is, each customized high-performance graphics rendering workstation is a relatively independent processing node. There is no absolute master-slave distinction between nodes. Each node can obtain control and become a master node. At each moment, there is only one node as the master node, which accepts input from peripherals and transmits it to other slave nodes through the network. Fig.15 shown.

[0053] Performance Optimization In one frame, the Game, Draw, and GPU threads run in parallel. The total time taken for one frame depends on the thread with the most overhead, that is, the longest time taken. In the scene, it is necessary to locate the specific affected links and threads and make reasonable judgments and adjustments. Large scenes often cause lag due to the large number of static resources. The most effective adjustment method is to reduce the number of DrallCalls to improve performance. Reducing DrallCalls through a series of engine-level combinations is the shortest path to efficient scene rendering and operation. Fig.16 shown.

[0054] Main screen synchronous rendering module The system accesses the WebRTC module to render the large screen content to the tablet in real time. The tablet converts the operation input to the large screen through screen pixel conversion, realizing real-time interaction with the large screen in 0-device detection mode. This solution is introduced in this example to allow the large screen and the tablet to communicate in real time and give feedback to each other. Fig.17 shown.

[0055] This invention has been implemented in many projects, including the digital sandbox interactive system of Yimou City Planning Museum as an example: After the implementation of this case, the display screen is a CAVE space, where the three LEDs on the wall are of P2 specifications, the on-site space size is about 13m*12m*7.8m, and the display resolutions are: left wall resolution 5440*3440, middle wall resolution 6560*3440, right wall resolution 5440*3440, ground resolution 6560*5440, equipped with four hosts, and the graphics cards are all NVIDIA RTXA5000. A detailed 3D city model of a certain city area is achieved with an area of ​​more than 500 square kilometers, and the rendering frame rate is not less than 30fps, such as Fig.18 and Fig.19 shown.

[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A hardware system for large-scale scene interaction based on multi-screen integrated interpretation, including a control system, Features: The control system is connected to a playback and rendering system, the playback and rendering system is connected to a display system, and the display system is connected to an audio system; The control system includes a console, a network communication module and a rendering node. The playback rendering system includes a multi-node rendering host, perspective deformation correction, network communication, multi-level frame synchronization, and output to a display system. The display system includes screen layout, picture stitching, and CAVE space display. The sound system includes an audio processing module and impact output.

2. According to claim 1, a hardware system for large-scale scene interaction based on multi-screen integrated interpretation, Features: The control system uses touch-screen interactive operation as the core management unit of the system, communicates with other rendering nodes through the network, receives peripheral input, and passes instructions to other modules. The control system has the function of selecting the main node to ensure the orderly operation of the entire system.

3. According to claim 1, a hardware system for large-scale scene interaction based on multi-screen integrated interpretation, Features: The playback rendering system is composed of a multi-node high-performance graphics rendering workstation. Each node serves as a relatively independent processing unit. The system adopts a distributed design. All nodes communicate through the network and perform synchronous rendering. Each node can become a master node to process external input and synchronize other nodes.

4. According to claim 1, a hardware system for large-scale scene interaction based on multi-screen integrated interpretation, Features: The display system is designed as a multi-screen layout to form a CAVE space, including three wall screens on the left, middle and right, a ceiling screen and a ground screen. Each screen is configured with one or more hosts as needed to achieve high-resolution display. By splicing the pictures, it is ensured that the overall picture presents the correct perspective relationship when viewed from the main perspective.

5. According to claim 1, a hardware system for large-scale scene interaction based on multi-screen integrated interpretation, Features: The sound system is synchronized with the display system to provide spatial sound effects and enhance the user's immersive experience. Through audio processing, the system can synchronously render the sound effects in the three-dimensional scene to be consistent with the visual content.

6. A software system for large-scale scene interaction based on multi-screen integrated interpretation according to any one of claims 1 to 5, comprising a connected screen deformation correction module, Features: The connected screen deformation correction module is connected to a multi-machine frame synchronization module; The multi-machine frame synchronization module is connected to a performance optimization module, and the performance optimization module is connected to a main screen synchronization rendering module.

7. A software system for large-scale scene interaction based on multi-screen integrated interpretation according to claim 6, Features: The connected screen deformation correction module is used to solve the problem of combining the display contents of multiple hosts into an integrated picture, especially in the CAVE space. The module performs secondary perspective deformation calculation for each screen to ensure that when viewed from the main perspective, the display contents of all screens form a complete whole and the perspective relationship is correct.

8. According to claim 6, a software system for large-scale scene interaction based on multi-screen integrated interpretation, Features: In order to prevent screen fragmentation and asynchrony when multiple hosts are running simultaneously, the system provides a multi-machine frame synchronization module, which ensures that each rendering node renders the same frame content at the same time, thereby avoiding the problem of display asynchrony.

9. According to claim 6, a software system for large-scale scene interaction based on multi-screen integrated interpretation, Features: In the performance optimization module, the system optimizes the scheduling of multiple threads of Game, Draw, and GPU to ensure smooth rendering performance in high-resolution and high-data-volume scenarios. The performance optimization module also provides the function of batch generating proxy models to reduce the number of DrawCalls and alleviate the rendering burden.

10. A software system for large-scale scene interaction based on multi-screen integrated interpretation according to claim 6, Features: The main screen synchronous rendering module supports real-time synchronous rendering of a tablet or all-in-one terminal, which is consistent with the three-dimensional scene content of the main screen. Users can interact with the three-dimensional scene in a variety of interactive ways, and the main screen synchronously responds to the operation results.

Citation Information

Cited By

  • Distributed interactive control method and system for multi-machine projection fusion equipment

    CN120856723A