A virtual-real fusion method and device for a performance exhibition based on the metaverse

By integrating and optimizing the 3D models and videos of the Metacos performance exhibition venue, and combining with the Playcanvas engine, the Metacos performance exhibition venue's effects improvement and interactivity problems are solved, efficient artistic effects and participation are achieved, and cross-space is created. Interactive experience with time.

CN116563498BActive Publication Date: 2024-11-15GUANGDONG WANGYAN CULTURAL TOURISM DIGITAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310203673.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-03
Publication Date
2024-11-15
Estimated Expiration
2043-03-03

AI Technical Summary

Technical Problem

In the context of metacosmic technology, how to improve the artistic effect and simulation effect of the performance exhibition venue, achieve its sharing, dissemination, participation and interaction, and enhance the immersive experience of the metacosmic.

Method used

By integrating 3D model scenes and performance videos virtually and real, using the WEB-side 3D editing engine for optimization and adjustment, adding digital human resources and video playback link interfaces, realizing web video playback links, and performing online dynamic updates and streaming loading, combining them with Playcanvas engine technology.

Benefits of technology

Improve the artistic effect and simulation effect of the performance exhibition venue, realize sharing, dissemination, participation and interaction, enhance the technology of the meta-universe performance exhibition venue, create an innovative, interactive and exploratory performance experience, and realize people Real-time interaction with people, multi-directional interaction across space and time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116563498B_ABST
    Figure CN116563498B_ABST
Patent Text Reader

Abstract

The virtual-real fusion method and device of the performance exhibition venue based on the metaverse of the present invention include building and producing a 3D model that matches the actual performance scene data; performing matting and post-processing on the pre-recorded performance video; integrating the built and produced 3D model scene and the processed performance video file in a 3D editing engine that supports the WEB end; optimizing and adjusting the material resources in the 3D editing engine on the WEB end; and outputting the web page video playback link for the optimized material resources. The rendered animation special effects and camera recording are used to combine the live performance video with the 3D scene, and the artistic simulation effect of the performance exhibition venue is improved through animation software processing and 3D material ball configuration; new technical paths are added to the artistic special effects processing of the performance and the interactive mode of the audience participating in the performance, further strengthening the technology of the metaverse performance exhibition venue, and improving the art, simulation effect, sharing, dissemination, participation, and interactivity of the metaverse performance exhibition venue.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of metaverse technology, and more specifically, to a method and device for integrating virtuality and reality in a performance exhibition venue based on the metaverse. Background Art

[0002] With the vigorous development of the concept of the metaverse, the standardization of related technologies from concept description to the implementation of specific technical paths has become the only way to establish the foundation of metaverse technology. The metaverse performance venue uses technology to present stage performances to the audience in real-time broadcast or data storage and reproduction, and realizes interactive functions at different dimensional levels.

[0003] However, in the context of metaverse technology, how to improve the artistic and simulation effects of performance exhibitions and realize their shareability, dissemination, participation and interactivity has become an urgent problem to be solved in order to enhance the immersive experience of the metaverse. Summary of the invention

[0004] In order to overcome the defects of the prior art, the present invention provides a method and device for integrating virtuality and reality based on a metaverse performance exhibition venue, which solves the problem of how to improve the artistic effect and simulation effect of the performance exhibition venue and realize its sharing, dissemination, participation and interactivity.

[0005] To achieve this object, the present invention adopts the following technical solutions:

[0006] The present invention provides a virtual-real fusion method for a performance exhibition venue based on a metaverse, comprising the following steps:

[0007] Build and produce a 3D model that matches the actual performance scene data to form a built and produced 3D model scene;

[0008] Performing matting and post-processing on the pre-recorded performance video to form a processed performance video file; wherein the post-processing at least includes lighting processing, color grading processing, and video portal special effects processing;

[0009] The constructed 3D model scene and the processed performance video file are integrated into a 3D editing engine supporting the WEB end, forming a material resource of the 3D editing engine supporting the WEB end;

[0010] Optimizing and adjusting the material resources in the 3D editing engine on the WEB end to form optimized and adjusted material resources; wherein the optimization and adjustment at least includes polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively;

[0011] The optimized and adjusted material resources are output as web page video playback links and shared on the WEB side.

[0012] A further technical solution of the present invention is that the resources in the 3D editing engine on the WEB side are optimized and adjusted to form optimized and adjusted resources, including:

[0013] Polishing and adjusting parameters of the materials of the constructed 3D model scene in the 3D editing engine on the WEB side; wherein the parameter adjustment at least includes the adjustment of diffuse parameters, opacity parameters, and emission parameters;

[0014] Adding digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources at least include the model, texture, and animation of the digital human;

[0015] Creating an interactive operation of the audience login page;

[0016] Add the video playback link interface to the diffuse layer and replace the color layer;

[0017] Add the video playback link interface to the opacity layer and play the masked part;

[0018] Add the video playback link interface to the emission layer as part of the lighting simulation.

[0019] A further technical solution of the present invention is that the 3D editing engine supporting the WEB end adopts the Playcanvas engine based on the WebGL technology.

[0020] A further technical solution of the present invention is that before the optimized and adjusted material resources are output as a web page video playback link and shared to the WEB end, it also includes:

[0021] Performing online dynamic updating on each frame of the image in the optimized and adjusted material resources;

[0022] Each frame of video in the optimized and adjusted material resource is stream-loaded.

[0023] The present invention also provides a virtual-real fusion device based on a metaverse performance exhibition venue, comprising:

[0024] A construction and production module is used to construct and produce a 3D model that matches the actual performance scene data to form a constructed and produced 3D model scene;

[0025] The matting and post-processing module is used to perform matting and post-processing on the pre-recorded performance video to form a processed performance video file; wherein the post-processing at least includes lighting processing, color adjustment processing, and video portal special effects processing;

[0026] A virtual-real fusion module is used to integrate the constructed 3D model scene and the processed performance video file into a 3D editing engine supporting the WEB end, thereby forming a material resource of the 3D editing engine supporting the WEB end;

[0027] An optimization and adjustment module is used to optimize and adjust the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment content at least includes polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively;

[0028] The output module is used to output the web video playback link of the optimized and adjusted material resources and share it to the WEB end.

[0029] A further technical solution of the present invention is that the optimization and adjustment module is specifically used to polish the material and adjust the parameters of the constructed 3D model scene in the 3D editing engine of the WEB end; wherein the parameter adjustment at least includes the adjustment of the diffuse parameter, the opacity parameter, and the emission parameter;

[0030] Adding digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources at least include the model, texture, and animation of the digital human;

[0031] Creating an interactive operation of the audience login page;

[0032] Add the video playback link interface to the diffuse layer and replace the color layer;

[0033] Add the video playback link interface to the opacity layer and play the masked part;

[0034] Add the video playback link interface to the emission layer as part of the lighting simulation.

[0035] A further technical solution of the present invention is that the 3D editing engine supporting the WEB end adopts the Playcanvas engine based on the WebGL technology.

[0036] A further technical solution of the present invention is that it also includes an online dynamic update module and a streaming loading module;

[0037] The online dynamic update module is used to perform online dynamic update on each frame of the optimized and adjusted material resources;

[0038] The streaming loading module is used to stream load each frame of video in the optimized and adjusted material resources.

[0039] The beneficial effects of the present invention are:

[0040] The present invention provides a virtual-real fusion method for a performance exhibition venue based on a metaverse, comprising step S101: constructing and producing a 3D model that matches actual performance scene data to form a constructed and produced 3D model scene; step S102: performing matting and post-processing on a pre-recorded performance video to form a processed performance video file; wherein the post-processing at least includes lighting processing, color adjustment processing, and video portal special effects processing; step S103: integrating the constructed and produced 3D model scene and the processed performance video file into a 3D editing engine that supports a WEB end, to form a 3D that supports a WEB end. Editing engine material resources; step S104: optimizing and adjusting the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment content at least includes polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively; step S105: outputting the web page video playback link for the optimized and adjusted material resources, and sharing it to the WEB side.

[0041] Specifically, the above implementation first builds and produces a 3D model that matches the actual performance scene data in the 3D software; then, the required equipment is used in the green screen studio to record the live performance video, for example, live band performances, singing performances, opera performances and other music festival activities in advance; then, the pre-recorded performance video is subjected to green screen matting and post-processing to increase the artistic effect of the video and add real-time artistic effects to the live video; the built and produced 3D model scene and the processed performance video file, for example, the key objects and characters such as the performers and musical instruments are truly integrated into the 3D editing engine that supports the WEB side, forming the material resources of the 3D editing engine that supports the WEB side; and the material resources in the 3D editing engine on the WEB side are processed. Optimize and adjust to form optimized and adjusted material resources, output the optimized and adjusted material resources as web page video playback links, and share them on the WEB side; wherein the optimization and adjustment content at least includes the polishing and parameter adjustment of materials in the 3D model scene, the addition of digital human resources, the creation of interactive operations for the audience, and the addition of video playback link interfaces in the diffuse diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively; in the form of performance combining virtuality and reality, the performance is linked online through WEB technology, so that the audience with Internet receiving devices within the coverage of the Internet can watch the performance online, and a virtual image is generated for each logged-in audience, so as to realize the interactive function of the audience participating in the program and the on-demand function of the program.

[0042] The present invention can realize the virtual-real combination of computer-rendered animation special effects and real-person live performances recorded by cameras, combine real-person performance videos with 3D scenes, and improve the artistic effect and simulation effect of the performance venue through AE software processing and 3D material ball configuration, so as to achieve the purposes of sharing, dissemination, interactivity and participation; the present invention can use limited resources to add new technical paths in the artistic special effect processing link of the performance and the interactive mode link of the audience participating in the performance, create a more innovative and interactive and exploratory performance experience, further deepen and strengthen the technology of the metaverse performance venue, improve the artistic effect, simulation effect, participation and interactivity of the metaverse performance venue, and make the communication more real and rich; people can interact with each other in real time by connecting the network through user-end devices at different locations as needed, without being restricted by space and time; it can realize multi-faceted interaction of voice, video, action and somatosensory in the same virtual space by connecting the network through multiple user-end devices, making the interaction between multiple people more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a virtual-real fusion method of a performance exhibition venue based on the metaverse provided by a specific embodiment of the present invention;

[0044] Figure 2 It is a schematic diagram of a visual effect diagram of a 3D model of a virtual performance of a metaverse provided by a specific embodiment of the present invention;

[0045] Figure 3 It is a screenshot schematic diagram of a recording scene during green screen recording provided by a specific embodiment of the present invention;

[0046] Figure 4 It is a screenshot schematic diagram after the original background-removed material and the mask layer are unified into a video, provided by a specific embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of the presentation effect of a virtual-real fusion method of a music festival performance venue based on the Metaverse Internet terminal provided by a specific embodiment of the present invention;

[0048] Figure 6 It is a structural schematic diagram of a virtual-reality fusion device for a performance exhibition venue based on the metaverse provided by a specific embodiment of the present invention.

[0049] In the figure: 201, construction and production module; 202, matting and post-processing module; 203, virtual-reality fusion module; 204, optimization and adjustment module; 205, online dynamic update module; 206, streaming loading module; 207, output module. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0051] The performance venue in the virtual-real fusion method of the performance venue based on the metaverse of the present invention can be a performance venue for a pure music festival, or a performance venue for a stage play, an opera, etc. In the existing metaverse performance venue based on the WEB end, there are mainly the following common modular combination methods. The first is the video output method link: mainly video real-scene recording, green screen recording and live broadcast; the second is the artistic special effects processing link of the performance: mainly the artistic effect rendering method through the LED screen environment on site, the method of adding artistic effects to the recorded video through post-processing, and the method of adding real-time artistic effects to the live video; finally, the interactive method link of the audience participating in the performance: mainly through text and expression comments, and the interactive method of sending real-time barrage functions. The above three different modular combinations form most of the current metaverse performance venues based on the WEB end. However, under the background of metaverse technology, how to use computer-rendered animation special effects and the virtual-real combination of real-person live performances recorded by cameras to improve the artistic effect and simulation effect of the performance venue, and realize its sharing, dissemination, participation and interactivity, has become an urgent problem to be solved in enhancing the immersive experience of the metaverse. How to further deepen and strengthen the metaverse performance exhibition technology based on the performance exhibition methods proposed above has become the main direction of this technical implementation method.

[0052] Figure 1 is a flowchart of a method for integrating virtuality and reality based on a performance exhibition venue of the metaverse provided by a specific embodiment of the present invention, such as Figure 1 A method for integrating virtuality and reality of a performance exhibition based on the metaverse includes: step S101: constructing and producing a 3D model that matches the actual performance scene data to form a constructed and produced 3D model scene; step S102: performing matting and post-processing on a pre-recorded performance video to form a processed performance video file; wherein the post-processing at least includes lighting processing, color adjustment processing, and video portal special effects processing; step S103: integrating the constructed and produced 3D model scene and the processed performance video file into a 3D editing engine that supports the WEB end, to form a 3D editing engine that supports the WEB end. The material resources of the engine; step S104: optimizing and adjusting the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment content at least includes polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively; step S105: outputting the web page video playback link for the optimized and adjusted material resources, and sharing it to the WEB side.

[0053] Specifically, the above implementation first builds and produces a 3D model that matches the actual performance scene data in the 3D software; then, the required equipment is used in the green screen studio to record the live performance video, for example, live band performances, singing performances, opera performances and other music festival activities in advance; then, the pre-recorded performance video is subjected to green screen matting and post-processing to increase the artistic effect of the video and add real-time artistic effects to the live video; the built and produced 3D model scene and the processed performance video file, for example, the key objects and characters such as the performers and musical instruments are truly integrated into the 3D editing engine that supports the WEB side, forming the material resources of the 3D editing engine that supports the WEB side; and the material resources in the 3D editing engine on the WEB side are processed. Optimize and adjust to form optimized and adjusted material resources, output the optimized and adjusted material resources as web page video playback links, and share them on the WEB side; wherein the optimization and adjustment content at least includes the polishing and parameter adjustment of materials in the 3D model scene, the addition of digital human resources, the creation of interactive operations for the audience, and the addition of video playback link interfaces in the diffuse diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively; in the form of performance combining virtuality and reality, the performance is linked online through WEB technology, so that the audience with Internet receiving devices within the coverage of the Internet can watch the performance online, and a virtual image is generated for each logged-in audience, so as to realize the interactive function of the audience participating in the program and the on-demand function of the program.

[0054] The present invention can realize the virtual-real combination of computer-rendered animation special effects and real-person live performances recorded by cameras, combine real-person performance videos with 3D scenes, and improve the artistic effect and simulation effect of the performance venue through AE software processing and 3D material ball configuration, so as to achieve the purposes of sharing, dissemination, interactivity and participation; the present invention can use limited resources to add new technical paths in the artistic special effect processing link of the performance and the interactive mode link of the audience participating in the performance, create a more innovative and interactive and exploratory performance experience, further deepen and strengthen the technology of the metaverse performance venue, improve the artistic effect, simulation effect, participation and interactivity of the metaverse performance venue, and make the communication more real and rich; people can interact with each other in real time by connecting the network through user-end devices at different locations as needed, without being restricted by space and time; it can realize multi-faceted interaction of voice, video, action and somatosensory in the same virtual space by connecting the network through multiple user-end devices, making the interaction between multiple people more convenient.

[0055] Among them, in step S101, a 3D model matching the actual performance scene data is constructed and produced to form a constructed and produced 3D model scene, specifically: (1) using field investigation and shooting and other means to collect information about the real scene to be virtualized, that is, the 3D model scene. Mainly collect the appearance of the facade of the main building to be constructed, collect the three-dimensional space of the main building, and evaluate the auxiliary environment. The facade of the main building to be constructed and produced can be recorded and saved by taking photos with a mobile phone or camera; the 3D model of the main building can be obtained from the developer of the building or related institutions, or a rough 3D reconstruction can be performed based on the building space structure. These methods can obtain a 3D model that matches the performance scene data; finally, according to the overall effect to be presented, the surrounding auxiliary environment is selected, such as: vegetation trees in green belts, public roads, other buildings, sky environment and other auxiliary environments; (2) According to the needs of the performer, the virtual main stage is designed, and some animation designs with artistic effects are added, such as the way the stage opens and the way the performers go on stage; (3) Using the main building photo information and auxiliary environment information collected above as a reference, the virtual 3D model scene is produced and built in MAYA and other software. After constructing the final presentation effect of the main stage and the main building model, the texture map of the surface of the main building is produced using the photo reference. The production of the texture map of the surface of the main building includes the following steps:

[0056] (a) After the virtual 3D model scene is produced, the virtual 3D model scene is decomposed into UV in a 3D software, such as MAYA software;

[0057] (b) After the UV decomposition is completed, the scene is processed with lighting simulation, material simulation, and texture simulation in the 3D software; the processing effects of lighting simulation, material simulation, and texture simulation can be adjusted according to the specific implementation situation, and will not be described in detail here;

[0058] (c) After the lighting, material and texture simulation processing is completed, use renderers such as Vray, redshift, Oc, etc. to convert lighting information, material mapping, texture bumps and other information into image information using texture baking technology. The texture content used is: light map, color map, luminous map, normal map, reflection map, transparent map, etc.;

[0059] (d) After the texture baking information is transferred, go to the Web3D engine, import the model, material, and texture information, and restore the rendering effect in the 3D software.

[0060] After completing the production of the texture map, add the selected surrounding environment elements, and then integrate the main stage, main building and ancillary environmental resources to form a preliminary metaverse exhibition virtual environment.

[0061] In addition, according to: (2) according to the needs of the performers, the virtual main stage is designed, and some animation designs with artistic effects are added, such as the way the stage opens and the way the performers go on stage. Specifically, the stage opening animation and the artistic special effects of the performers going on stage can be produced in MAYA software: if there are animation calls in the scene, it is necessary to perform animation production in 3D software. The specific method is: record the model coordinate information at the starting position of the animation, record the position coordinate information again at the end position of the animation, and finally adjust the coordinate information during the animation movement to achieve the animation effect; package all files and export them in FBX format for backup.

[0062] In the above implementation, the 3D model includes the main building and the ancillary environment. The main building is the facade of the main building to be built, the three-dimensional space of the main building, and the main stage; the ancillary environment is the vegetation and trees in the green belt, public roads, other buildings, sky environment, etc.; the construction and production of the 3D model can be built and produced in professional surface modeling software such as MAYA, 3DMAX, C4D, Blender, etc. Figure 2 It is a visual effect diagram of a 3D model of a virtual performance of the metaverse provided by a specific embodiment of the present invention, such as Figure 2 shown.

[0063] In step S102, the pre-recorded performance video is matted and post-processed to form a processed performance video file; wherein the post-processing includes at least lighting processing, color processing, and video portal special effects processing; specifically, the live performance video needs to be recorded first; in order to integrate real people and performances into the stage of the metaverse performance, the performance needs to be recorded on the spot in real time on the green screen in advance, so that it can be naturally integrated into the built virtual 3D model scene later. To complete the recording of the level of simulated real live performance, the following equipment is required: 4K movie-level recorder, model: SONYPXW-Z150; sound recording equipment: wired condenser microphone, model: SONY ECM-100N; background wall made of soft light fabric green environment, specification: 2.5m×8m; according to the main stage height designed in the above, the average visual height of the performance character matches the viewing angle and height of the live camera; at the same time, according to the spatial structure of the recording site, find the best microphone reception position; reasonably arrange the performer's position in front of the green screen; and shoot the green screen live performance that meets the scene according to the above parameters.

[0064] Specifically, the lighting processing may be: simulating real-time lighting on the video according to the recording scene, adjusting its self-luminous properties, and synchronously simulating the ambient lighting; specifically, the color adjustment processing may be: before generating the final image, the image may contain more green or blue color information due to the recording, and the video may be color adjusted by post-production video software, such as AE, to remove excess green or blue color information to restore the true color effect and further restore the authenticity; then, the following methods may be used to create a video portal effect to enrich the video interpretation effect, including: (1) using post-production software tools to create transition effects; (2) using video materials (portals) directly as transition effects; and (3) using scene objects for occlusion.

[0065] Figure 3 is a schematic diagram of a screenshot of a recording scene during green screen recording provided by a specific embodiment of the present invention, such as Figure 3 shown.

[0066] In step S102: the pre-recorded performance video is matted. In the specific implementation, the recorded video can be matted using AfterEffect software. In order to achieve a more realistic and natural real-life shooting matte, the following process is required: first, the alpha transparent layer of the video is removed; then the transparency is processed using masking technology; finally, the original removed material and the mask layer are combined into a video. Figure 4 As shown, Figure 4 It is a screenshot schematic diagram after the original background-removed material and the mask layer are unified into a video, provided by a specific implementation method of the present invention.

[0067] Step S103: The constructed and produced 3D model scene and the processed performance video file are integrated into the 3D editing engine that supports the WEB side, so as to form the material resources of the 3D editing engine that supports the WEB side; specifically, the constructed and produced 3D model scene and the processed performance video file are transplanted into the Playcanvas engine based on WebGL technology.

[0068] After all necessary materials are integrated into the Playcanvas engine, due to changes in the engine, these resources need to be finally optimized and adjusted in the Playcanvas engine.

[0069] A further technical solution of the present invention is to optimize and adjust the resources in the 3D editing engine on the WEB side to form optimized and adjusted resources, including: polishing the materials and adjusting the parameters of the 3D model scene built and produced in the 3D editing engine on the WEB side; wherein the parameter adjustment at least includes the adjustment of the diffuse parameter, the opacity parameter, and the emission parameter; adding digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources at least include the model, texture, and animation of the digital human; creating interactive operations for the audience login page; adding a video playback link interface in the diffuse layer and replacing the color layer; adding a video playback link interface in the opacity layer and playing the masked part; adding a video playback link interface in the emission layer and using it as a part of simulating lighting.

[0070] Specifically, the material properties of the virtual scene and the environment may have slight differences in the above implementation, so further polishing and adjustment are required, including: 1. (1) It is mainly based on the effect rendered by the Playcanvas engine, and then fine-tuning the various parameters of the virtual scene material (such as diffuse parameters, opacity parameters, emission parameters, etc.) to make the scene rendering effect closer to the design effect. (2) Then, arrange and schedule the stage opening and closing animations in the FBX format that have been produced previously; when each program starts, schedule the stage opening animation; when the program ends, schedule the stage closing animation; in this way, the complete opening and closing process of the performance is presented. (3) Finally, the model resources and animation resources of the virtual scene and the scheduled animation are packaged and converted into JSON and GLB formats that can be called by the WebGL environment.

[0071] 2. Add existing digital human resources to the Playcanvas engine, where (1) digital human resources include digital human models, textures, animations, etc. (2) Then, create relevant interactions after the audience logs in to the page, so that the audience can select the digital human image after logging in (this solution provides two options for male and female roles), select the live program (through the drop-down menu floating on the right side of the screen), and select the digital human action (select digital human actions such as celebration and applause through the floating window at the bottom of the screen) through the device end connected to the Internet (such as mobile phones, tablets, computers, etc.).

[0072] 3. On the virtual stage, the cutouts and post-processed live performance video clips are integrated with the virtual scene through the following three methods. The specific implementation method is as follows: (1) Add a video playback link interface to the diffuse map layer of the dynamic image (video), and replace the color layer (the applicable scope is the part of video playback); the diffuse map layer mainly shows the reflection and surface color of the object surface; (2) Then add the same video playback link to the opacity, that is, the transparent map layer of the dynamic image (video), and play the masked part; the transparent map layer defines the opacity of the map. Black is the transparent part, white is the opaque part, and gray is the translucent part; such as Figure 4 As shown in ; the content displayed in the video playback link interface is mainly divided into two parts, the upper part is used as the data of the transparent layer, and everything other than the characters is transparent and hidden. The lower part is used as the data of the diffuse map layer and the self-luminous map. (3) Finally, add the same video playback link interface to the emission map layer of the dynamic image (video) as the part of simulating lighting. The setting of its luminous value parameter is between 0.1 and 0.7 based on different on-site recorded video references (the applicable range is the part of video playback); the self-luminous map controls the color and brightness of the surface emitted light, which is used to supplement the brightness of the object surface.

[0073] A further technical solution of the present invention is to support a 3D editing engine on the WEB side, using a Playcanvas engine based on WebGL technology.

[0074] A further technical solution of the present invention is that before outputting the web video playback link of the optimized and adjusted material resources and sharing it to the WEB end, it also includes: online dynamic updating of each frame of the image in the optimized and adjusted material resources; and streaming loading of each frame of the video in the optimized and adjusted material resources.

[0075] Specifically, after all necessary materials are integrated into the Playcanvas engine, the final stage is to make final optimization and adjustments to these resources in the Playcanvas engine (such as converting the model format to gltf format and compressing it), and finally output it as a web link and publish it online. Figure 5 : is a schematic diagram of the presentation effect of a virtual-real fusion method of a music festival performance venue based on the Metaverse Internet terminal provided by a specific embodiment of the present invention, such as Figure 5As shown. Because the performance on the virtual stage is a live performance shot in real life, it already has a high degree of authenticity. In order to further improve its fidelity, it is necessary to dynamically update each frame of the image in the video playback stream online to achieve delay-free animation. Online dynamic update requires obtaining the video stream of the video tag of the playing video, and storing each frame of the video stream into the memory buffer of a Texture object. In the update method, the frame data obtained from the current video stream is continuously stored in the memory buffer of the Texture object to replace the old frame data. In this way, the performance content on the virtual stage can be updated in real time. In addition, since the audience needs to obtain the performance resources to the local device through the Internet, ensuring the smoothness of the streaming media playback has become an extremely important part of improving the viewing experience. Here, the characteristics of HTML5 Video tag and MP4 format are utilized in Playcanvas so that the video can be played while being streamed (HTML5 Video soundtrack supports downloading and playing at the same time, and can be played without buffering the entire video), thus reducing the user's waiting time and the occupation of network resources, and effectively improving the smoothness of online playback of live performance videos; finally, all packaged content needs to be output as web links in the Playcanvas engine, so that audiences with Internet devices can watch the Metaverse Live Music Festival performances without time and space barriers.

[0076] Figure 6 is a structural schematic diagram of a virtual-real fusion device based on a metaverse performance exhibition venue provided by a specific embodiment of the present invention, such as Figure 6Therefore, the present invention also provides a virtual-reality fusion device for a performance exhibition venue based on the metaverse, including: a construction and production module 201, which is used to construct and produce a 3D model that matches the actual performance scene data to form a constructed and produced 3D model scene; a matting and post-processing module 202, which is used to matting and post-process the performance video recorded in advance to form a processed performance video file; wherein the post-processing at least includes lighting processing, color adjustment processing, and video portal special effects processing; a virtual-reality fusion module 203, which is used to integrate the constructed and produced 3D model scene and the processed performance video file into a 3D editing engine that supports the WEB end, to form a virtual-reality fusion module that supports the WEB end. The material resources of the 3D editing engine on the WEB side; an optimization and adjustment module 204, used to optimize and adjust the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment content at least includes the polishing and parameter adjustment of the materials in the 3D model scene, the addition of digital human resources, the creation of interactive operations for the audience, and the addition of video playback link interfaces in the diffuse reflection map layer, the opacity transparent map layer, and the emission luminous map layer; an output module 207, used to output the web page video playback link of the optimized and adjusted material resources and share it to the WEB side.

[0077] A further technical solution of the present invention is that the optimization and adjustment module 204 is specifically used to polish the materials and adjust the parameters of the 3D model scene built and produced in the 3D editing engine on the WEB side; wherein the parameter adjustment at least includes the adjustment of the diffuse parameter, the opacity parameter, and the emission parameter; add digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources at least include the model, texture, and animation of the digital human; create interactive operations for the audience login page; add a video playback link interface to the diffuse layer and replace the color layer; add a video playback link interface to the opacity layer and play the masked part; add a video playback link interface to the emission layer and use it as a part of simulating lighting.

[0078] A further technical solution of the present invention is to support a 3D editing engine on the WEB side, using a Playcanvas engine based on WebGL technology.

[0079] A further technical solution of the present invention is that it also includes an online dynamic update module 205 and a streaming loading module 206; the online dynamic update module 205 is used to perform online dynamic update on each frame of image in the optimized and adjusted material resources; the streaming loading module 206 is used to perform streaming loading on each frame of video in the optimized and adjusted material resources.

[0080] The present invention can realize the virtual-real combination of computer-rendered animation special effects and real-person live performances recorded by cameras, combine real-person performance videos with 3D scenes, and improve the artistic effect and simulation effect of the performance venue through AE software processing and 3D material ball configuration, so as to achieve the purposes of sharing, dissemination, interactivity and participation; the present invention can use limited resources to add new technical paths in the artistic special effect processing link of the performance and the interactive mode link of the audience participating in the performance, create a more innovative and interactive and exploratory performance experience, further deepen and strengthen the technology of the metaverse performance venue, improve the artistic effect, simulation effect, participation and interactivity of the metaverse performance venue, and make the communication more real and rich; people can interact with each other in real time by connecting the network through user-end devices at different locations as needed, without being restricted by space and time; it can realize multi-faceted interaction of voice, video, action and somatosensory in the same virtual space by connecting the network through multiple user-end devices, making the interaction between multiple people more convenient.

[0081] The present invention is described by preferred embodiments, and those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. The present invention is not limited to the specific embodiments disclosed herein, and other embodiments falling within the claims of this application are within the scope of protection of the present invention.

Claims

1. A virtual-real fusion method for performance exhibition based on the metaverse, It is characterized in that The following steps are involved: Build and produce a 3D model that matches the actual performance scene data to form a built and produced 3D model scene; Performing matting and post-processing on the pre-recorded performance video to form a processed performance video file; wherein the post-processing includes lighting processing, color adjustment processing, and video portal special effects processing; The constructed 3D model scene and the processed performance video file are integrated into a 3D editing engine supporting the WEB end, forming a material resource of the 3D editing engine supporting the WEB end; Optimize and adjust the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment include polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer, wherein: Polishing and adjusting parameters of the materials of the constructed 3D model scene in the 3D editing engine on the WEB side; wherein the parameter adjustment includes the adjustment of diffuse parameters, opacity parameters, and emission parameters; Adding digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources include the model, texture, and animation of the digital human; Creating an interactive operation of the audience login page; Add the video playback link interface to the diffuse reflection map layer and replace the color layer; Add the video playback link interface to the opacity transparent map layer, and play the masked part; Add the video playback link interface to the emission map layer as a part of simulating lighting; The optimized and adjusted material resources are output as web page video playback links and shared on the WEB side.

2. A method for integrating virtuality and reality based on a performance venue in a metaverse as claimed in claim 1, It is characterized in that The 3D editing engine supporting the WEB side adopts the Playcanvas engine based on the WebGL technology.

3. A method for integrating virtuality and reality based on a performance venue in a metaverse as claimed in claim 1, It is characterized in that Before outputting the web video playback link of the optimized and adjusted material resources and sharing it to the WEB end, the method further includes: Performing online dynamic updating on each frame of the image in the optimized and adjusted material resources; Each frame of video in the optimized and adjusted material resource is stream-loaded.

4. A virtual-reality fusion device for a performance exhibition based on a metaverse, applicable to the virtual-reality fusion method for a performance exhibition based on a metaverse as described in any one of claims 1 to 3, It is characterized in that include: A construction and production module is used to construct and produce a 3D model that matches the actual performance scene data to form a constructed and produced 3D model scene; The matte and post-processing module is used to matte and post-process the pre-recorded performance video to form a processed performance video file; wherein the post-processing includes lighting processing, color processing, and video portal special effects processing; A virtual-real fusion module is used to integrate the constructed 3D model scene and the processed performance video file into a 3D editing engine supporting the WEB end, thereby forming a material resource of the 3D editing engine supporting the WEB end; The optimization and adjustment module is used to optimize and adjust the material resources in the 3D editing engine on the WEB side to form optimized and adjusted material resources; wherein the optimization and adjustment content includes polishing and parameter adjustment of materials in the 3D model scene, adding digital human resources, creating interactive operations for the audience, and adding video playback link interfaces to the diffuse reflection map layer, opacity transparent map layer, and emission luminous map layer respectively; The output module is used to output the web video playback link of the optimized and adjusted material resources and share it to the WEB end.

5. A virtual-real fusion device based on a metaverse performance exhibition venue as claimed in claim 4, It is characterized in that The optimization and adjustment module is specifically used to polish the material and adjust the parameters of the constructed 3D model scene in the 3D editing engine of the WEB end; wherein the parameter adjustment includes the adjustment of the diffuse parameter, the opacity parameter, and the emission parameter; Adding digital human resources in the 3D editing engine on the WEB side; wherein the digital human resources include the model, texture, and animation of the digital human; Creating an interactive operation of the audience login page; Add the video playback link interface to the diffuse reflection map layer and replace the color layer; Add the video playback link interface to the opacity transparent map layer, and play the masked part; Add the video playback link interface to the emission map layer as part of the lighting simulation.

6. A virtual-real fusion device based on a metaverse performance exhibition venue as claimed in claim 4, It is characterized in that The 3D editing engine supporting the WEB side adopts the Playcanvas engine based on the WebGL technology.

7. A virtual-real fusion device based on a metaverse performance exhibition venue as claimed in claim 4, It is characterized in that It also includes an online dynamic update module and a streaming loading module; The online dynamic update module is used to perform online dynamic update on each frame of the optimized and adjusted material resources; The streaming loading module is used to stream load each frame of video in the optimized and adjusted material resources.