Digital program production system and method based on interactive hybrid virtual reality technology

Abstract

The invention discloses a digital program production system and method based on an interactive hybrid virtual reality technology. The digital program production system comprises a background display screen arranged in a photostudio; the XR rendering synthesis server is used for rendering the three-dimensional scene into video content and outputting the video content to the background display screen for display; the interaction props are used for collecting operation signals of the props; the AR rendering workstation is used for superposing the AR rendering content on the background display screen picture shot by the camera; the camera system is used for shooting the content of the display screen and shooting the image of the person in front of the display screen; the camera tracking system is used for acquiring motion data of a camera lens and a holder in real time; and the AR rendering workstation is also used for generating and outputting a final digital program video signal. An audio auxiliary system is further included in a preferable implementation mode, under the condition that the actor cannot see the virtual elements through vision, the audio auxiliary system can help the actor to conduct auditory positioning based on three-dimensional stereophonic sound, and then interaction is completed, and the audio auxiliary system is an important supplement of the system.

Description

Technical Field

[0001] The present invention relates to the field of mixed virtual reality and metaverse technology, and in particular to a digital program production system and production method based on interactive mixed virtual reality technology. Background Art

[0002] The Metaverse is a virtual world constructed using digital technology. It is a reflection of the real world or transcends the real world and can interact with the real world. It is a digital living space with a new social system.

[0003] For traditional radio and television programs, although some virtual reality and augmented reality technologies have been used, they are still not mature enough, especially there is no technical solution that deeply integrates metaverse technology with radio and television programs.

[0004] For example, some television programs that used live-action performances in virtual settings were filmed in a green box or blue box. This means the performers stood in a single-color screen space, such as a green box or blue box, to capture the original footage. Then, in post-production, the footage files involved in the virtual background were loaded onto the footage. However, this technology has the following drawbacks:

[0005] 1. In the green box or blue box, during live games or performances, participants cannot see their own environment and have no sense of space or immersion.

[0006] 2. It is impossible to interact with the virtual environment. This is essentially because the performers cannot see the environment they are in. Without seeing, there is no interaction, and interactive aspects such as sound, light and vibration are even more impossible to achieve.

[0007] 3. The presentation effect is unrealistic because it is recorded in a green box and a blue box. Real-time keying cannot achieve the true restoration of physical elements such as hair, shadows, reflections, smoke translucency, and metal reflective objects.

[0008] 4. Because the lighting of the virtual scene is post-produced, the lighting will not affect real people. The lighting of real people is always inconsistent with that of the virtual scene, and the visual effect is stiff.

[0009] 5. Since post-processing is required, it cannot meet the requirements of live broadcast and the workload of post-production is huge.

[0010] Currently, no effective solution has been proposed to the technical problems in the above-mentioned related technologies, such as poor immersion of performers in TV programs with real-person performances in virtual scenes, poor interactivity, stiff program presentation effects, and large post-production workload.

[0011] The recently emerged XR system, although the background display screen content solves the problem of environmental reference for actors' performances, is usually only used as a background for recording interview programs or as a background atmosphere for recording advertisements. It does not have the interactive function of actors' performances, cannot realize real-time interaction between real props and virtual scenes and virtual elements, and cannot realize digital program production based on interactive mixed virtual reality technology. Summary of the Invention

[0012] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a digital program production system and production method based on interactive mixed virtual reality technology to solve the technical problems in related technologies such as poor immersion of performers in television programs based on virtual scenes and real-life performances, poor interactivity, stiff program presentation effects, and large post-production workload.

[0013] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0014] According to one aspect of the present invention, a digital program production system based on interactive mixed virtual reality technology is provided, comprising:

[0015] A background display screen is provided in the studio; the background display screen surrounds or partially surrounds the actors participating in the digital program performance;

[0016] An XR rendering and synthesis server, the XR rendering and synthesis server is connected to the background display screen and is used to output the content rendered by the XR rendering and synthesis server to the background display screen for display; in addition, the XR rendering and synthesis server is connected to the output end of the motion capture workstation to receive prop motion information; the XR rendering and synthesis server is also wirelessly connected to the output end of the interactive prop trigger switch to receive the trigger operation information of the interactive prop;

[0017] A plurality of motion capture cameras are provided in the studio, the motion capture cameras being used to capture motion image information of the actors using the interactive props. The output ends of the motion capture cameras are connected to a motion capture workstation for transmitting the captured motion images to the motion capture workstation; the motion capture workstation calculates the prop motion angle and displacement information in the X, Y, and Z directions in real time;

[0018] Interactive props; the interactive props are equipped with a trigger switch for the actor to trigger and generate a trigger signal. In addition, the props are equipped with infrared reflective markers for motion capture, which facilitates the motion capture camera to obtain prop motion information.

[0019] An AR rendering workstation, the AR rendering workstation being connected to an output end of the motion capture workstation for receiving the prop motion information; the AR rendering workstation being further connected to an output end of the camera for superimposing AR rendering content on the image captured by the camera; the AR rendering workstation being further wirelessly connected to an output end of the interactive prop trigger switch for receiving trigger operation information of the interactive prop;

[0020] The UE rendering engine is installed in the XR rendering synthesis server and AR rendering workstation; the biggest feature of the UE rendering engine is real-time rendering, which is suitable for interactive game interaction, has high rendering quality, and good peripheral compatibility.

[0021] A camera system, disposed toward the background display screen and connected to the AR rendering workstation, configured to record the content of the background display screen and the image of the person in front of the background display screen;

[0022] The camera tracking system obtains real-time motion data of the camera lens and gimbal, and outputs it to the XR rendering and composition server and AR rendering workstation. When the camera moves, the scenes and elements in the XR rendering and composition server and AR rendering workstation are also rendered in real time based on the camera's motion data.

[0023] The prompt feedback system is used to inform actors of the appearance, location, and results of interactive objects, helping them determine their progress and status. This feedback system can be a software module for both the AR and XR systems. For example, a bird might emit one sound when it's hit and another when it's not. This allows actors to determine the outcome of the interaction based on the sounds. Audio feedback can be implemented by connecting the AR and XR systems to speakers via a sound card.

[0024] The AR rendering workstation is also used to generate the final digital program video signal and output it as a PGM;

[0025] The XR rendering synthesis server, the trigger switch of the interactive props, the AR rendering workstation, and the motion capture workstation are connected to the same local area network.

[0026] Furthermore, the background display screen includes a display screen arranged on the facade and a display screen on the ground, wherein the display screen on the facade is used to display the circumferential environment in the XR scene, and the display screen on the ground is used to display the ground environment in the XR scene.

[0027] Furthermore, the AR rendering content of the AR rendering workstation includes AR interactive game elements and background display screen extended AR elements. The AR rendering workstation superimposes the AR interactive game elements on the background display screen captured by the camera, so that the background display screen content only serves as the background, while the AR content can serve as the foreground. At the same time, the background display screen extended AR elements are also superimposed on the outside of the background display screen captured by the camera and seamlessly connected with the background display screen, so as to achieve the expansion of the scene content in the background display screen in a larger space, and can achieve 360-degree extended shooting, solving the problem of insufficient background display screen size. Of course, the real scene outside the background display screen is also blocked by the AR content.

[0028] Furthermore, the AR interactive game elements of the AR rendering workstation can be directly output on the background display screen captured by the camera system.

[0029] Furthermore, after receiving the operation data of the interactive prop, the AR rendering workstation updates the AR interactive game element attributes according to the operation data of the interactive prop.

[0030] Furthermore, the XR scene includes the on-screen scene displayed on the background display screen. The extended scene beyond the range of the background display screen is rendered and output by the AR rendering workstation and superimposed on the background display screen image captured by the camera, and together they are output as the final digital program video signal.

[0031] Furthermore, it also includes an audio auxiliary system, which includes a sound card installed in the XR rendering synthesis server and the AR rendering workstation, and a speaker array set in the studio area for outputting a three-dimensional sound field. The sound card is connected to the speaker signal, and the pre-produced audio is bound to the interactive game elements in the three-dimensional scene. During the recording of the game show, the interactive game elements in the XR rendering synthesis server trigger the playback of the audio according to the settings, and the speaker array generates a three-dimensional sound field, so that the actors can perceive the spatial position of the interactive game elements in the three-dimensional space through hearing, and thus perform corresponding interactive operations.

[0032] The present invention also provides a digital program production method based on interactive mixed virtual reality technology. Based on the above-mentioned digital program production based on interactive mixed virtual reality technology, the method includes:

[0033] The XR rendering and synthesis server receives the XR scene designed by the design workstation, and performs XR rendering synthesis based on the motion data of the camera system and the prop movement information and prop switch information of the motion capture workstation, and outputs the synthesized XR scene to the background display screen; the prop information is used to calculate whether the virtual elements in the XR scene or the scene will interact with the virtual props, such as collision, relative displacement, opening and closing, etc.

[0034] The AR rendering workstation also receives scenes from the design workstation and overlays the AR rendering content onto the background display screen captured by the camera. This AR rendering content includes AR interactive game elements and extended AR elements on the background display screen. It also receives prop motion information, interactive prop switch trigger data, and camera system motion data from the motion capture workstation. Prop-related information is used to render virtual elements associated with the physical props, such as virtual bullets fired from a prop gun or virtual arrows fired from a prop bow. Of course, the AR rendering workstation also generates extended content beyond the background display screen content, as well as virtual foreground elements. These elements, such as extended forests, birds flying in the sky, and plants and insects on the ground, can serve as foreground elements and are then overlaid with the background display screen captured by the camera to produce the final digital program video signal.

[0035] The present invention provides a digital program production system and production method based on interactive mixed virtual reality technology. The system includes an XR rendering and synthesis server, an AR rendering workstation, a display screen, a motion capture camera, interactive props, a camera system and other components. The XR rendering and synthesis server and the AR rendering workstation work together. The former is used to generate an XR scene as a virtual game environment, and the latter is used to generate AR game content and an extension of the background display screen content, and superimposed on the XR scene screen to generate a final program video signal. The XR scene is output through the display screen, and the display screen surrounds the performers, giving the performers an immersive feeling. At the same time, the performers can play games through interactive props. The data of the interactive props and the motion capture information will be sent back to the AR rendering workstation for real-time change of the game process, and then output to the display screen through the XR rendering and synthesis server. In this way, the performers do not need to perform on a green screen as in traditional technology, and then integrate the virtual scene through post-production special effects. Instead, they can play games directly in the virtual reality scene, which has strong interactivity, vivid program effects, and small post-production workload. The camera system records the screen and the performers together, and outputs them to the TV terminal through the AR rendering workstation, generating the final program signal for the audience to watch, thus completing the program production. This is to solve the technical problems in related technologies of TV programs with real-person performances in virtual scenes, such as poor immersion of performers, poor interactivity, stiff program presentation effects, and large workload in post-production. In the preferred embodiment, it also includes an audio assistance system, which can help actors locate their hearing based on three-dimensional stereo when the actors cannot see the virtual elements through vision, and thus complete the interaction. It is an important supplement to this system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A structural diagram of a digital program production system based on interactive mixed virtual reality technology is provided for an embodiment of the present invention;

[0037] Figure 2 A structural diagram of an audio auxiliary system in a digital program production system based on interactive mixed virtual reality technology is provided for an embodiment of the present invention;

[0038] Figure 3 A flowchart of a digital program production method based on interactive mixed virtual reality technology is provided for an embodiment of the present invention;

[0039] Figure 4 This is a live picture of the green screen production scene recording of a traditional program;

[0040] Figure 5 This is a scene diagram of a digital program production system based on interactive mixed virtual reality technology in an embodiment of the present invention (before the display screen displays any content);

[0041] Figure 6This is a scene diagram of a digital program production system based on interactive mixed virtual reality technology in an embodiment of the present invention (after the display screen displays the content);

[0042] Figure 7 A block diagram of the game interaction system design in an application example of the present invention;

[0043] Figure 8 This is a logic block diagram of a game interaction system in an application example of the present invention;

[0044] Figure 9 This is a schematic diagram of an actor standing in a background display screen and holding a prop gun to hit a virtual target in the background display screen in an application example of the present invention;

[0045] Figure 10 This is a schematic diagram of shooting at virtual targets outside the background display screen and smashing four targets inside the background display screen in an application example of the present invention;

[0046] Figure 11 A site diagram of the test environment in an application example of the present invention;

[0047] Figure 12 This is a site diagram of the connection system in an application example of the present invention;

[0048] Figure 13 This is a prop with reflective material installed in the application example of the present invention, and the surrounding area is a scene of the motion capture system;

[0049] Figure 14 Special effects diagram for interactive props;

[0050] Figure 15 This is a special effects image of another interactive prop;

[0051] Figure 16 This is a special effects image for another interactive prop. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0053] An embodiment of the present invention provides a digital program production system based on interactive mixed virtual reality technology. The system ultimately outputs program content as television images for television viewers to watch through terminals such as televisions, mobile phones, tablets, laptops, and desktop computers.

[0054] Specifically, the system includes:

[0055] A display screen 1 is provided in the studio; the display screen surrounds or partially surrounds the personnel participating in the digital program performance. For example, in some embodiments, the display screen can be provided as a vertical display screen, such as a ring screen (or three display screens spliced together, such as Figure 4 The back LED, left LED, and right LED shown in the figure also include a display screen set on the ground, wherein the display screen on the facade can be used to display the XR scene transmitted by the XR rendering synthesis server, that is, the circumferential environment of the XR virtual environment, and the display screen on the ground can be used to display the ground in the virtual environment to enhance the immersion of the performers.

[0056] An XR rendering and synthesis server is connected to the display screen, and is used to output the content data of the XR rendering and synthesis server, such as the rendered and synthesized XR scene, to the display screen for display; the XR rendering and synthesis server is used to generate a virtual environment for a digital program, such as a tropical rainforest, an ice field, a city, etc. Various types of virtual environment modeling files can be pre-set in the XR rendering and synthesis server. These modeling files can be pre-designed by the designer in the design workstation and then transmitted to the XR rendering and synthesis server. The XR rendering and synthesis server can adopt the XR system provided by Pixotope or the XR system provided by disguise. In addition, in other embodiments, the XR rendering and synthesis server can also output game interactive elements. For example, the targets to be shot (such as flying birds) mentioned below can be output to the background display screen so that the actors can observe the interactive content of the game.

[0057] Multiple motion capture cameras 2 are installed in the studio. These capture motion information from the actors' interactive props. The outputs of these cameras are connected to a motion capture workstation, which transmits the captured motion information to the workstation. The workstation calculates the prop's motion angle and displacement in the X, Y, and Z directions in real time. The workstation can use a VICON motion capture system or a domestically produced Bronze system.

[0058] Interactive props 3: The interactive props are equipped with a trigger switch for the actor to trigger and generate a trigger signal. In addition, the props are equipped with infrared reflective markers for motion capture, which facilitate the motion capture camera to obtain prop motion information. Interactive props are usually determined based on the AR interactive game content required by the digital program. For example, for gun battles and shooting games, interactive props are firearms. For close combat games, interactive props can be swords, gloves, etc. For vehicle racing games, interactive props can be racing cars, rowing boats, etc. These interactive props can also be equipped with force sensors to collect signals from the interactive props. For example, a six-axis sensor can collect the three forces (FX, FY, FZ) and three moments (MX, MY, MZ) in the neutral coordinate system (OXYZ) of the interactive props. This makes the collected data more comprehensive and helps improve the accuracy of AR interactive game interactions.

[0059] An AR rendering workstation is connected to the output end of the motion capture workstation to receive the prop motion information; the input end of the AR rendering workstation is also connected to the output end of the camera to superimpose the AR rendering content on the background display screen captured by the camera; the AR rendering workstation is also wirelessly connected to the output end of the interactive prop trigger switch to receive the trigger operation data of the interactive prop.

[0060] There are many brands and types of AR rendering workstations. For example, you can use the AR rendering workstation provided by Xin Ao Te (Totem) or the AR rendering workstation of PIXOTOPE.

[0061] The UE rendering engine is installed in the XR rendering synthesis server and AR rendering workstation. The biggest feature of the UE rendering engine is real-time rendering, which is suitable for interactive game interaction, has high rendering quality, and good peripheral compatibility.

[0062] As an example, the scene elements of an AR rendering workstation include AR interactive game elements and AR background display extension elements. That is, the rendered content of the AR workstation includes AR interactive game elements and AR background display extension elements. For example, in a shooting game, AR interactive game elements can serve as virtual supplements to foreground targets or physical props, such as virtual bullets, arrows, birds, plants, and insects. The corresponding XR scene can be a forest, and the AR background display extension elements serve as an extension of the forest outside the background display. In this case, when the coordinates of a bird in the XR or AR scene overlap with the coordinates of a fired virtual bullet or arrow, the game logic determines a collision, and the target is hit; otherwise, it is not hit. This design allows actors participating in digital program performances to observe the AR interactive game content in the virtual environment, such as the forest and birds, on a large display screen and engage in AR interactive games using the interactive props. If an actor wants to shoot at a target on the background display screen, a corresponding small screen can be hung or placed in the corresponding location in the studio to display the target. Similarly, when the virtual bullet overlaps with the target coordinates, the target is hit.

[0063] For example, the AR interactive scene of an AR rendering workstation can be a shooting game, and the corresponding interactive props are guns with force sensors. As mentioned above, the AR rendering workstation superimposes AR elements (bullets, arrows, tropical rainforest expansion, etc.) on the XR scene synthesized by the XR rendering synthesis server (such as a tropical rainforest). In this way, when the XR scene that integrates game interactive elements is output to the display screen, the actor can intuitively see the environment he is in and know the game goals that need to be interacted with, which has a strong sense of immersion. On this basis, the actor only needs to operate the interactive props and make corresponding movements. The prop motion information collected by the motion capture system and the interactive prop trigger data will be sent back to the AR rendering workstation to update the AR interactive game data of the AR rendering workstation. For example, in the aforementioned shooting game, the motion capture camera captures the motion and angle data of the physical props and transmits this data to the AR rendering workstation. The AR rendering workstation can then use this motion data. After the force sensor of the firearm prop transmits the data to the AR rendering workstation, the AR rendering workstation can use the force sensor data, the trigger signal, and the coordinate information of the target to be shot in the game to determine the trajectory of the shot. If the coordinate information corresponding to the trajectory coincides with the coordinate information of the target to be shot, it is determined that the target has been hit. After the hit, the game logic determines that an object collision has occurred, and the corresponding game data is updated, such as displaying the animation effect of the target being hit. This is synchronized to the XR rendering and synthesis server, and finally transmitted to the display screen. In this way, during the performance, the actor can not only see the surrounding virtual environment, but also can instantly observe the interaction with all targets in the virtual environment, providing an excellent interactive experience.

[0064] A camera system 4 is arranged toward the display screen and connected to the AR rendering workstation, and is used to record the contents of the display screen and the image of the person in front of the display screen, thereby transmitting the recorded data to the AR rendering workstation. When recording, the camera system can record the entire content of the display screen to more comprehensively display the interactive game content. The camera of the camera system can be set to multiple positions for shooting from different angles.

[0065] Furthermore, the present invention also includes a camera tracking system that acquires real-time motion data of the camera lens and gimbal, and outputs it to the XR rendering and composition server and AR rendering workstation. When the camera moves, the scenes and elements in the XR rendering and composition server and AR rendering workstation are also rendered in real time based on the camera's motion data. In this way, the data acquired by the camera tracking system is transmitted to the AR rendering system and the XR rendering system, which render the scene therein, so that when the camera moves, the rendered scene also moves accordingly.

[0066] In some embodiments, the display screen 3 is not sufficient to display the entire content of the XR scene. Taking the forest scene as an example, part of the content can be displayed on the display screen (referred to as on-screen content 5), while the content beyond the display range of the display screen (referred to as off-screen content 6) cannot be displayed. If the screen image is only captured and output by the camera system 4, the audience may not be able to see the full picture of the designer's design scene, affecting the observation experience. In order to solve this technical problem, in an embodiment of the present invention, the XR rendering and synthesis server completes the rendering of the scene on the background display screen, including the on-screen content 5. The AR rendering workstation completes the rendering of the scene outside the background display screen, including the off-screen content 6. The off-screen content 6 and the on-screen content 5 can be seamlessly connected to enhance the visual integrity effect, so as to achieve the expansion of the scene content in the background display screen in a larger space, and can achieve 360-degree extended shooting to solve the problem of insufficient size of the background display screen. Of course, the real scene outside the background display screen is also blocked by the AR content. In this way, when the AR rendering workstation superimposes its AR rendering elements on the background display screen content captured by the camera (completed by the XR rendering server), the picture displayed in the final video signal generated is the entire scene including the on-screen content 5 and the off-screen content 6, which improves the audience's viewing experience.

[0067] The output end of the AR rendering workstation is connected to the TV, which is the final program synthesis screen. The AR rendering workstation also receives the motion information transmitted by the motion capture workstation, the operation data of the interactive props, and the background display screen image and camera tracking data taken by the camera system. For example, after the AR rendering workstation superimposes the AR interactive game elements and extended content on the XR screen, it generates a video screen of an actor playing an interactive game in a virtual scene. According to the motion tracking data of the camera system, the rendering angle of the scene and graphic elements can be changed accordingly. According to the sensor trigger data of the interactive props and the motion information transmitted by the motion capture workstation, the game content data of the AR interactive game can be refreshed accordingly, such as the effect of a flying bird being hit by a virtual bullet fired by a physical prop gun.

[0068] The embodiment of the present invention also includes a prompt feedback system, which is used to prompt the actors of the appearance, location and interaction results of interactive objects, and to assist the actors in judging the progress and status of participating in the program. The feedback system can be a software module of the AR system and the XR system (as part of the AR system and the XR system, it is not included in the Figure 1 For example, a bird makes one sound when it's hit and another when it's not. This way, the actor can determine the interaction outcome by listening to the sounds. For audio feedback, the AR and XR systems can be connected to speakers via a sound card.

[0069] Since the content captured by the camera system includes the actors and the entire display screen, and the final program image actually already contains the content fused by the AR rendering workstation and the XR rendering synthesis server, there is no need for post-production, which greatly saves post-production costs. Moreover, since no post-production is required, the program can be produced and broadcast in a live broadcast manner.

[0070] In digital program production systems and methods based on interactive mixed virtual reality technology, an important function is that actors can see the virtual elements they interact with, such as birds, targets, and airplanes. During the scene creation phase, these virtual objects are set to collision attributes according to program requirements, which means they will be hit and produce collision effects. There are two ways to present these virtual objects. One is to present them on the background display screen. As described in the previous embodiment, the XR rendering and synthesis server generates them and outputs them to the background display screen so that the actors can see them and aim and shoot with props. Based on the motion capture system and prop triggers, the exact posture and state of the props can be obtained. The engine then fires virtual bullets, hitting virtual objects with collision attributes, producing special effects such as explosions. However, due to the limited area of the background display screen, it is impossible to fully cover the studio area. To solve this problem, another approach is to set up one or more display screens in the studio that only the actors can see, namely independent display screens 7. AR interactive game elements (such as birds to be shot, which have collision attributes) can be directly displayed on these independent display screens 7. The independent display screen 7 can only be seen by the actors and will not be captured by the camera system 4, so it will not be exposed and affect the program effect. In addition, the actors can use interactive props to directly interact with the element, such as using a gun to aim and shoot a flying bird, thereby completing the interaction of the AR game. In addition, it is supplemented that multiple AR rendering workstations can be configured in the system. They do not input camera signals, but load scenes and motion capture data. They can render different scenes separately, similar to fully virtual game scenes. Their output can be used as the output signal of the program for the director to select the required scene screen for broadcast. The above two methods solve the problem of actors seeing the interactive targets and increase the realism of the game, rather than pure performance.

[0071] In other embodiments, the actor may not be able to see the interactive virtual elements through the large screen (i.e., display screen 3) or the small screen (i.e., independent display screen 7). However, in this case, the actor still needs to interact with the virtual elements, such as shooting a flying bird. To address this issue, embodiments of the present invention also provide an audio assistance system to enable the actor to complete the interaction when the actor cannot observe the virtual element interaction through the screen.

[0072] As an example, further combining Figure 2The audio auxiliary system includes a sound card installed in the XR rendering synthesis server and the AR rendering workstation, and a speaker array set in the studio area for outputting a three-dimensional sound field. The speaker array can be composed of a plurality of speakers 8 arranged geometrically. The sound card is connected to the speaker 8 signal, and the pre-made audio is bound to the interactive game elements in the three-dimensional scene. During the recording of the game program, the interactive game elements in the XR rendering synthesis server trigger the playback of the audio according to the settings, and the speaker array generates a three-dimensional sound field, so that the actors can perceive the spatial position of the interactive game elements in the three-dimensional space through hearing, and thus perform corresponding interactive operations.

[0073] In the example above, the XR rendering and composition servers and the AR rendering and composition servers are equipped with sound cards, serving as both the rendering device for the 3D scene and the audio playback device. The speaker array installed in the studio is the output and reproduction device for the 3D sound field. They can be connected via a network switch. During the pre-recording phase of the game show, pre-produced audio is bound to the interactive game elements in the 3D scene. This audio supports a 3D sound field. During the recording process, the interactive game elements in the XR rendering and composition server trigger the audio playback according to the settings. Next, the 3D spatial reproduction of the sound requires the placement of a speaker array in the actor's activity area. Based on the needs of the specific game segments, multiple speakers are arranged in a specific geometric shape around the studio to form a speaker array and generate a 3D sound field. To ensure that the speakers do not interfere with the filming process, as a further improvement, the speaker placement should be strategically designed, such as in areas that are out of reach of the camera to avoid being captured, thereby enhancing the realism of the recording environment. With the above audio equipment set up, actors can use their hearing to perceive the spatial position of interactive game elements in 3D space during the game show recording and perform corresponding interactive actions. The audio auxiliary system makes up for the shortcomings of the visual system, helps the interactive program to be completed smoothly, and realizes immersive interactive mixed virtual reality digital program recording.

[0074] The audio assistance system is an important supplement to the XR reality system. As mentioned above, when the actor cannot see the interactive virtual elements through the display screen (large screen or small screen), the three-dimensional surround sound system set up in the studio can help the actor judge the spatial position of the virtual target, so that the actor can make a judgment through hearing and perform interactive operations. For example, in the game link setting, if the interactive element is a flying bird to be shot down, this embodiment can set the flight path of the bird, and its flight path includes three processes: presentation on a large screen, presentation on a small screen, and presentation without a screen. With screen presentation, the actor can make a visual judgment and make a shooting action. When there is no screen presentation, the sound matrix sound field arranged in the performance area can point to the spatial position of the bird for the actor, such as directly above the head or in front of the right, and the actor makes a corresponding shooting action through auditory judgment. The role of the audio assistance system in other game links is similar.

[0075] Audio-assisted systems effectively complement the shortcomings of display visual systems. For example, covering a studio with a 360-degree large screen would be prohibitively expensive. However, the integration of audio-assisted systems with large screen systems effectively solves the positioning problem of virtual interactive elements, representing a perfect marriage of audio and video technologies. As long as the 3D sound field is properly arranged, a 360-degree sound field can be delivered seamlessly for virtual interactive elements in games.

[0076] As described in the previous embodiments, in some scenarios, some interactive game elements can be generated by the XR rendering and synthesis server, while the remaining interactive game elements still need to be superimposed on the image captured by the camera system by the AR rendering workstation and ultimately output as a video program signal. For example, in a shooting game, the target to be shot, such as a flying bird, can be generated by the XR rendering and synthesis server, while elements such as fired bullets still need to be generated by the AR rendering workstation based on the collected interactive prop operation signals and superimposed on the captured image. In this way, the audience can watch the program screen including the complete interactive game elements in the final program.

[0077] The system preparation phase can include the following tasks: The initial preparation phase includes XR scene creation, AR prop creation (which can be completed in the design workstation), XR and AR equipment preparation, VR interactive prop preparation, and game program development. Multiple XR scenes and AR elements must be created based on the test objectives, each with different styles. For example, if three scenes are included, they could be modern, ancient, and natural environments. The modern scene is used for shooting interactions, while the ancient scene is used for interactions with umbrellas, fans, and kites. The natural environment features a rowing competition, with real-time measurement of the rowing speed to determine the speed of the boats on the water. AR props need to generate animations based on the on / off switches of the tracking device in the actor's hand. The choice of VR interactive props should be based on the test objectives. The gun is used for shooting, while the trigger can control the firing of AR bullets. The controller is used to control the sword, fan, umbrella, and kite. Triggering the switch on the controller can toggle the fan, umbrella, and laser sword on and off. When developing the game program, a set of logical processes are being developed for the use of virtual props, such as shooting and hitting objects to generate hit feedback, the switch of props triggering virtual prop animation, data statistics and other functional modules.

[0078] The on-site testing phase includes XR debugging, camera tracking debugging, lens calibration, VR interactive prop tracking debugging, and AR debugging. XR debugging aims to complete the performance background environment, allowing the actors to see their surroundings and their positions. They can also see interactive objects, ensuring accurate interaction, which avoids the issue of missing reference objects in the blue box method. Camera tracking debugging and lens calibration aim to create a correct perspective relationship between the actors and the scene, and ensure that AR virtual props seamlessly integrate with real-world props during camera and lens movement, eliminating any sense of misalignment. VR interactive prop tracking uses a motion capture system to capture the motion coordinates and orientation changes of the physical props, which then drives the movement of the virtual props. Furthermore, turning the VR interactive props on and off triggers the corresponding animation of the virtual props. Through the coordinated debugging of these steps, the interaction between the real actors and the virtual environment is ultimately achieved.

[0079] This embodiment also provides a digital program production method based on interactive mixed virtual reality technology, based on the above-mentioned digital program production based on interactive mixed virtual reality technology, combined with Figure 3 , the method includes: after the process starts:

[0080] In step S101, the XR rendering and synthesis server receives the XR scene designed by the design workstation, performs XR synthesis based on the motion data of the camera system and the motion information of the motion capture workstation, and outputs the synthesized XR scene to the background display screen.

[0081] For example, after receiving the XR scene, the XR rendering and synthesis server renders the image of the corresponding perspective based on the motion tracking data of the camera system and outputs it to the background display screen for display.

[0082] In step S102, the AR rendering workstation overlays the AR rendering content on the background display screen captured by the camera, wherein the AR rendering of the AR rendering workstation includes AR interactive game elements and content extension outside the background display screen; the AR rendering workstation also receives prop motion information, interactive prop operation data, and camera system motion data transmitted by the motion capture workstation, generates the final digital program video signal, and outputs it.

[0083] The present invention is different from the traditional virtual studio (VS) or virtual implant (AR) or virtual extension (XR) or 3D head-mounted display interaction that uses a blue box or a green box. Instead, it aims to enable real people to interact with virtual elements in a virtual environment in real time through real props in a virtual environment, thereby solving the problem that competitive programs such as reality shows require location shooting or the construction of a large number of real-scene props. At the same time, it solves the technical problems of poor immersion of performers, poor interactivity, stiff program presentation effects, and large post-production workload in live-action performance programs using virtual scenes. The present invention allows the program team and actor guests to realize imaginative creations on the background display screen in the studio. By changing the content of the background display screen, they can realize the travel through ancient and modern times, China and foreign countries, and the travel through the earth and space. More importantly, the actors can complete real-time interaction with virtual elements in the scene. This is a brand-new program shooting technical means, and it also enriches the diversity of program formats. In short, it is a reality show in a virtual space.

[0084] Application Examples

[0085] Further referring to FIG. 4-16 , in the variety show produced, the actors control external devices with sensors (i.e., the interactive props described in the above embodiment), and the motion data of the external devices is transmitted in real time to the UE rendering engine. The rendering engine has loaded the virtual devices or virtual props corresponding to the peripherals. Through the developed program, the motion mode of the virtual devices or virtual props is calculated according to the real-time input data, and finally the interactive effect is reflected in the XR and AR virtual scenes.

[0086] Different from fully virtual games, the present invention can regard digital variety shows as real-life games in a virtual environment. Therefore, successful application methods and strategies in games can also be used in digital variety shows.

[0087] The interactive system design process includes:

[0088] The core design of competitive games is to form the enduring content of PVP (Player VS Player) games through a diverse combination of character types and attributes in a single scene.

[0089] · Set up a mentor role to explain and stimulate the atmosphere when players are playing the game. At the same time, consumable game skill cards will be randomly drawn when each program is initialized, that is, the mentor random card attributes.

[0090] After each round of initialization, players will randomly draw matching character attribute values and skills, which will affect the physical properties of the game console, that is, the player's character card attributes.

[0091] The engine's built-in physics engine, that is, physical properties are built into the engine. The weather system is added with a certain degree of randomization to influence the properties. The combination of the above three dimensions creates an infinite combination of game experience and on-site topics.

[0092] Interactive system design block diagram

[0093] Modular consoles – By transferring various hardware values to the UE and restoring them within the UE, standardized, decoupled modules are formed through structured development. For example, this could include modular consoles for shooting, balancing, and endurance.

[0094] Physical Camera - A physical camera with FreeD data that transmits data to the UE in real time

[0095] Virtual Cameras – Virtual multi-cameras set up in a virtual scene. This can include fixed virtual cameras and mobile virtual cameras.

[0096] Real-time XR recording – Recording in an XR studio.

[0097] Equipment composition:

[0098] XR rendering and synthesis server, including XR screen, graphics rendering workstation, camera tracking system

[0099] AR rendering workstation

[0100] Interactive props (install sensors).

[0101] Real-time interaction:

[0102] XR background display shows the large environment scene

[0103] The player controls the physical props and searches for and aims at the interactive target through the background display or small screen.

[0104] Players trigger physical props to produce corresponding virtual effects through AR and XR background display screen scenes.

[0105] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A digital program production system based on interactive mixed virtual reality technology, characterized in that: include: Background display screens set up in the studio; The background display screen is arranged to surround or partially surround the actors participating in the digital program performance; An XR rendering and synthesis server, the XR rendering and synthesis server being connected to the background display screen and configured to output the rendered content of the XR rendering and synthesis server to the background display screen for display; In addition, the XR rendering and synthesis server is connected to the output end of the motion capture workstation to receive prop motion information; the XR rendering and synthesis server is also wirelessly connected to the output end of the interactive prop trigger switch to receive the trigger operation information of the interactive prop; A plurality of motion capture cameras are provided in the studio, the motion capture cameras being used to capture motion information of the interactive props used by the actors, the output ends of the motion capture cameras being connected to a motion capture workstation for transmitting the captured motion information to the motion capture workstation, which calculates the prop motion angle and displacement information in the X, Y, and Z directions in real time; Interactive props; the interactive props are equipped with a trigger switch for the actor to trigger the operation and generate a trigger signal. In addition, the props are equipped with infrared reflective markers for motion capture, which facilitate the motion capture camera to obtain prop motion information; An AR rendering workstation, wherein the AR rendering workstation is connected to the output end of the motion capture workstation and is used to receive the prop motion information; the input end of the AR rendering workstation is also connected to the output end of the camera and is used to overlay the AR rendering content on the image captured by the camera; the AR rendering workstation is also wirelessly connected to the output end of the interactive prop trigger switch and is used to receive the trigger operation information of the interactive prop; The UE rendering engine is installed in the XR rendering synthesis server and AR rendering workstation; A camera system, disposed toward the background display screen and connected to the AR rendering workstation, configured to record the content of the background display screen and the image of the person in front of the background display screen; The camera tracking system obtains real-time motion data of the camera lens and gimbal, and outputs it to the XR rendering and composition server and AR rendering workstation. When the camera moves, the scenes and elements in the XR rendering and composition server and AR rendering workstation are also rendered in real time based on the camera's motion data. Prompt feedback system, used to inform actors of the appearance, location, and interaction results of interactive objects, helping actors judge the progress and status of their participation in the program; The feedback system can be a software module of the AR system and the XR system. For sound feedback, the AR system and the XR system can be connected to the speakers through a sound card. The AR rendering workstation is also used to generate the final digital program video signal and output it as a PGM; The XR rendering synthesis server, the trigger switch of the interactive props, the AR rendering workstation, and the motion capture workstation are connected to the same local area network.

2. The digital program production system based on interactive mixed virtual reality technology according to claim 1 is characterized in that: The background display screen includes a display screen arranged on the facade and a display screen on the ground, wherein the display screen on the facade is used to display the surrounding environment in the XR scene, and the display screen on the ground is used to display the ground environment in the XR scene.

3. The digital program production system based on interactive mixed virtual reality technology according to claim 2 is characterized in that: The AR rendering content of the AR rendering workstation includes AR interactive game elements and background display screen extended AR elements. The AR rendering workstation superimposes the AR interactive game elements on the background display screen image captured by the camera, so that the background display screen content is only used as the background, and the AR content can be used as the foreground. At the same time, the background display screen extended AR elements are also superimposed on the outside of the background display screen image captured by the camera and seamlessly connected with the background display screen image, so as to realize the expansion of the scene content in the background display screen in a larger space, and the real scene outside the background display screen is also blocked by the AR content.

4. The digital program production system based on interactive mixed virtual reality technology according to claim 3 is characterized in that: The AR interactive game elements of the AR rendering workstation can be directly output on the background display screen captured by the camera system.

5. The digital program production system based on interactive mixed virtual reality technology according to any one of claim 3, characterized in that: After receiving the operation data of the interactive prop, the AR rendering workstation updates the AR interactive game element attributes according to the operation data of the interactive prop.

6. The digital program production system based on interactive mixed virtual reality technology according to claim 1 is characterized in that: The XR scene includes the on-screen scene displayed on the background display screen and the extended scene beyond the range of the background display screen, which are rendered and output by the AR rendering workstation and superimposed on the background display screen image captured by the camera, and together are output as the final digital program video signal.

7. The digital program production system based on interactive mixed virtual reality technology according to claim 1 is characterized in that: It also includes an audio assistance system, which includes a sound card installed in the XR rendering and synthesis server and the AR rendering workstation, and a speaker array set in the studio area for outputting a three-dimensional sound field. The sound card is connected to the speaker signal, and the pre-produced audio is bound to the interactive game elements in the three-dimensional scene. During the recording of the game show, the interactive game elements in the XR rendering and synthesis server trigger the playback of the audio according to the settings, and the speaker array generates a three-dimensional sound field, so that the actors can perceive the spatial position of the interactive game elements in the three-dimensional space through hearing, and thus perform corresponding interactive operations.

8. A digital program production method based on interactive mixed virtual reality technology, characterized in that: Based on the digital program production based on interactive mixed virtual reality technology according to claim 1, the method includes: The XR rendering and synthesis server receives the XR scene designed by the design workstation, and performs XR rendering synthesis based on the motion data of the camera system and the prop movement information and prop switch information of the motion capture workstation, and outputs the synthesized XR scene to the background display screen. The prop information is used to calculate whether the virtual elements in the XR scene or the scene will interact with the virtual props. The AR rendering workstation overlays the AR rendering content on the background display screen captured by the camera. The AR rendering content of the AR rendering workstation includes AR interactive game elements and background display screen extended AR elements. The AR rendering workstation also receives prop action information, interactive prop switch trigger data, and camera system motion data transmitted by the motion capture workstation. The information related to the props is used to render the virtual elements related to the physical props. The AR rendering workstation also generates extended content beyond the background display screen content, as well as virtual elements in the foreground, and then overlays the background display screen captured by the camera as the final digital program video signal and outputs it.

Images