Immersive interactive live broadcast construction method, system and medium based on XR technology

Abstract

The present invention discloses a method, system and medium for constructing an immersive interactive live broadcast based on XR technology. The method includes: setting a live three-dimensional imaging interval of a live virtual scene and a rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; calculating the pupil distance and viewing posture of the reference audience in the live virtual scene; rendering and generating a stereoscopic visual live picture of the live three-dimensional imaging interval of the live virtual scene, and sending the stereoscopic visual live picture to each audience; any audience receives the stereoscopic visual live picture, and / or the stereoscopic visual live picture is combined with the stereoscopic visual picture of other experience content in the audience experience space to form a stereoscopic visual picture that allows the audience to experience other content while watching the live virtual scene, and the picture is displayed to the audience; the audience interacts with the live virtual scene. The present invention can adapt to the pupil distance of different users, so that the live scene is correctly integrated into the user's experience space, and can adapt to all users with different pupil distances.

Description

Technical Field

[0001] The present invention relates to the field of XR technology, and in particular to a method, system, and medium for constructing an immersive interactive live broadcast based on XR technology. Background Art

[0002] Technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) are of great value in solving the problems of being unable to see, touch, or access what is difficult to achieve in vocational education and training. VR, AR, MR, and other technologies (abbreviated as XR, extended reality) are interconnected and integrated. Empowered by high-performance wireless network technologies such as 5G networks and WiFi 6, the storage, computing, and rendering services required by XR mobile terminals (5G mobile phones, headsets, etc.) can be put into the cloud. As a result, based on cloud services such as cloud storage, cloud computing, and cloud rendering, the computing, storage, and rendering capabilities of a single XR terminal can be unlimited.

[0003] Currently, when implementing the live teaching activity function of vocational education training, due to the different pupil distances of different users, huge computing resources are required to adapt to all users with different pupil distances and generate correct live broadcast images and interactive functions, which increases the difficulty of implementation. Summary of the Invention

[0004] The main purpose of the present invention is to provide an immersive interactive live broadcast construction method, system and storage medium based on XR technology, which can adapt to all users with different pupil distances with less computing resources and achieve correct live broadcast images and interactive functions.

[0005] To achieve the above objectives, the present invention proposes a method for constructing an immersive interactive live broadcast based on XR technology, the method comprising the following steps:

[0006] Step S10: setting a live 3D imaging interval of the live virtual scene and a rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system;

[0007] Step S20: Calculating the pupil distance and viewing posture of the reference audience in the live virtual scene according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system;

[0008] Step S30: Based on the pupil distance and viewing posture of the reference viewer in the live virtual scene, rendering and generating a stereoscopic live picture of the live 3D imaging interval of the live virtual scene, and sending the stereoscopic live picture to each viewer;

[0009] Step S40: Any audience receives a stereoscopic live broadcast picture, and / or the stereoscopic live broadcast picture is combined with the stereoscopic visual picture of other experience contents in the audience experience space to form a stereoscopic visual picture for watching the live broadcast of the virtual scene while experiencing other content, and displayed to the audience.

[0010] The stereoscopic live broadcast picture generated in step S30 has depth information. In step S40, for any viewer h k , h k The received stereoscopic live broadcast image and the stereoscopic image of other experience contents in the experience space are synthesized into h k While watching the live broadcast of virtual scenes, you can experience other contents in stereoscopic visual images. k When synthesizing the stereoscopic visual images generated by other content experiences, it is necessary to k The depth information of the field of view is used for occlusion calculation.

[0011] Among them, in the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system set in step S10, the live virtual scene s m The scaling factor from the coordinate system of the reference audience field of view coordinate system is λ m The step S20 calculates the benchmark audience in the live virtual scene s m The pupil distance in in The actual pupil distance of the benchmark audience is based on the live virtual scene s m The rotation, scaling and translation relationship between the coordinate system and the reference viewer's field of view coordinate system is used to calculate the reference viewer's position in the live virtual scene. The step S30 is based on the reference viewer's position in the live virtual scene s m The posture and pupil distance of is the left eye image of the stereoscopic vision picture, is the right eye image of the stereoscopic vision picture, is the corresponding depth image, according to the scaling factor λ m , calculated and The depth information under the reference audience field of view is the depth image and The step S40 of synthesizing the stereoscopic visual live broadcast image and the stereoscopic visual image of other experience contents in the audience experience space is specifically implemented as follows: any viewing scene s m Live audience h k , receiving scene s m Stereoscopic live broadcast picture, h k The left eye image of the stereoscopic visual experience screen generated by other experience contents in the immersive experience space is The right eye image is and In h k The depth information of the field of view is the depth map and h k The ratio of pupil distance to the reference audience is Scenes m Stereoscopic live broadcast and Depth image under the reference audience field of view and Convert to h k Depth image of the field of view Stereoscopic visual experience images generated by the live stereoscopic visual image and other experience content are based on the depth image Perform occlusion calculations to generate a composite image.

[0012] The step S10, setting the live 3D imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system, specifically includes: m Select the 3D bounding box Ω′ m , set the bounding box Ω′ m In scenes m The pose in the bounding box Ω′ m In scenes m Select a three-dimensional interval Ω m ,Ω m It's the scene m Live 3D imaging interval, set the bounding box Ω′ m In the reference audience's field of view, the pose and scale relationship is calculated to calculate the virtual scene s m Live 3D imaging interval Ω m Corresponding 3D display range in the reference audience field of view Set the bounding box Ω′ m In live virtual scenes m The pose and Ω′ in m The position and scaling relationship in the reference audience field of view, that is, the live virtual scene s is set m The rotation, scaling and translation relationship between the coordinate system and the reference audience field of view coordinate system.

[0013] In step S40, for any viewer h k , live virtual scenes m In h k The three-dimensional display range of the experience space is Generate h k When experiencing the stereoscopic visual images of other experience contents in the experience space, h kOther experience contents in the experience space are in the 3D display area The content is not imaged.

[0014] The specific implementation of step S40 to calculate the three-dimensional imaging interval of the live virtual scene in the three-dimensional display interval of any audience experience space is: the live virtual scene s m The 3D display range in the reference audience field of view is Ω user , by any viewer h k The ratio of pupil distance to the reference audience pupil distance Calculate s m Three-dimensional imaging interval Ω m In h k The three-dimensional display range in the field of view is

[0015] Wherein, after step S40, there is step S50: any viewer h k In its experience space, the virtual scene of the live broadcast m Interact and generate interactive operation command A in the experience space k , A k Converted into live virtual scenes m The interactive operation command A′ k , put A′ k Send to scenes m , scenes m For A′ k Respond.

[0016] Wherein, the step S50 specifically includes:

[0017] Step S501: Any viewer h k Perform interactive operations in your own experience space to generate audience h k Interactive operation command A in the experience space k , command A k The pose parameters included are the viewer h k Experience the posture in the spatial coordinate system and judge A k Is it a live virtual scene? m If it is a live virtual scene m If the interaction is successful, then go to step S502, otherwise h k Other content in the experience space responds to interactive operation commands A k , do not proceed to step S502;

[0018] Step S502: According to the audience h k Experience space coordinate system and h k Rotation and translation relationship of the field of view coordinate system, interactive operation command A k The pose parameters are converted into hk Pose parameters in the field of view coordinate system;

[0019] Step S503: According to the audience h k Pupillary distance d k Interpupillary distance to the reference audience The ratio of interactive operation command A k The pose parameters are obtained from h k The pose parameters in the field of view coordinate system are converted into pose parameters in the reference audience field of view coordinate system; Step S504: according to the reference audience field of view coordinate system and the live virtual scene s m Coordinate system rotation, scaling and translation relationship, interactive operation command A k The pose parameters are transformed from the reference audience field of view coordinate system to the live virtual scene s m Pose parameters in the coordinate system, generating interactive operation commands A′ k ;

[0020] Step S505: Interactive operation command A′ k , sent to the virtual live scene s m , s m For A′ k response.

[0021] The present invention also proposes an immersive interactive live broadcast construction system based on XR technology, which includes: an immersive interactive live broadcast management server, multiple XR application servers, and a storage server. The user's XR terminal accesses the management server and all XR application servers through a wireless network. The management server is deployed with an immersive interactive live broadcast management control center, and the storage server is deployed with an XR resource library:

[0022] The XR resource library is used to store XR application resources;

[0023] The immersive interactive live broadcast management control center is used to receive virtual scene experience applications from XR terminals, instruct the XR application server to allocate virtual scene instances, and assist the XR terminals in establishing communication connections with the virtual scene instances; receive user live broadcast sharing applications, instruct the XR application server to allocate virtual scene instances for live broadcast; set the live 3D imaging interval of the live virtual scene; set the rotation, scaling, and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; calculate the reference audience's position and pupil distance in the live virtual scene and send them to the live virtual scene;

[0024] The XR application server is used to receive the virtual scene instance allocation instruction sent by the immersive interactive live broadcast management control center, and allocate the corresponding virtual scene instance according to the scene ID of the virtual scene in the scene instance allocation instruction. When the existing scene instance does not have sufficient computing resources, the corresponding XR application and related resource package are called to generate a virtual scene instance, and the newly generated scene instance establishes a communication connection with the original scene instance to synchronize the scene status.

[0025] The XR application server is further configured to receive a scene instance destruction instruction sent by the immersive interactive live broadcast management control center, and destroy the virtual scene instance according to the instruction;

[0026] The XR terminal is used to send a virtual scene experience application to the immersive interactive live broadcast management and control center, establish a communication connection with the virtual scene instance assigned to the XR terminal user; collect the XR terminal's posture parameters and interactive operation information and send them to the virtual scene instance assigned to the XR terminal user; receive the immersive experience picture sent by the virtual scene instance; apply to the immersive interactive live broadcast management and control center for watching live broadcasts, establish a communication connection with the live broadcast scene, and receive the live broadcast picture sent by the live broadcast scene; determine whether the user's interactive operation is an interaction with the live broadcast scene. When interacting with the live broadcast scene, convert the posture parameters of the interactive operation command into the live broadcast scene coordinate system, and send the converted interactive operation command to the live broadcast scene; receive the live stereoscopic visual picture sent by the live virtual scene; synthesize the live stereoscopic visual picture and the stereoscopic visual experience picture of other contents in the experience space to generate a complete interactive experience picture for watching the virtual scene live broadcast while experiencing other content, and display it to the user.

[0027] The present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the computer program executes the steps of the above-mentioned method for constructing an immersive interactive live broadcast based on XR technology.

[0028] The present invention proposes an immersive interactive live broadcast construction method, system and storage medium based on XR technology, which sets the live three-dimensional imaging range of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system, calculates the pupil distance and viewing posture of the reference audience in the live virtual scene; based on the pupil distance and viewing posture of the reference audience in the live virtual scene, renders and generates a stereoscopic visual live picture of the live three-dimensional imaging range of the live virtual scene, and sends the stereoscopic visual live picture to each audience; any audience receives the stereoscopic visual live picture, and / or the stereoscopic visual live picture is synthesized with the stereoscopic visual picture of other experience content in the audience experience space into a stereoscopic visual picture for watching the live virtual scene while experiencing other content, and displays it to the audience.

[0029] Based on this immersive interactive live broadcast system, immersive experience activities of single or multiple virtual scenes built based on XR technology can be shared live. Viewers watching the immersive interactive live broadcast can participate in the interaction of the live virtual scene while watching, and can also immerse themselves in other content in parallel. Because the present invention specifies a baseline user pupil distance and defines a baseline field of view based on this baseline pupil distance, the present invention only needs to render and generate a stereoscopic visual image of this baseline field of view. Through algorithmic processing, it can adapt it to the pupil distance of different users, so that the live scene can be correctly integrated into the user's experience space, and the function of correctly interacting with the live virtual scene can be provided, thereby using fewer computing resources to adapt to all users with different pupil distances.

[0030] This system can be applied to conventional live broadcast activities and can also be applied to teaching. For example, in the classroom, the teacher performs a virtual practical demonstration in a virtual training scene constructed by XR technology. This system can share the teacher's virtual practical live broadcast with all students in the classroom. Students can watch the teacher's practical three-dimensional picture in their own experience space, interact with the teacher's virtual practice, and operate in parallel in their own virtual practice environment, so as to "learn by doing" and "learn by doing"; in the classroom, the teacher can also specify a student's virtual practice process to be shared live for everyone's evaluation and reference; it can also be applied to practical competitions. Each person or group participating in the competition has an independent virtual practice scene. They can see the virtual practice progress of other people or groups in real time and three-dimensionally through live broadcast, thereby intuitively forming a competition comparison. Therefore, the immersive interactive live broadcast construction method and system based on XR technology of the present invention have high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the method for constructing an immersive interactive live broadcast based on XR technology of the present invention.

[0032] Figure 2 Schematic diagram of the scenario and scenario example of the present invention.

[0033] Figure 3 Schematic diagram of the user experience space coordinate system of the present invention.

[0034] Figure 4 Schematic diagram of the user field of view coordinate system of the present invention.

[0035] Figure 5 Schematic diagram of pupil distance and stereoscopic vision according to the present invention.

[0036] Figure 6 A schematic diagram of setting a live three-dimensional imaging interval for the present invention.

[0037] Figure 7 A schematic diagram of the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system is set for the present invention.

[0038] Figure 8 This is a schematic diagram of the hardware structure of the immersive interactive live broadcast system of the present invention.

[0039] Figure 9 This is a schematic diagram of the software structure of the immersive interactive live broadcast system of the present invention. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] Reference Figure 1 The present invention proposes a method for constructing an immersive interactive live broadcast based on XR technology, the method comprising the following steps:

[0042] Step S10, setting a live 3D imaging interval of the live virtual scene and a rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system;

[0043] Step S20, calculating the pupil distance and viewing posture of the reference viewer in the live virtual scene according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference viewer field of view coordinate system;

[0044] Step S30, based on the pupil distance and viewing posture of the reference viewer in the live virtual scene, rendering and generating a stereoscopic live picture of the live 3D imaging interval of the live virtual scene, and sending the stereoscopic live picture to each viewer;

[0045] Step S40: any viewer receives a stereoscopic live broadcast image, and / or the stereoscopic live broadcast image is combined with a stereoscopic image of other experience content in the viewer experience space to form a stereoscopic image for viewing the live broadcast of the virtual scene while experiencing other content, and the image is displayed to the viewer;

[0046] Step S50: The audience interacts with the live virtual scene.

[0047] The stereoscopic live broadcast picture generated in step S30 has depth information. In step S40, for any viewer h k , h k The received stereoscopic live broadcast image and the stereoscopic image of other experience contents in the experience space are synthesized into h k While watching the live broadcast of virtual scenes, you can experience other contents in stereoscopic visual images. k When synthesizing the stereoscopic visual images generated by other content experiences, it is necessary to k The depth information of the field of view is used for occlusion calculation.

[0048] Among them, in the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system set in step S10, the live virtual scene s m The scaling factor from the coordinate system of the reference audience field of view coordinate system is λ m The step S20 calculates the benchmark audience in the live virtual scene s m The pupil distance in in The actual pupil distance of the benchmark audience is based on the live virtual scene s m The rotation, scaling and translation relationship between the coordinate system and the reference viewer's field of view coordinate system is used to calculate the reference viewer's position in the live virtual scene. The step S30 is based on the reference viewer's position in the live virtual scene s m The posture and pupil distance of is the left eye image of the stereoscopic vision picture, is the right eye image of the stereoscopic vision picture, is the corresponding depth image, according to the scaling factor λ m , calculated and The depth information under the reference audience field of view is the depth image and The step S40 of synthesizing the stereoscopic visual live broadcast image and the stereoscopic visual image of other experience contents in the audience experience space is specifically implemented as follows: any viewing scene s m Live audience h k , receiving scene s m Stereoscopic live broadcast picture, h k The left eye image of the stereoscopic visual experience screen generated by other experience contents in the immersive experience space is The right eye image is and In h k The depth information of the field of view is the depth map and h k The ratio of pupil distance to the reference audience is Scenes m Stereoscopic live broadcast and Depth image under the reference audience field of view and Convert to h k Depth image of the field of view Stereoscopic visual experience images generated by the live stereoscopic visual image and other experience content are based on the depth image Perform occlusion calculations to generate a composite image.

[0049] The step S10, setting the live 3D imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system, specifically includes: m Select the 3D bounding box Ω′ m , set the bounding box Ω′ m In scenes m The pose in the bounding box Ω′ m In scenes m Select a three-dimensional interval Ω m ,Ω m It's the scene m Live 3D imaging interval, set the bounding box Ω′ m In the reference audience's field of view, the pose and scale relationship is calculated to calculate the virtual scene s m Live 3D imaging interval Ω m Corresponding 3D display range in the reference audience field of view Set the bounding box Ω′ m In live virtual scenes m The pose and Ω′ in m The position and scaling relationship in the reference audience field of view, that is, the live virtual scene s is set m The rotation, scaling and translation relationship between the coordinate system and the reference audience field of view coordinate system.

[0050] In step S40, for any viewer h k , live virtual scenes m In h k The three-dimensional display range of the experience space is Generate h k When experiencing the stereoscopic visual images of other experience contents in the experience space, h k Other experience contents in the experience space are in the 3D display area The content is not imaged.

[0051] The specific implementation of step S40 to calculate the three-dimensional imaging interval of the live virtual scene in the three-dimensional display interval of any audience experience space is: the live virtual scene s m The 3D display range in the reference audience field of view is Ω user , by any viewer h k The ratio of pupil distance to the reference audience pupil distance Calculate s m Three-dimensional imaging interval Ω m In h k The three-dimensional display range in the field of view is

[0052] After step S40, there is step S50: any viewer h k In its experience space, the virtual scene of the live broadcast m Interact and generate interactive operation command A in the experience space k , A k Converted into live virtual scenes m The interactive operation command A′ k , put A′ k Send to scenes m , scenes m For A′ k Respond.

[0053] Wherein, the step S50 specifically includes:

[0054] Step S501: Any viewer h k Perform interactive operations in your own experience space to generate audience h k Interactive operation command A in the experience space k , command A k The pose parameters included are the viewer h k Experience the posture in the spatial coordinate system and judge A k Is it a live virtual scene? m If it is a live virtual scene m If the interaction is successful, then go to step S502, otherwise h k Other content in the experience space responds to interactive operation commands A k , do not proceed to step S502;

[0055] Step S502: According to the audience h k Experience space coordinate system and h k Rotation and translation relationship of the field of view coordinate system, interactive operation command A k The pose parameters are converted into h k Pose parameters in the field of view coordinate system;

[0056] Step S503: According to the audience hk pupil distance d k Interpupillary distance to the reference audience The ratio of interactive operation command A k The pose parameters are obtained from h k The pose parameters in the field of view coordinate system are converted into pose parameters in the reference audience field of view coordinate system; Step S504: according to the reference audience field of view coordinate system and the live virtual scene s m Coordinate system rotation, scaling and translation relationship, interactive operation command A k The pose parameters are transformed from the reference audience field of view coordinate system to the live virtual scene s m Pose parameters in the coordinate system, generating interactive operation commands A′ k ;

[0057] Step S505: Interactive operation command A′ k , sent to the virtual live scene s m , s m For A′ k response.

[0058] Therefore, by setting the live three-dimensional imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system, the pupil distance and viewing posture of the reference audience in the live virtual scene are calculated; based on the pupil distance and viewing posture of the reference audience in the live virtual scene, a stereoscopic visual live picture of the live three-dimensional imaging interval of the virtual scene is rendered and generated, and the stereoscopic visual live picture is sent to each audience; any audience receives the stereoscopic visual live picture, and / or the stereoscopic visual live picture is synthesized with the stereoscopic visual picture of other experience content in the audience experience space into a stereoscopic visual picture for watching the live virtual scene while experiencing other content, and displayed to the audience; any audience can also interact with the live virtual scene.

[0059] Based on this immersive interactive live broadcast system, immersive experience activities of single or multiple virtual scenes built based on XR technology can be shared live. Viewers watching the immersive interactive live broadcast can participate in the interaction of the live virtual scene while watching, and can also immerse themselves in other content in parallel. Because the present invention specifies a baseline user pupil distance and defines a baseline field of view based on this baseline pupil distance, the present invention only needs to render and generate a stereoscopic visual image of this baseline field of view. Through algorithmic processing, it can adapt it to the pupil distance of different users, so that the live scene can be correctly integrated into the user's experience space, and the function of correctly interacting with the live virtual scene can be provided, thereby using fewer computing resources to adapt to all users with different pupil distances.

[0060] In addition, the present invention also proposes an immersive interactive live broadcast construction system based on XR technology, which includes: an immersive interactive live broadcast management server, multiple XR application servers, and a storage server. The user's XR terminal accesses the management server and all XR application servers through a wireless network. The management server is deployed with an immersive interactive live broadcast management control center, and the storage server is deployed with an XR resource library:

[0061] The XR resource library is used to store XR application resources;

[0062] The immersive interactive live broadcast management control center is used to receive virtual scene experience applications from XR terminals, instruct the XR application server to allocate virtual scene instances, and assist the XR terminals in establishing communication connections with the virtual scene instances; receive user live broadcast sharing applications, instruct the XR application server to allocate virtual scene instances for live broadcast; set the live 3D imaging interval of the live virtual scene; set the rotation, scaling, and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; calculate the reference audience's position and pupil distance in the live virtual scene and send them to the live virtual scene;

[0063] The XR application server is used to receive the virtual scene instance allocation instruction sent by the immersive interactive live broadcast management control center, and allocate the corresponding virtual scene instance according to the scene ID of the virtual scene in the scene instance allocation instruction. When the existing scene instance does not have sufficient computing resources, the corresponding XR application and related resource package are called to generate a virtual scene instance, and the newly generated scene instance establishes a communication connection with the original scene instance to synchronize the scene status.

[0064] The XR application server is further configured to receive a scene instance destruction instruction sent by the immersive interactive live broadcast management control center, and destroy the virtual scene instance according to the instruction;

[0065] The XR terminal is used to send a virtual scene experience application to the immersive interactive live broadcast management and control center, establish a communication connection with the virtual scene instance assigned to the XR terminal user; collect the XR terminal's posture parameters and interactive operation information and send them to the virtual scene instance assigned to the XR terminal user; receive the immersive experience picture sent by the virtual scene instance; apply to the immersive interactive live broadcast management and control center for watching live broadcasts, establish a communication connection with the live broadcast scene, and receive the live broadcast picture sent by the live broadcast scene; determine whether the user's interactive operation is an interaction with the live broadcast scene. When interacting with the live broadcast scene, convert the posture parameters of the interactive operation command into the live broadcast scene coordinate system, and send the converted interactive operation command to the live broadcast scene; receive the live stereoscopic visual picture sent by the live virtual scene; synthesize the live stereoscopic visual picture and the stereoscopic visual experience picture of other contents in the experience space to generate a complete interactive experience picture for watching the virtual scene live broadcast while experiencing other content, and display it to the user.

[0066] Compared to existing technologies, this system, based on XR technology, allows for live streaming and sharing of immersive experiences of single or multiple virtual scenes. Viewers watching these immersive interactive live broadcasts can participate in the interaction within the live virtual scenes while also engaging in other immersive experiences simultaneously. Because the present invention specifies a baseline user interpupillary distance (IPD), which defines a baseline field of view (FOV), the present invention simply renders and generates a stereoscopic image of this baseline FOV. Through algorithmic processing, it can adapt to the interpupillary distances of different users, generating the correct live images and interactive features. This allows the system to accommodate users of all interpupillary distances using fewer computing resources.

[0067] This system can be applied to conventional live broadcast activities and can also be applied to teaching. For example, in the classroom, the teacher performs a virtual practical demonstration in a virtual training scene constructed by XR technology. This system can share the teacher's virtual practical live broadcast with all students in the classroom. Students can watch the teacher's practical three-dimensional picture in their own immersive experience space, interact with the teacher's virtual practice, and operate in parallel in their own virtual practice environment, so as to "learn by doing" and "learn by doing"; in the classroom, the teacher can also specify a student's virtual practice process to be shared live for everyone's evaluation and reference; it can also be applied to practical competitions. Each person or group participating in the competition has an independent virtual practice scene. They can see the virtual practice progress of other people or groups in real time and three-dimensionally through live broadcast, thereby intuitively forming a competition comparison. Therefore, the immersive interactive live broadcast construction method and system based on XR technology of the present invention have high application value.

[0068] The present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the computer program executes the steps of the above-mentioned method for constructing an immersive interactive live broadcast based on XR technology.

[0069] The following is a detailed explanation of the implementation principle of the immersive interactive live broadcast construction method based on XR technology of the present invention:

[0070] The technical terms involved in the present invention include:

[0071] "Scene" and "scene instance" are two closely related terms. A scene defines the objects, object states, object logic, and the logic of interaction between objects within a three-dimensional space. A scene instance is a program process that the system uses to run in real time according to the scene definition, using computing resources such as the computer processor, memory, and graphics card. This program process calculates the state of each object within the scene and renders the image in real time. When a single scene has multiple users experiencing it simultaneously, if the computing resources available to a single scene instance are insufficient to generate the experience images for all users in real time, multiple scene instances need to be generated for this scene and allocated to each user. These scene instances synchronize the state of the objects within the scene, and each scene instance generates an experience image for its corresponding user in real time, allowing all users to experience the scene together. Similarly, if a scene instance does not have sufficient computing resources to generate live images in addition to directly generating an immersive experience for the user, additional scene instances need to be added to the scene. These additional scene instances must synchronize their scene states with the other instances of the scene.

[0072] by Figure 2 For example, there is a virtual scene 1. User A and User B collaborate on this virtual scene experience. When a single scene instance cannot generate an immersive experience for both users at the same time, the system generates two scene instances, namely Virtual Scene 1 Instance 1 and Virtual Scene 1 Instance 2. The two instances are connected through communication to exchange scene state information and synchronize the scene states. Instance 1 generates an immersive experience screen for User A, and Instance 2 generates an immersive experience screen for User B. This enables User A and User B to collaboratively experience Virtual Scene 1. Further, Virtual Scene 1 is live-streamed and shared with other users. When the existing two scene instances do not have sufficient computing resources to regenerate the real-time live screen, the system needs to regenerate Instance 3 of Virtual Scene 1. The new instance exchanges scene state parameters with Instance 2, thereby synchronizing the scene states with Instances 1 and 2. The live screen generated by Instance 3 is sent to Users C, D, and E.

[0073] User experience space coordinate system. The mainstream XR headsets on the market all have built-in positioning functions, which can automatically establish a coordinate system in the user's environment, such as Figure 3 As shown, Figure 3This is a schematic diagram of the user experience space coordinate system of the present invention. This coordinate system defines the positive y-axis in the vertically upward direction on the horizontal plane, and the x-axis and z-axis on the horizontal plane. The user experience space coordinate system is the coordinate system automatically established by the XR headset.

[0074] The user field of view coordinate system is the coordinate system defined by the human eye field of view. Figure 4 For example, on the line connecting the center points of the two eyes of the human eye, the midpoint between the center points of the two eyes is used as the origin of the coordinate system. The direction of the line connecting the two eyes from the left eye to the right eye is defined as the positive direction of the x-axis, the direction of attention is defined as the positive direction of the z-axis, and the positive direction of the y-axis is defined as perpendicular to the x-axis and the z-axis and satisfies the left-hand coordinate system.

[0075] Reference pupil distance and reference audience. The interpupillary distance between different people is different. In the present invention, a pupil distance value is specified. The pupil distance is the reference value, The user with the reference pupil distance is set or assumed to be the reference audience.

[0076] Interpupillary distance and stereoscopic vision

[0077] When the human eye looks at an object, the left eye and the right eye form an image of the object respectively. The image position of the same object point in the left eye and the right eye are different. This difference is parallax. Parallax allows the human eye to correctly perceive the three-dimensional information of the object point. Figure 5 As shown in Figure (a), the pupil distance is O L For the left eye of man, O R The optical center of the right eye is the object point C, and the image position of the left eye is point A, and the image position of the right eye is point B. Figure 5 As shown in Figure (b), the pupil distance is d k When the object point remains at the same position in the human eye, it is still at point A in the left eye and point B in the right eye. The perceived three-dimensional position of the object point is Triangle O L O R C and triangle Similar but not equal, with a size ratio of Therefore, when the same left-eye and right-eye stereoscopic images are adapted to different human pupil distances, people with different pupil distances perceive different stereoscopic visual information.

[0078] Rotation, scaling, translation matrix and conversion between rotation matrix and Euler angle

[0079] In this embodiment, the left-handed coordinate system is used, and any two coordinate systems Ψ to The relationship between rotation, scaling and translation is: the rotation angles around the axis in the order of z, x, and y are θ, β, and α respectively, and the translation along the x, y, and z axes are t respectively. x , ty , t z , the scaling factor of the three axes z, x, and y is λ, and the coordinate value of any point p in the center is (c x c y c z ),exist The coordinate value is (w x w y w z ), from Ψ to The coordinate value conversion relationship is:

[0080] [w x w y w z ] T =λR·[c x c y c z ] T +T (1) Where R is the rotation matrix, T is the translation matrix, and the conversion relationship between the rotation angle and the rotation matrix R is as follows:

[0081] The conversion relationship between the translation amount and the translation matrix T is as follows:

[0082] T=[t x t y t z ] T (3)

[0083] The rotation, scaling and translation matrices are:

[0084]

[0085] have:

[0086] [w x w y w z 1] T =H[c x c y c z 1] T (5) The rotation matrix R is converted into the rotation angle as follows:

[0087] Let the known rotation, scaling and translation matrix but:

[0088] When h 13 、h 33 、h 21 、h 22 When it is not 0 at the same time,

[0089]

[0090] When h 13 、h 33 、h 21 、h 22 When both are 0, If β is but:

[0091] α-θ=atan2(h 12 ,h 11 ) (7)

[0092] Among them, α can take any angle value, as long as θ satisfies the above equation.

[0093] If β is but:

[0094] α+θ=atan2(-h 12 ,h 11 ) (8)

[0095] Among them, α can take any angle value, as long as θ satisfies the above equation.

[0096] The implementation scenarios of the immersive interactive live broadcast construction method based on XR technology of the present invention are as follows:

[0097] There are multiple users participating in the immersive experience activity based on XR technology. The user set is P = {h0 h1…h n-1}, any user h k The pupil distance is expressed as d k Indicates that the pupil distance of each user is different. Among them, there are users h m Virtual scenes m For an immersive experience, h m Put the pair m The immersive experience process is shared in real time, and any other user in P can watch h m to s m Experience the three-dimensional live screen, you can m You can also interact with other virtual scenes and experience them immersively. Figure 1 As shown, this immersive interactive live broadcast construction method specifically includes the following steps:

[0098] (1) Setting the live 3D imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system

[0099] The live three-dimensional imaging interval and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system can be set in a forward or reverse direction. The forward setting is to manually or systematically select a three-dimensional bounding box, set the position and posture of this bounding box in the live virtual scene, and the three-dimensional interval of the live virtual scene selected by the bounding box is the live three-dimensional imaging interval of the live virtual scene. The scene content of this imaging interval will be presented to the audience. Then set the position and scaling factor of the three-dimensional bounding box in the reference audience field of view. The three-dimensional interval selected by this three-dimensional bounding box in the reference audience field of view is the three-dimensional display interval of the live virtual scene in the reference audience field of view. The rotation and translation relationship between the set three-dimensional bounding box coordinate system and the live virtual scene, as well as the three-dimensional bounding box coordinate system and the reference audience field of view coordinate system are determined. The rotation, scaling and translation relationship of the reference system determines the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; the reverse setting is to manually or systematically select a three-dimensional bounding box, set the position and scaling coefficient of this bounding box in the reference audience field of view, and determine the three-dimensional display range of the live virtual scene in the reference audience field of view, set the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system, and the live three-dimensional imaging range in the live virtual scene can be calculated based on the set three-dimensional display range and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system. The reverse setting method can be easily obtained according to the forward setting method. The following only introduces the forward setting method in detail:

[0100] ① Set the live 3D imaging range of the live virtual scene

[0101] Select a 3D bounding box Ω′ m The shape of the three-dimensional bounding box can be arbitrary, and the bounding box is set in the virtual scene s m The pose in s. m The 3D interval selected in the box is used as the live 3D imaging interval for the virtual scene. During the live broadcast, only the content within the live 3D imaging interval will be shared with other users. m The pose in the video can be adjusted as needed during the live broadcast. This 3D imaging range only restricts the live broadcast sharing and does not affect the video itself. m Users who are immersed in the experience m There are no restrictions. Details are as follows.

[0102] Give virtual training scenes m Select the 3D bounding box Ω′ m Ω′ m Define the coordinate system Ψ′ m ,Ψ m For virtual scenes m The coordinate system of Ω′. mIn s m The pose in the coordinate system Ψ′ is set m To m The rotation and translation relationship of Ψ′ m With Ψ m In the rotation relationship of the present invention, the real-time example defines Ψ′ m Can only go around m Rotate the y-axis, let Ψ′ m To m The rotation amount in the rotation-translation relationship can be expressed as a rotation angle around the y-axis of The zoom factor is 1 and the translation amount is Let the coordinate system Ψ′ m Any point in In the coordinate system Ψ m The median coordinate value is (c x c y c z ) indicates that, according to formula (4), we can get Ψ′ m To m The rotation, scaling and translation matrix H m ,in:

[0103]

[0104] but:

[0105]

[0106] For the three-dimensional bounding box Ω′ m Any point in the coordinate system Ψ′ m Substitute the coordinate values ​​under Ψ into formula (10) to calculate its m The coordinate values ​​in Ω′ are determined m In s m The three-dimensional imaging interval Ω defined in m .by Figure 6 For example, select a cylinder as the bounding box, set the position of the cylinder in the virtual scene, and thus frame the area in the virtual scene that needs to be shared live.

[0107] The 3D bounding box can also be infinitely large, so that all content in the virtual scene will be within the live sharing range.

[0108] ② Set the 3D display range of the virtual scene live 3D imaging range in the reference audience field of view

[0109] Set the virtual scene live 3D imaging interval Ω m The three-dimensional display interval in the user's field of view is as follows: set the three-dimensional bounding box Ω′ m Based on the position and scaling relationship of the reference audience’s field of view, the live three-dimensional imaging interval Ω of the virtual scene can be calculated.m The corresponding display interval in the reference audience field of view is as follows. Let the reference audience field of view coordinate system be Ψ user Indicates that setting a three-dimensional bounding box Ω′ m The position and scaling relationship in the reference audience field of view, that is, setting the three-dimensional bounding box Ω′ m Cartesian coordinate system Ψ′ m To the reference audience field coordinate system Ψ user The rotation, scaling and translation relationship of m To user The scaling of is the same scaling of each axis. Set Ψ′ m To user The rotation amount in the rotation-scaling-translation relationship is expressed as [α m β m θ m ], the translation amount is The scaling factor is denoted as λ m According to formula (4), we can get the coordinate system Ψ′ m To user The rotation, scaling and translation matrix Let Ω′ m Any point in the coordinate system Ψ′ m The lower coordinate value is In the reference audience field coordinate system Ψ user The median coordinate value is (w x w y w z ) means:

[0110]

[0111] Ω′ m Substituting any point into formula (11), Ω′ can be calculated m 3D display range in the reference audience field of view It is also the live 3D imaging interval Ω m Corresponding 3D display range in the reference audience field of view

[0112] Set the three-dimensional bounding box Ω′ m Coordinate system to live virtual scene m The rotation and translation transformation relationship of the coordinate system, and the three-dimensional bounding box Ω′ m Coordinate system to reference audience field of view coordinate system Ψ user The rotation, scaling and translation relationship of the live virtual scene coordinate system and the reference audience field of view coordinate system is set synchronously. The rotation, scaling and translation relationship of the live virtual scene coordinate system to the reference audience field of view coordinate system can be expressed by the rotation, scaling and translation matrix Conversely, the rotation, scaling and translation matrix from the reference audience field of view coordinate system to the live virtual scene coordinate system is like Figure 7 As shown, Figure 7 This is a schematic diagram of the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system in the reference user field of view of the present invention.

[0113] (2) Calculate the viewing posture and pupil distance of the benchmark audience in the live virtual scene

[0114] In the embodiment of the present invention, the reference audience is in the reference audience field coordinate system Ψ user The position coordinate is (000), and the attitude angle is (000). According to the reference audience field coordinate system Ψ user To live virtual scenes m Coordinate system Ψ m The rotation, scaling and translation matrix Any point in Ψ user The median coordinate value is (w x w y w z ), in m The coordinate value is (c x c y c z ),but:

[0115]

[0116] Put the reference audience in the reference audience field coordinate system Ψ user Substituting the position coordinate (000) into formula (12) can calculate the reference audience in scene s m Coordinate system Ψ m The position coordinate W m Since the reference audience is in the reference audience field coordinate system Ψ user The middle attitude angle is (000), then according to equations (6), (7), and (8), the rotation, scaling, and translation matrices are The converted three-dimensional rotation angle vector Q around the axes z, x, and y m , which is the reference audience field of view in scene s m Coordinate system Ψ m The attitude angle under Ψ is obtained. m Lower pose value Because of the live virtual scene m The live 3D imaging interval is scaled and displayed in the field of view of the reference audience, and the pupil distance used by the live virtual scene to generate the live stereoscopic visual image for the reference audience is also scaled. m The pupil distance used to generate the live stereoscopic vision picture is

[0117] (3) Rendering and generating stereoscopic visual images of the three-dimensional imaging interval of the live virtual scene

[0118] Pose value in the field of view and pupil distance Render the scene in real time for the audience m Imaging interval Ω m Binocular stereo vision image and corresponding depth image. is the left eye image of the stereoscopic vision picture, is the right eye image of the stereoscopic vision picture, is the corresponding depth image, and then the depth image needs to be scaled according to the scaling factor λ m Converted into a depth image under the reference audience field of view, For any pixel in the left eye image Its depth value is For any pixel of the right eye image Its depth value is Send stereoscopic visual images and depth images to each viewer.

[0119] (4) The audience receives the stereoscopic visual live picture, and / or the stereoscopic visual live picture is combined with the stereoscopic visual picture of other experience contents in the audience experience space to form a stereoscopic visual picture in which the audience can experience other contents while watching the live broadcast of the virtual scene, and display it to the audience.

[0120] Any audience h k Receive s in real time m When the left and right eye pictures of the live screen are displayed to the left and right eyes respectively, the audience h k You will directly get the scene m Three-dimensional imaging interval Ω m However, when the audience is watching the live broadcast while also experiencing other immersive content, the live stereoscopic visual image needs to be synthesized with the stereoscopic visual experience image of other content to generate the audience h k The complete stereoscopic visual experience picture integrates the live virtual scene into the audience's immersive experience space. When synthesizing the live stereoscopic picture with the stereoscopic visual experience picture of other content, it is necessary to perform occlusion calculation between images. Only pixels on the same line of sight of the user will be occluded. In the embodiment of the present invention, it is required to use the same field of view angle and resolution when rendering and generating all stereoscopic visual images and depth images. Therefore, the pixels with the same image coordinates between multiple left-eye or right-eye pictures are on the same line of sight of the user. The stereoscopic visual live picture and the stereoscopic visual picture of other experience content in the audience experience space are synthesized into a stereoscopic visual picture that allows users to experience other content while watching the live broadcast of the virtual scene. The specific implementation is as follows.

[0121] When generating a stereoscopic visual image of other experience contents in the audience's immersive experience space, the three-dimensional display interval of the live virtual scene in the audience's experience space is calculated. The content of other experience contents in the audience's experience space in this three-dimensional interval is not imaged. Through step (1), the display interval Ω of the live virtual scene under the reference audience field of view has been obtained. user , for any viewer user h k , whose pupil distance is d k ,Due to the difference in pupil distance, live virtual scene s m In user h k The display interval in the field of view is not Ω user , let Ω user Any point (u x u y u z ), which is in h k The corresponding coordinate value in the field of view coordinate system is but According to this mapping relationship, Ω user Mapped to user h k The display range of the live scene under the user's field of view can be obtained by according to Then the three-dimensional display range of the live virtual scene in any audience field of view is

[0122] For any user h k , while watching scenes m Live broadcast, while also experiencing other content, such as scene s k For immersive experience. It has been calculated that the live virtual scene is k The three-dimensional display range in the field of view is The audience experience space and other experience contents are in the three-dimensional area The content in the image is not imaged, but the virtual interactive tool can be retained in this interval for imaging. m Outside of live streaming, users h k The left eye image of the experience screen generated by other content in the experience space is The right eye image is In h k The corresponding depth value of the field of view is and Received scenes m Live stereoscopic images and Need and Synthesized into user h k The user can experience the complete stereoscopic visual picture while watching the live broadcast of the virtual scene and experiencing other contents in the experience space. Occlusion calculation is required when synthesizing the picture. k Pupillary distance dk With reference pupil distance The difference between user h k Perceived and Depth information and depth image under the reference audience field of view There is a difference, user h k Perceived and The depth image is Let the left eye picture of the synthesized experience picture be The right eye picture is The corresponding depth image is initialization Traverse any pixel in the image, if but Otherwise, no processing is done; similarly, if but Otherwise, no processing is done. After traversing all pixels in the image according to the above calculation, the picture synthesis is completed. The synthesized picture is displayed to the audience h k look.

[0123] (5) Audiences interact with the live broadcast virtual scene

[0124] Any audience member interacts with the live virtual scene in his or her own immersive experience space, generates interactive commands in the user experience space coordinate system, correctly converts the posture parameters of the interactive commands into the live virtual scene coordinate system, thereby converting the interactive commands into interactive commands in the virtual live scene, and sending the converted interactive commands to the live virtual scene. The live virtual scene responds to this interactive command, completing the interaction between the audience and the live virtual scene.

[0125] Any audience user h k , h k The immersive experience space coordinate system is used Indicates. k In its experience space, based on s m Live screen, for s m Perform interactive operations to generate user h k Interactive operation command A of experience space k , interactive operation command A k Contains coordinate systems The pose parameters under , where the position parameter is The attitude angle parameter is Set the interactive operation command A k Convert to scenes m The interactive operation command A′ k , you need to put posture A k The pose parameters are converted into scene s m Coordinate system Ψm The pose parameters under the real-time transmission of the interactive operation command A′ after the pose parameters are converted k To scene m In the scene m For A′ k To respond, the audience can interact with the live virtual scene.

[0126] The specific implementation of the audience's interaction with the live virtual scene is as follows:

[0127] ① Determine whether the interactive operation is an interaction with the live virtual scene

[0128] Live virtual scene in the audience h k The display range in the field of view is Calculate the three-dimensional display range of the live virtual scene in any audience field of view by step 4 When the interactive operation command A k Positional parameters Belong to the interval When , it is judged that this command belongs to the live virtual scene s m If the user has not yet interacted with the command, the system will proceed to the next step; otherwise, the system will not proceed to the next step, and other content in the user experience space will respond to this command.

[0129] ② The rotation transformation relationship from the audience field of view coordinate system to the audience experience space coordinate system, converting the interactive operation command posture parameters from the audience experience space coordinate system to the audience field of view coordinate system;

[0130] Allows the XR terminal to locate the audience h in real time k In its immersive experience space coordinate system The pose under [W k,0 Q k,0 ],in, Q k,0 =(α k β k θ k ). Thus, arrive In the coordinate system transformation relationship, the rotation angle is Q k,0 , the translation amount is W k,0 , the scaling factor is 1, according to formula (4), we can get arrive The rotation, scaling and translation matrix H k , let h k User experience space coordinate system Any point in In h k Field of view coordinate system The lower coordinate value is Then from arrive The coordinate transformation relationship is:

[0131]

[0132] The positional parameters of interactive operation commands Substituting into formula (13), we can calculate the interactive operation command A k Positional parameters are converted to h k User field of view coordinate system The coordinate value W under user , change the interactive operation command A k attitude angle When rotating the angle, the position parameter When the translation and scaling coefficient are 1, a rotation, scaling and translation matrix H′ can be constructed according to formula (4): k , then H k ·H′ k According to equations (6), (7), and (8), the rotation angle vector Q is converted user ,Q user It is the interactive operation command A k Convert to h k User field of view coordinate system The attitude angle under k for arrive The rotation, scaling and translation matrix.

[0133] ③ The interactive command pose parameters are converted to the reference audience field of view coordinate system

[0134] According to the audience h k pupil distance d k Interpupillary distance to the reference audience When the interactive command pose parameters are converted to the reference audience field of view coordinate system, the pose angle remains unchanged, but the coordinate value changes. The interactive operation command is in h k User field of view coordinate system The coordinate value W under user , then the interactive operation command is in the reference audience field coordinate system Ψ user The lower coordinate value is

[0135] ④Convert the interactive command pose parameters from the reference user field of view coordinate system to the live virtual scene coordinate system

[0136] Bundle Substituting into formula (12), we can calculate the interactive operation command A k Position parameters in live virtual scene m Coordinate system Ψ m The corresponding coordinate value W, According to formulas (6), (7), and (8), the interactive operation command A can be converted k The attitude angle parameter is m The corresponding attitude angle value Q, where The reference audience field coordinate system Ψ user To the live virtual scene coordinate system Ψ m Assign [WQ] to the interactive operation command A k The pose parameters of the interactive operation command A k Converted into scenes m The interactive operation command A′ k , scenes m For A′ k Response, correctly implement any audience user h k Based on live screen and scene m interaction.

[0137] Reference Figure 8 The hardware structure of the immersive interactive live broadcast construction system based on XR technology of the present invention is as follows: Figure 8 As shown, the immersive interactive live broadcast hardware system based on XR technology of the present invention mainly includes: a cloud server cluster, network equipment, and an XR terminal.

[0138] In a cloud server cluster, servers include an immersive interactive live broadcast management server, multiple XR application servers, and storage servers. XR application servers must have high-performance graphics rendering capabilities and must be interconnected. Users' XR terminals, which can be XR helmets or glasses, access the management server and all XR application servers via a wireless network.

[0139] Reference Figure 9 The software structure of the immersive interactive live broadcast construction system based on XR technology of the present invention is as follows: Figure 9 As shown, the immersive system software based on XR technology of the present invention mainly includes four modules:

[0140] Immersive interactive live broadcast management and control center, XR resource library, user terminal, and XR application server. Among them:

[0141] The immersive interactive live broadcast management control center is installed on the immersive interactive live broadcast management server. The XR resource library is deployed on the storage server. The user end is installed on the user's XR terminal. The XR application server is installed on each XR application server. The main functions of each module are described below.

[0142] 1. XR Resource Library:

[0143] The XR resource library stores XR application resources such as XR applications and XR resource packages.

[0144] 2. Immersive interactive live broadcast management and control center:

[0145] Installed on the immersive interactive live broadcast management server, its functions include:

[0146] (1) Accept the user's request for virtual scene experience, instruct the XR application server to allocate a virtual scene instance, and assist the user in establishing a communication connection with the corresponding virtual scene instance.

[0147] (2) Receive the user's application for live broadcast sharing and instruct the XR application server to allocate a virtual scene instance for live broadcast.

[0148] (3) The user or the system sets the three-dimensional imaging range of the virtual scene live broadcast sharing in the immersive interactive live broadcast management and control center, and sets the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system.

[0149] (4) Any user applies to the management and control center to watch the live broadcast. The management and control center assists the user in establishing a communication connection with the live broadcast scene. Based on this communication connection, the user can receive the live stereo image and send the interaction with the live broadcast scene to the live broadcast scene.

[0150] 3. XR application server:

[0151] Specific features include:

[0152] After receiving the virtual scene instance allocation instruction sent by the immersive interactive live broadcast management control center, the XR application server will allocate the corresponding virtual scene instance according to the virtual scene ID in the instruction. When the existing scene instance does not have sufficient computing resources, it will call the corresponding XR application and related resource packages to generate a virtual scene instance, and establish a communication connection between the newly generated scene instance and the original scene instance to synchronize the scene status.

[0153] Receive the scene instance destruction instruction sent by the immersive interactive live broadcast management control center and destroy the virtual scene instance.

[0154] 4. User side:

[0155] Specific features include:

[0156] (1) Send a virtual scene experience application to the immersive interactive live broadcast management control center and establish a communication connection with the virtual scene instance assigned to this user;

[0157] (2) Collect the user terminal’s posture parameters, interactive operation information, etc. and send them to the scene instance;

[0158] (3) Receive the immersive experience image sent by the scene instance;

[0159] (4) Apply to the immersive interactive live broadcast management and control center to watch the live broadcast, establish a communication connection with the live broadcast scene, and receive the live broadcast images sent by the live broadcast scene;

[0160] (5) Determine whether the user's interactive operation is an interaction with the live scene. If the user is interacting with the live scene, convert the posture parameters of the interactive operation command into the live scene coordinate system, and send the converted interactive operation command to the live scene;

[0161] (6) Synthesize the live broadcast image and the images of other contents in the experience space to generate a complete three-dimensional visual experience image in which the user can watch the live broadcast of the virtual scene while experiencing other contents in the experience space, and display it to the user.

[0162] Based on this immersive interactive live broadcast system, immersive experience activities of single or multiple virtual scenes constructed with XR technology can be shared live. Audiences watching the immersive interactive live broadcast can participate in the interaction of the live virtual scene while watching, and can also immerse themselves in other content in parallel. Because the present invention specifies a benchmark user pupil distance and defines a benchmark audience field of view based on this benchmark pupil distance, the present invention only needs to render and generate a stereoscopic visual live broadcast picture of this benchmark audience field of view. Through algorithmic processing, it can adapt it to the pupil distance of different users, so that the live scene can be correctly integrated into the user's experience space, and the function of correctly interacting with the live virtual scene can be provided, thereby using fewer computing resources to adapt to all users with different pupil distances.

[0163] In addition, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is called by a processor, the steps of the immersive interactive live broadcast construction method based on XR technology are executed as described above.

[0164] The present invention proposes an immersive interactive live broadcast construction method, system and storage medium based on XR technology. By setting the live three-dimensional imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system, the pupil distance and viewing posture of the reference audience in the live virtual scene are calculated; based on the pupil distance and viewing posture of the reference audience in the live virtual scene, a stereoscopic visual live broadcast picture of the live three-dimensional imaging interval of the virtual scene is rendered and generated, and the stereoscopic visual live broadcast picture is sent to each audience; any audience receives the stereoscopic visual live broadcast picture, and / or the stereoscopic visual live broadcast picture is combined with the stereoscopic visual picture of other experience content in the audience experience space to form a stereoscopic visual picture that allows the audience to watch the live virtual scene while experiencing other content, and display it to the audience. Based on this immersive interactive live broadcast system, the immersive experience activities of a single or multiple virtual scenes constructed by XR technology can be shared live, and the audience watching the immersive interactive live broadcast can participate in the interaction of the live virtual scene while watching, and can also immerse themselves in other content in parallel. Since the present invention specifies a benchmark audience pupil distance and defines a benchmark audience field of view based on this benchmark audience distance, the present invention only needs to render and generate a stereoscopic visual live broadcast image of the live virtual scene under the pupil distance and posture of this benchmark audience in the live virtual scene. Through algorithm processing, it can adapt to the pupil distances of different users, correctly integrate the live scene into the user's experience space, and provide the function of correctly interacting with the live virtual scene, thereby using less computing resources and being able to adapt to all users with different pupil distances.

[0165] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Equivalent structures or equivalent process changes made by the scheme using the contents of the description and drawings of the present invention, or direct or indirect application in other related technical fields are also included in the patent protection scope of the present invention.

Claims

1. A method for constructing an immersive interactive live broadcast based on XR technology, characterized in that: The method comprises the following steps: Step S10: setting a live 3D imaging interval of the live virtual scene and a rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; Step S20: Calculating the pupil distance and viewing posture of the reference audience in the live virtual scene according to the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; Step S30: Based on the pupil distance and viewing posture of the reference viewer in the live virtual scene, rendering and generating a stereoscopic live picture of the live 3D imaging interval of the live virtual scene, and sending the stereoscopic live picture to each viewer; Step S40: Any audience receives a stereoscopic live broadcast picture, and / or the stereoscopic live broadcast picture is combined with the stereoscopic visual picture of other experience contents in the audience experience space to form a stereoscopic visual picture for watching the live broadcast of the virtual scene while experiencing other content, and displayed to the audience.

2. The method according to claim 1, characterized in that The stereoscopic live broadcast picture generated in step S30 has depth information. In step S40, for any viewer h k , h k The received stereoscopic live broadcast image and the stereoscopic image of other experience contents in the experience space are synthesized into h k While watching the live broadcast of virtual scenes, you can experience other contents in stereoscopic visual images. k When synthesizing the stereoscopic visual images generated by other content experiences, it is necessary to k The depth information of the field of view is used for occlusion calculation.

3. The method according to claim 2, characterized in that In the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system set in step S10, the live virtual scene s m The scaling factor from the coordinate system of the reference audience field of view coordinate system is λ m The step S20 calculates the benchmark audience in the live virtual scene s m The pupil distance in in The actual pupil distance of the benchmark audience is based on the live virtual scene s m The rotation, scaling and translation relationship between the coordinate system and the reference viewer's field of view coordinate system is used to calculate the reference viewer's position in the live virtual scene. The step S30 is based on the reference viewer's position in the live virtual scene s m The posture and pupil distance of is the left eye image of the stereoscopic vision picture, is the right eye image of the stereoscopic vision picture, is the corresponding depth image, according to the scaling factor λ m , calculated and The depth information under the reference audience field of view is the depth image and The step S40 of synthesizing the stereoscopic visual live broadcast image and the stereoscopic visual image of other experience contents in the audience experience space is specifically implemented as follows: any viewing scene s m Live audience h k , receiving scene s m Stereoscopic live broadcast picture, h k The left eye image of the stereoscopic visual experience screen generated by other experience contents in the immersive experience space is The right eye image is and In h k The depth information of the field of view is the depth map and h k The ratio of pupil distance to the reference audience is Scenes m Stereoscopic live broadcast and Depth image under the reference audience field of view and Convert to h k Depth image of the field of view Stereoscopic visual experience images generated by the live stereoscopic visual image and other experience content are based on the depth image Perform occlusion calculations to generate a composite image.

4. The method according to claim 3, characterized in that The step S10, setting the live 3D imaging interval of the live virtual scene and the rotation, scaling and translation relationship between the live virtual scene coordinate system and the reference audience field coordinate system, specifically includes: m Select the 3D bounding box Ω′ m , set the bounding box Ω′ m In scenes m The pose in the bounding box Ω′ m In scenes m Select a three-dimensional interval Ω m ,Ω m It's the scene m Live 3D imaging interval, set the bounding box Ω′ m In the reference audience's field of view, the pose and scale relationship is calculated to calculate the virtual scene s m Live 3D imaging interval Ω m Corresponding 3D display range in the reference audience field of view Set the bounding box Ω′ m In live virtual scenes m The pose and Ω′ in m The position and scaling relationship in the reference audience field of view, that is, the live virtual scene s is set m The rotation, scaling and translation relationship between the coordinate system and the reference audience field of view coordinate system.

5. The method according to claim 4, characterized in that In step S40, for any viewer h k , live virtual scenes m In h k The three-dimensional display range of the experience space is Generate h k When experiencing the stereoscopic visual images of other experience contents in the experience space, h k Other experience contents in the experience space are in the 3D display area The content is not imaged.

6. The method according to claim 5, characterized in that The specific implementation of step S40 of calculating the three-dimensional imaging interval of the live virtual scene in the three-dimensional display interval of any audience experience space is: the live virtual scene s m The 3D display range in the reference audience field of view is Ω user , by any viewer h k The ratio of pupil distance to the reference audience pupil distance Calculate s m Three-dimensional imaging interval Ω m In h k The three-dimensional display range in the field of view is 7. The method according to any one of claims 1 to 6, characterized in that After step S40, there is step S50: any viewer h k In its experience space, the virtual scene of the live broadcast m Interact and generate interactive operation command A in the experience space k , A k Convert to live virtual scenes m The interactive operation command A′ k , put A′ k Send to scenes m , scenes m For A′ k Respond.

8. The method according to claim 7, characterized in that: The step S50 specifically includes: Step S501: Any viewer h k Perform interactive operations in your own experience space to generate audience h k Interactive operation command A in the experience space k , command A k The pose parameters included are the viewer h k Experience the posture in the spatial coordinate system and judge A k Is it a live virtual scene? m If it is a live virtual scene m If the interaction is successful, then go to step S502, otherwise h k Other content in the experience space responds to interactive operation commands A k , do not proceed to step S502; Step S502: According to the audience h k Experience space coordinate system and h k Rotation and translation relationship of the field of view coordinate system, interactive operation command A k The pose parameters are converted into h k Pose parameters in the field of view coordinate system; Step S503: According to the audience h k Pupillary distance d k Interpupillary distance to the reference audience The ratio of interactive operation command A k The pose parameters are obtained from h k The pose parameters in the field of view coordinate system are converted into pose parameters in the reference audience field of view coordinate system; Step S504: Based on the reference audience field coordinate system and the live virtual scene s m Coordinate system rotation, scaling and translation relationship, interactive operation command A k The pose parameters are transformed from the reference audience field of view coordinate system to the live virtual scene s m Pose parameters in the coordinate system, generating interactive operation commands A′ k ; Step S505: Interactive operation command A′ k , sent to the virtual live scene s m , s m For A′ k response.

9. An immersive interactive live broadcast construction system based on XR technology, characterized in that: The system includes: an immersive interactive live broadcast management server, multiple XR application servers, and a storage server. The user's XR terminal accesses the management server and all XR application servers through a wireless network. The management server is deployed with an immersive interactive live broadcast management control center, and the storage server is deployed with an XR resource library: The XR resource library is used to store XR application resources; The immersive interactive live broadcast management control center is used to receive virtual scene experience applications from XR terminals, instruct the XR application server to allocate virtual scene instances, and assist the XR terminals in establishing communication connections with the virtual scene instances; receive user live broadcast sharing applications, instruct the XR application server to allocate virtual scene instances for live broadcast; set the live 3D imaging interval of the live virtual scene; set the rotation, scaling, and translation relationship between the live virtual scene coordinate system and the reference audience field of view coordinate system; calculate the reference audience's position and pupil distance in the live virtual scene and send them to the live virtual scene; The XR application server is configured to receive a virtual scene instance allocation instruction sent by the immersive interactive live broadcast management control center, allocate a corresponding virtual scene instance according to the scene ID of the virtual scene in the scene instance allocation instruction, and when an existing scene instance does not have sufficient computing resources, call the corresponding XR application and related resource package to generate a virtual scene instance, and establish a communication connection between the newly generated scene instance and the original scene instance to synchronize the scene status; The XR application server is further configured to receive a scene instance destruction instruction sent by the immersive interactive live broadcast management control center, and destroy the virtual scene instance according to the instruction; The XR terminal is used to send a virtual scene experience application to the immersive interactive live broadcast management and control center, establish a communication connection with the virtual scene instance assigned to the XR terminal user; collect the XR terminal's posture parameters and interactive operation information and send them to the virtual scene instance assigned to the XR terminal user; receive the immersive experience picture sent by the virtual scene instance; apply to the immersive interactive live broadcast management and control center for watching live broadcasts, establish a communication connection with the live broadcast scene, and receive the live broadcast picture sent by the live broadcast scene; determine whether the user's interactive operation is an interaction with the live broadcast scene. When interacting with the live broadcast scene, convert the posture parameters of the interactive operation command into the live broadcast scene coordinate system, and send the converted interactive operation command to the live broadcast scene; receive the live stereoscopic visual picture sent by the live virtual scene; synthesize the live stereoscopic visual picture and the stereoscopic visual experience picture of other contents in the experience space to generate a complete interactive experience picture for watching the virtual scene live broadcast while experiencing other content, and display it to the user.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when called by the processor, executes the steps of the method for constructing an immersive interactive live broadcast based on XR technology according to any one of claims 1 to 8.

Images