A VR content live broadcast method, device, and storage medium

Abstract

An embodiment of the present invention discloses a VR content live broadcast method, device and storage medium. The method includes: obtaining first video data and second video data; the first video data is video data of a sensitive area of a first HMD rendered by a host device; the second video data is video data of a non-sensitive area after rendering; performing image merging on the first video data and the second video data to generate panoramic VR video data; obtaining third video data, and generating fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host body video data sent by a shooting device; sending the panoramic VR video data and / or the fourth video data to a live broadcast server; the live broadcast server is configured to send corresponding video data to a second HMD.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular, to a VR content live broadcast method, apparatus, and storage medium. Background Art

[0002] Virtual Reality (VR) live broadcast uses VR technology. Users can watch the live broadcast through the application (APP) provided by the platform by wearing relevant hardware devices. The host needs to use a 360° panoramic shooting device to capture multi-angle pictures. After multi-picture transmission, viewers can observe the live broadcast experience from any angle, enabling viewers to feel more immersive. VR content live broadcast is the live broadcast of the VR interaction experience that the host is undergoing, such as the host experiencing a VR game or a VR education program, etc. Viewers can watch the host's experience process, which can be the video of the host's perspective during the experience, or the host's current experience from multiple perspectives or panoramas from other perspectives, or mixed reality (MR) viewing (watching the video synthesized by the VR image and the host image).

[0003] VR live broadcast generally uses a professional panoramic video shooting device for video acquisition. Generally, multiple wide-angle lenses are required for shooting (usually 6 or more), and in order to ensure seamless splicing of the videos collected by different cameras, sensor synchronization technology is used to keep the cameras synchronously collected. Panoramic video splicing requires a large amount of calculation and is generally completed on a local server. The original videos collected by multiple cameras need to be frame-by-frame real-time spliced. After a series of processes such as brightness and color adjustment, alignment, distortion correction, and projection onto a spherical surface of the multi-shot pictures, a complete panoramic video can be formed. The bit rate and resolution of the original videos are very high. To ensure the real-time nature of the live broadcast, splicing and projection transformation are generally performed on the local server directly connected to the acquisition device.

[0004] If the VR content live broadcast is the video of the host's perspective during the experience, usually the content of the host's current perspective is collected. Generally, a video stream is split from the head-mounted display (HMD) of the helmet to the live broadcast device, and the live broadcast device broadcasts this video stream. This method is relatively simple to implement and has low requirements for the performance of the host. However, viewers can only see the current perspective of the host player and cannot change the perspective according to the rotation of their heads. Therefore, viewers cannot experience the same immersive feeling as the host player.

[0005] If the multi - perspective or panoramic view of VR content live shows the host's current experience from other perspectives, usually the rendering engine not only renders the host's current perspective, but also needs to render a 360 - degree perspective centered on the host at the same time. Then, a panoramic video stream is split from the HMD to the live - broadcast device for live - broadcast. Viewers can then choose any perspective to watch according to the rotation of their heads. However, in this way, the host - side device needs to render a larger area of video, and at the same time, it also needs to encode and transmit a larger area of video, which brings an excessive burden to the host - side device and will directly affect the user experience of the host itself.

[0006] If the MR method is used to watch the host's live broadcast, the hardware device requires a camera to record the host's video and a high - performance host to process the video. The specific processing process includes: switching to the third - person perspective, recording the host's video, then editing and synthesizing the host's video and the VR content video, and finally live - broadcasting the mixed video. This requires a very powerful device to complete the editing, synthesis, encoding and transmission of the host's video and the VR content video.

[0007] In the existing VR live - broadcast methods, the first one is to only live - broadcast the video that the host perspective is experiencing. This is a completely passive viewing method for viewers and cannot let them experience the feelings of VR live - broadcast users immersive. Other VR live - broadcast methods require the host - side to process additional videos, including rendering parts that the host cannot see and video synthesis. This has high requirements for the performance of the host - side device, and at the same time, it may affect the host's interaction experience due to the consumption of device resources for live - broadcast processing. Summary of the Invention

[0008] To solve the related technical problems, embodiments of the present application provide a VR content live - broadcast method, device and storage medium.

[0009] The technical solution of the embodiments of the present application is implemented as follows:

[0010] An embodiment of the present invention provides a VR content live - broadcast method, which is applied to an edge server. The method includes:

[0011] Obtain first video data and second video data; the first video data is the video data of the sensitive area of the first HMD rendered by the host - side device; the second video data is the video data of the non - sensitive area after rendering.

[0012] Perform image merging on the first video data and the second video data to generate panoramic VR video data.

[0013] Obtain third video data, and generate fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host body video data sent by the shooting device.

[0014] Send the panoramic VR video data and / or the fourth video data to the live server; the live server is used to send the corresponding video data to the second HMD.

[0015] In the above solution, the obtaining of the first video data and the second video data includes:

[0016] Obtain the video data of the rendered sensitive area sent by the host device as the first video data;

[0017] Obtain the sensitive area information sent by the host device, determine the other areas except the sensitive area as non-sensitive areas according to the sensitive area information; render the video data of the non-sensitive areas to obtain the second video data.

[0018] In the above solution, the obtaining of the third video data and generating the fourth video data according to the third video data and the panoramic VR video data includes:

[0019] Obtain the third video data and the human body position information sent by the shooting device;

[0020] Merge the third video data with the panoramic VR video data according to the human body position information to obtain the fourth video data; wherein, the third video data is used as the foreground stream and the panoramic VR video data is used as the background stream.

[0021] In the above solution, the sending of the fourth video data to the live server includes:

[0022] Perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation;

[0023] Perform video compression on the fourth video data after the projection transformation to obtain the first target video data;

[0024] Send the first target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0025] In the above solution, the sending of the panoramic VR video data to the live server includes:

[0026] Perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation;

[0027] Perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data;

[0028] Send the second target video data to the live server; the live server is configured to generate third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0029] In the above solution, the method further includes:

[0030] Perform projection transformation on the fourth video data to obtain the fourth video data after projection transformation; perform video compression on the fourth video data after projection transformation to obtain the first target video data;

[0031] Perform projection transformation on the panoramic VR video data to obtain the panoramic VR video data after projection transformation; perform video compression on the panoramic VR video data after projection transformation to obtain the second target video data;

[0032] Merge the first target video data and the second target video data to generate third target video data;

[0033] Send the first target video data and / or the third target video data to the live server; the third target video data is used to be sent by the live server to the second HMD.

[0034] An embodiment of the present invention provides a VR content live broadcast method, which is applied to a shooting device, and the method includes:

[0035] Obtain the host video data;

[0036] Identify the host human body contour in the host video data; crop the video data in the area including the host human body contour in the host video data based on the identified host human body contour;

[0037] Compress the video data in the area including the host human body contour to obtain the third video data, and send the third video data to the edge server; the edge server is configured to generate fourth video data based on the third video data and the panoramic VR video data.

[0038] An embodiment of the present invention provides a VR content live broadcast method, which is applied to a host device, and the method includes:

[0039] Obtain the video data and operation information sent by the first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD;

[0040] Determine a sensitive area according to the operation information; render the video data in the sensitive area to obtain the first video data;

[0041] Send the first video data to the first HMD and / or the edge server; the edge server is used to generate panoramic VR video data based on the first video data.

[0042] In the above solution, the method further includes:

[0043] Send sensitive area information and operation information to the edge server.

[0044] An embodiment of the present invention provides a VR content live broadcast device, which is applied to a terminal and includes:

[0045] A first acquisition module, configured to acquire first video data and second video data; the first video data is video data of a sensitive area of a rendered first HMD sent by a host device; the second video data is video data of a non-sensitive area after rendering;

[0046] A first merging module, configured to perform image merging on the first video data and the second video data to generate panoramic VR video data;

[0047] A first generation module, configured to acquire third video data, and generate fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the video data of the host body sent by a shooting device;

[0048] A first sending module, configured to send the panoramic VR video data and / or the fourth video data to a live broadcast server; the live broadcast server is used to send the corresponding video data to a second HMD.

[0049] In the above solution, the first acquisition module is configured to acquire the video data of the rendered sensitive area sent by the host device as the first video data;

[0050] Acquire the sensitive area information sent by the host device, determine other areas except the sensitive area as non-sensitive areas according to the sensitive area information; render the video data of the non-sensitive areas to obtain the second video data.

[0051] In the above solution, the first generation module is configured to acquire the third video data and the human body position information sent by the shooting device;

[0052] According to the human body position information, merge the third video data with the panoramic VR video data to obtain the fourth video data; wherein, the third video data is used as the foreground stream, and the panoramic VR video data is used as the background stream.

[0053] In the above solution, the first generation module is used to perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation;

[0054] Perform video compression on the fourth video data after the projection transformation to obtain the first target video data;

[0055] The first sending module is used to send the first target video data to the live server; the live server is used to generate third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0056] In the above solution, the first generation module is used to perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation;

[0057] Perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data;

[0058] The first sending module is used to send the second target video data to the live server; the live server is used to generate third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0059] In the above solution, the first generation module is further used to perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation; perform video compression on the fourth video data after the projection transformation to obtain the first target video data;

[0060] Perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation; perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data;

[0061] Merge the first target video data and the second target video data to generate third target video data;

[0062] The first sending module is further used to send the first target video data and / or the third target video data to the live server; the third target video data is used to be sent by the live server to the second HMD.

[0063] An embodiment of the present invention provides a VR content live broadcast device, which is applied to a shooting device and includes:

[0064] A second acquisition module, which is used to acquire the host video data;

[0065] The first processing module is used to identify the contour of the host's body in the host video data through the human figure separation technology; and crop the video data of the area containing the contour of the host's body in the host video data based on the identified contour of the host's body.

[0066] The second processing module is used to compress the video data of the area containing the contour of the host's body to obtain the third video data.

[0067] The second sending module is used to send the third video data to the edge server; the edge server is used to generate the fourth video data based on the third video data and the panoramic VR video data.

[0068] An embodiment of the present invention provides a VR content live broadcast device, which is applied to the host device and includes:

[0069] The third acquisition module is used to acquire the video data and operation information sent by the first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD.

[0070] The third processing module is used to determine the sensitive area according to the operation information; and render the video data in the sensitive area to obtain the first video data.

[0071] The third sending module is used to send the first video data to the first HMD and / or the edge server; the edge server is used to generate the panoramic VR video data based on the first video data.

[0072] In the above solution, the third sending module is further used to send the sensitive area information and operation information to the edge server.

[0073] An embodiment of the present invention provides a VR content live broadcast device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of any one of the above VR content live broadcast methods executed on the edge server side; or,

[0074] When the processor executes the program, it implements the steps of any one of the above VR content live broadcast methods executed on the shooting device side; or,

[0075] When the processor executes the program, it implements the steps of any one of the above VR content live broadcast methods executed on the host device side.

[0076] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of any one of the above VR content live broadcast methods executed on the edge server side; or,

[0077] When the computer program is executed by a processor, it implements the steps of any one of the above-described VR content live broadcast methods executed on the shooting device side; or,

[0078] When the computer program is executed by a processor, it implements the steps of any one of the above-described VR content live broadcast methods executed on the host device side.

[0079] A VR content live broadcast method, device, and storage medium provided by an embodiment of the present invention. The method includes: an edge server obtains first video data and second video data; the first video data is video data of a sensitive area of a first HMD rendered and sent by a host device; the second video data is video data of a non-sensitive area after rendering; the first video data and the second video data are image-combined to generate panoramic VR video data; third video data is obtained, and fourth video data is generated based on the third video data and the panoramic VR video data; the third video data is obtained based on host body video data sent by a shooting device; the panoramic VR video data and / or the fourth video data is sent to a live broadcast server; the live broadcast server is configured to send the corresponding video data to a second HMD; thus, by processing video data through an edge server, two types of videos (i.e., panoramic VR video data and fourth video data including host body video) are provided for viewers, and viewers can select different video data based on their own needs, improving the user experience without affecting the interaction experience of the host. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flowchart of a VR content live broadcast method provided by an embodiment of the present invention;

[0081] Figure 2 It is a flowchart of another VR content live broadcast method provided by an embodiment of the present invention;

[0082] Figure 3 It is a flowchart of still another VR content live broadcast method provided by an embodiment of the present invention;

[0083] Figure 4 It is a schematic structural diagram of a VR content live broadcast system provided by an application embodiment of the present invention;

[0084] Figure 5 It is a flowchart of a processing method of an edge server provided by an application embodiment of the present invention;

[0085] Figure 6 It is a schematic diagram of a VR content image provided by an application embodiment of the present invention;

[0086] Figure 7Schematic structural diagram of a VR content live broadcast device provided by an embodiment of the present invention;

[0087] Figure 8 Schematic structural diagram of another VR content live broadcast device provided by an embodiment of the present invention;

[0088] Figure 9 Schematic structural diagram of yet another VR content live broadcast device provided by an embodiment of the present invention;

[0089] Figure 10 Schematic structural diagram of still another VR content live broadcast device provided by an embodiment of the present invention. Detailed implementation manners

[0090] The present invention will be further described in detail below in conjunction with embodiments.

[0091] Figure 1 Schematic flow chart of a virtual reality (VR) content live broadcast method provided by an embodiment of the present invention; as Figure 1 shown, the method can be applied to an edge server, and the edge server refers to a network edge side server close to users, which can be an edge node of a content delivery network (CDN), or an edge node deployed in a base station, or a mobile edge computing (MEC) server deployed in a cell computer room; the method includes:

[0092] Step 101, obtain first video data and second video data;

[0093] Wherein, the first video data is video data of a sensitive area of a first HMD rendered and sent by a host device;

[0094] The second video data is video data of a non-sensitive area after rendering;

[0095] Step 102, perform image merging on the first video data and the second video data to generate panoramic VR video data;

[0096] Step 103, obtain third video data, and generate fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on host body video data sent by a shooting device;

[0097] Step 104, send the panoramic VR video data and / or the fourth video data to a live broadcast server; the live broadcast server is used to send the corresponding video data to a second HMD.

[0098] In actual application, the first video data can be generated and sent by the host device. Specifically, the host device can receive the video data sent by the first HMD (assuming that in a live broadcast scenario, the first HMD can be the HMD worn by the host), and at the same time obtain the operation information of the first HMD (including the head rotation information and / or operation instructions of the first HMD), determine the sensitive area based on the operation information, and render the partial video data corresponding to the sensitive area in the first initial video data to obtain the first video data.

[0099] Here, the sensitive area can be the visible area corresponding to the current field of view angle; it can also be the predicted rotation area obtained by predicting the head rotation position according to the device performance of the first HMD, and the union of the visible area and the predicted rotation area is used as the sensitive area. The method of predicting the head rotation position can be implemented by combining the content to be live broadcast and the device performance of the first HMD.

[0100] The second video data can be the video data of the rendered non-sensitive area (referring to other areas except the sensitive area).

[0101] In some embodiments, obtaining the first video data and the second video data includes:

[0102] Obtain the video data of the rendered sensitive area sent by the host device as the first video data;

[0103] Obtain the sensitive area information sent by the host device, determine the other areas except the sensitive area as the non-sensitive area according to the sensitive area information; render the video data of the non-sensitive area to obtain the second video data.

[0104] Here, the sensitive area information is used to describe the information of the sensitive area (ROI, region of interest). In VR live broadcast, the sensitive area is the area where the human eye is looking and the area where the human eye is predicted to look soon. The sensitive area information can be determined by the host device and sent to the edge server, the purpose of which is to inform the edge server which part of the area is the sensitive area, so that the edge server can obtain the non-sensitive area based on the sensitive area information and render the video data of the non-sensitive area.

[0105] For the rendering method, it can be to render the sensitive area and the non-sensitive area according to the preset VR content display rules (such as the rendering rules of VR games). The above rules can be preset and saved in the edge server and the host device.

[0106] In some embodiments, the edge server synthesizes the video data of the sensitive area obtained from the host device and the video data of the non-sensitive area generated by the edge server, and the complete panoramic VR video data can be obtained. That is, the first video data and the second video data are merged to generate panoramic VR video data.

[0107] In actual application, the edge server simultaneously obtains the host video from the shooting device (such as a camera) of the host, combines the body position information of the host, separates the body contour and background of the host in the host video, and obtains the video data including the host body (denoted as the third video data), as the foreground image.

[0108] Here, the body position information is the relative position of the host body in the panoramic VR video data. When the VR service starts, the spatial positions of the HMD and the handle will be located, that is, the relative positions of the HMD and the handle with respect to the video data. In application, the relative position of the entire body in the video data can be determined according to the HMD and the handle in the host video.

[0109] Based on the relative position of the body in the panoramic VR video data, the area including the body in the panoramic VR video data is determined, and this area is set as the current MR sensitive area; in this MR sensitive area, the video data of the body is used as the foreground stream, and the panoramic VR video data is used as the background stream to synthesize the host body and the live content to generate the MR image of the sensitive area.

[0110] Based on this, in some embodiments, the obtaining the third video data and generating the fourth video data according to the third video data and the panoramic VR video data includes:

[0111] Obtain the third video data and the body position information sent by the shooting device;

[0112] According to the body position information, merge the third video data with the panoramic VR video data to obtain the fourth video data; wherein, the third video data is used as the foreground stream, and the panoramic VR video data is used as the background stream.

[0113] Here, the body position information represents the relative position of the host body in the panoramic VR video data.

[0114] In some embodiments, the merging the third video data with the panoramic VR video data according to the body position information includes:

[0115] According to the body position information, determine the area where the body included in the third video data corresponds to in the panoramic VR video data (denoted as the MR sensitive area);

[0116] Use the video data of the MR sensitive region in the third video data as the foreground stream and the panoramic VR video data as the background stream, correspondingly determine the area where the human body is located in the panoramic VR video data, and merge the third video data into the panoramic VR video data to obtain the merged video data, which is the fourth video data.

[0117] In practical applications, the edge server can send the processed video data to the HMD of the viewer for presentation through the live server.

[0118] Based on this, in some embodiments, the sending the fourth video data to the live server includes:

[0119] Perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation;

[0120] Perform video compression on the fourth video data after the projection transformation to obtain the first target video data;

[0121] Send the first target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0122] In some embodiments, the sending the panoramic VR video data to the live server includes:

[0123] Perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation;

[0124] Perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data;

[0125] Send the second target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0126] Correspondingly, the live server merges the first target video data and the second target video data to generate the third target video data; and sends the third target video data to the second HMD.

[0127] In this way, only transmitting the MR video data of the sensitive region can reduce the video bit rate transmitted from the edge server to the live server, reduce the transmission bandwidth, similar to SVC encoding.

[0128] It can be understood that after receiving the compressed panoramic VR content video data (i.e., the second target video data) and the compressed MR video data (i.e., the first target video data) from the edge server, the live server first decodes the two video streams, and then synthesizes new MR panoramic video data using the MR video data (i.e., the first target video data) in the MR sensitive area and the panoramic VR video data (i.e., the second target video data) in the non-sensitive area. Transcode and protocol encapsulate the two panoramic video streams (referring to the panoramic VR video data, or the new panoramic VR video data merged with the MR video data in the MR sensitive area), and then distribute them to the CDN.

[0129] The viewing user selects the panoramic live content or the panoramic MR live content according to their own preferences, obtains the corresponding video stream from the live server (i.e., the user independently selects whether to watch the panoramic VR video data including the host's body), decodes, inverse-transforms, and displays it, and arbitrarily selects a viewing angle.

[0130] Of course, the edge server can also merge the video data by itself and send the merged third target video data to the live server, which is directly sent to the viewer's HMD by the live server. That is, the live server only acts as a relay party.

[0131] Based on this, in some embodiments, the method further includes:

[0132] Perform projection transformation on the fourth video data to obtain the fourth video data after projection transformation; perform video compression on the fourth video data after projection transformation to obtain the first target video data;

[0133] Perform projection transformation on the panoramic VR video data to obtain the panoramic VR video data after projection transformation; perform video compression on the panoramic VR video data after projection transformation to obtain the second target video data;

[0134] Merge the first target video data and the second target video data to generate the third target video data;

[0135] Send the third video data to the live server; the third target video data is used to be sent by the live server to the second HMD.

[0136] The third target video data includes: the first target video data (i.e., the video data merged with the MR sensitive area video data and the panoramic VR video data), and the second target video data (i.e., the panoramic VR video data);

[0137] The live server sends the third target video data to the viewer's HMD (i.e., the second HMD). After the second HMD decodes, inverse-transforms, etc. the third target video data, the viewer can choose to watch the first target video data or the second target video data, that is, the viewer can choose panoramic live content or panoramic MR live content according to their own preferences, that is, arbitrarily choose the viewing angle.

[0138] Figure 2 The flowchart of another VR content live method provided by the embodiment of the present invention; as Figure 2 shown, the method can be applied to a shooting device, and the shooting device can be a camera set at the host side for shooting the video of the host wearing the first HMD; the method includes:

[0139] Step 201, obtain the host video data;

[0140] Here, the shooting device shoots the host video data through its own multiple cameras;

[0141] Step 202, identify the host body contour in the host video data; crop the video data of the area containing the host body contour in the host video data based on the identified host body contour;

[0142] Here, the host body contour in the host video data can be identified by the portrait separation technology (also called portrait segmentation technology) to determine the video data of the area containing the host body contour.

[0143] Step 203, compress the video data of the area containing the host body contour to obtain the third video data, and send the third video data to the edge server; the edge server is used to generate the fourth video data based on the third video data and the panoramic VR video data.

[0144] Figure 3 The flowchart of yet another VR content live method provided by the embodiment of the present invention; as Figure 3 shown, the method can be applied to the host side device, and the host side device communicates with the first HMD worn by the host and communicates with the edge server; the method includes:

[0145] Step 301, obtain the video data and operation information sent by the first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD;

[0146] Step 302, determine the sensitive area according to the operation information; render the video data in the sensitive area to obtain the first video data;

[0147] Step 303: Send the first video data to the first HMD and / or the edge server; the edge server is used to generate panoramic VR video data based on the first video data.

[0148] Here, the host device can receive the video data sent by the first HMD (assuming that in a live broadcast scenario, the first HMD can be the HMD worn by the host), denoted as the first initial video data, and at the same time obtain the operation information of the first HMD (including the head rotation information and / or operation instructions of the first HMD), determine the sensitive area based on the operation information, and render the partial video data corresponding to the sensitive area in the first initial video data to obtain the first video data.

[0149] The first HMD receives and displays the host content (i.e., the first video data) rendered by the host device, such as games or educational programs, etc.

[0150] Moreover, the first HMD device sends the head rotation situation and operation instructions (such as relevant operation instructions performed by the handle) of the first HMD to the host device.

[0151] Specifically, the host device obtains the current head rotation situation and operation instructions of the host's first HMD from the first HMD of the host, and at the same time determines the spatial position information of the host in combination with the initial host positioning. Combining the head rotation situation, operation instructions, and current spatial position information of the host, etc., determine the sensitive area. The sensitive area can be the visible area corresponding to the current host's field of view, or the union of the visible area and the predicted rotation area after predicting the head rotation position according to the device performance.

[0152] The host device renders the video data of the sensitive area according to the VR content display rules (such as the rendering rules of VR games) to obtain the first video data. After rendering, the video data of the visible area is transmitted to the first HMD for the first HMD to display. Compress the video data of the rendered sensitive area, and send the compressed video data of the sensitive area and the marked sensitive area position information to the edge server.

[0153] The host device also sends the head rotation situation, operation instructions, and host position information of the first HMD to the edge server.

[0154] Based on this, in one embodiment, the method further includes:

[0155] Send sensitive area information and operation information to the edge server.

[0156] Based on Figures 1 to 3The method provided by the embodiment of the present invention shown herein, in VR video live streaming, the host device can only process the video rendering of the sensitive area, compress and encode the video data of the sensitive area and then transmit it to the edge server, and transmit the operation information of the first HMD of the host device and the host spatial position information to the edge server; at the same time, the camera captures the current situation of the host and transmits the video data containing the host's body contour to the edge server. After receiving the sensitive area information sent by the host device, the edge server renders the video data of the non-sensitive area according to the operation information of the host to obtain complete panoramic VR video data. In addition, according to the host video transmitted by the camera of the host device, the host video and the panoramic VR content in the host area are synthesized to generate MR sensitive area live content. The edge server encodes these two data streams and then sends them to the live server. The live server uses the video of the MR sensitive area and the panoramic VR content video of the non-sensitive area to synthesize a new MR panoramic video, and distributes the panoramic VR content video and the MR panoramic video to the CDN, and sends them to the second HMD of the viewing user through the CDN for presentation.

[0157] Figure 4 It is a schematic structural diagram of a VR content live streaming system provided by an application embodiment of the present invention; as Figure 4 shown, the VR content live streaming system includes: a first HMD, a host device, a host shooting device, an edge server, a live server, a CDN, and a second HMD.

[0158] Among them, the first HMD is the HMD held by the host, and the host uses the first HMD during the live streaming process;

[0159] The second HMD is the HMD held by the viewing user, and the viewing user watches the live stream through the second HMD;

[0160] The shooting device at the live end can adopt devices such as cameras that have shooting functions and communication functions.

[0161] The first HMD is used to perform the following operations:

[0162] Display the host content rendered by the host device, such as games or educational programs, etc.;

[0163] Send its own operation information to the host device; the operation information includes: the rotation situation of the HMD head, operation instructions (such as relevant operation instructions performed by the handle).

[0164] The shooting device on the host side is used to shoot the video data when the host wears the first HMD. Through the portrait separation technology, it identifies the host's body contour in the recorded video, crops the video data containing the host's body contour area, compresses the video data containing the host's body contour area, and transmits the compressed corresponding video to the edge server.

[0165] The host-side device is used to perform the following operations:

[0166] Obtain the operation information of the current first HMD (including: head rotation situation, operation instructions, etc.) from the first HMD, and combine the initial host positioning to determine the spatial position information of the host;

[0167] Combine the head rotation situation, operation instructions, and current host position of the first HMD to determine the sensitive area; the sensitive area can be the visible area corresponding to the current field of view angle, or it can be the combination of the visible area and the predicted rotation area; the predicted rotation area can be determined according to the device performance to predict the head rotation position;

[0168] According to the VR content display rules (such as the rendering rules of VR games), render the video data of the sensitive area (including: the visible area, and can also include: the predicted rotation area). After rendering, transfer the video data of the visible area to the first HMD for display on the first HMD;

[0169] Compress the video data of the rendered sensitive area, and send the compressed video data of the sensitive area (i.e., the video picture displayed in the host's first HMD) and the marked sensitive area position information to the edge server;

[0170] Send the head rotation situation, operation instructions, and host position information of the first HMD (i.e., HMD and host information) to the edge server.

[0171] Figure 5 It is a schematic flow diagram of a processing method for an edge server provided in the application embodiment of the present invention; as Figure 5 shown, after receiving the sensitive area information sent by the host-side device, the edge server renders the video data of other areas (non-sensitive areas) except the sensitive area according to the operation information of the first HMD and the host and in accordance with the VR content rendering rules.

[0172] The edge server synthesizes the video data of the sensitive area obtained from the host-side device and the video data of the non-sensitive area generated by the edge server to obtain the complete panoramic VR video data. Perform projection transformation on this panoramic VR video data, and then perform video compression to generate the compressed panoramic VR video data, and push it to the live server.

[0173] The edge server simultaneously obtains the host video from the host camera, considers the host position information, separates the host human body contour and the background in the host video image, and obtains the foreground image of the human body and the relative position of the human body in the panoramic VR video data. According to the relative position of the human body in the panoramic VR video data, the area containing the human body in the panoramic VR video data is determined, and this area is set as the current MR sensitive area. In this MR sensitive area, the human body is used as the foreground image and the panoramic VR video data is used as the background for host and content synthesis to generate MR sensitive area video data.

[0174] Perform a projection transformation on the MR sensitive area video data, and then perform video compression to generate compressed MR sensitive area video data, which is pushed to the live server. The purpose of only transmitting the MR sensitive area video data is to reduce the video bit rate transmitted from the edge server to the live server and reduce the transmission bandwidth, similar to scalable video coding (SVC).

[0175] The live server is used to decode the two video data after receiving the compressed panoramic VR video data and the compressed MR sensitive area video data from the edge server, and synthesize new MR panoramic video data using the MR sensitive area video data and the panoramic VR video data in the non-sensitive area; perform transcoding and protocol encapsulation on the two panoramic video data, and then distribute them to the CDN. Viewing users can choose panoramic live content or panoramic MR live content according to their preferences, pull the stream from the CDN server, perform decoding, inverse transformation, and display, and view from any perspective.

[0176] The method or system provided by the embodiments of the present invention reduces the performance requirements of the host device, and at the same time provides viewers with two ways of VR live broadcast experience, and can view VR live broadcast and hybrid MR live content from any perspective.

[0177] Figure 7 It is a schematic structural diagram of a VR content live broadcast device provided by an embodiment of the present invention; as Figure 7 shown, applied to the edge server, the device includes:

[0178] The first acquisition module is used to acquire the first video data and the second video data; the first video data is the video data of the sensitive area of the first HMD rendered by the host device; the second video data is the video data of the non-sensitive area rendered.

[0179] The first merging module is used to perform image merging on the first video data and the second video data to generate panoramic VR video data.

[0180] The first generation module is used to obtain the third video data and generate the fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host body video data sent by the shooting device;

[0181] The first sending module is used to send the panoramic VR video data and / or the fourth video data to the live server; the live server is used to send the corresponding video data to the second HMD.

[0182] Specifically, the first acquisition module is used to obtain the video data of the rendered sensitive area sent by the host device as the first video data;

[0183] Obtain the sensitive area information sent by the host device, determine the other areas except the sensitive area as non-sensitive areas according to the sensitive area information; render the video data of the non-sensitive areas to obtain the second video data.

[0184] Specifically, the first generation module is used to obtain the third video data and the human body position information sent by the shooting device;

[0185] According to the human body position information, merge the third video data with the panoramic VR video data to obtain the fourth video data; wherein, the third video data is used as the foreground stream and the panoramic VR video data is used as the background stream.

[0186] Specifically, the first generation module is used to perform projection transformation on the fourth video data to obtain the fourth video data after projection transformation;

[0187] Perform video compression on the fourth video data after projection transformation to obtain the first target video data;

[0188] The first sending module is used to send the first target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

[0189] Specifically, the first generation module is used to perform projection transformation on the panoramic VR video data to obtain the panoramic VR video data after projection transformation;

[0190] Perform video compression on the panoramic VR video data after projection transformation to obtain the second target video data;

[0191] The first sending module is configured to send the second target video data to a live server; the live server is configured to generate third target video data based on first target video data and the second target video data; the third target video data is configured to be sent to a second HMD.

[0192] Specifically, the first generating module is further configured to perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation; perform video compression on the fourth video data after the projection transformation to obtain the first target video data;

[0193] Perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation; perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data;

[0194] Merge the first target video data and the second target video data to generate the third target video data;

[0195] The first sending module is further configured to send the first target video data and / or the third target video data to the live server; the third target video data is configured to be sent by the live server to a second HMD.

[0196] It should be noted that: when the VR content live broadcast device provided in the above embodiment implements the corresponding VR content live broadcast method, only the division of the above program modules is used for illustration. In actual applications, the above processing can be allocated to different program modules according to needs, that is, the internal structure of the edge server is divided into different program modules to complete all or part of the above-described processing. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be elaborated here.

[0197] Figure 8 This is a schematic structural diagram of another VR content live broadcast device provided by an embodiment of the present invention; as Figure 8 shown, applied to a shooting device, the device includes:

[0198] A second obtaining module, configured to obtain host video data;

[0199] A first processing module, configured to identify the host human body contour in the host video data through a human figure separation technique; crop the video data of the area including the host human body contour in the host video data based on the identified host human body contour;

[0200] A second processing module, configured to compress the video data of the area including the host human body contour to obtain third video data;

[0201] A second sending module, configured to send the third video data to an edge server; the edge server is configured to generate fourth video data based on the third video data and panoramic VR video data.

[0202] It should be noted that: when the VR content live broadcast device provided in the above embodiment implements the corresponding VR content live broadcast method, only the above-mentioned division of each program module is used for illustration. In actual applications, the above-mentioned processing can be allocated to different program modules according to needs, that is, the internal structure of the shooting device is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be elaborated here.

[0203] Figure 9 This is a schematic structural diagram of another VR content live broadcast device provided by an embodiment of the present invention; as Figure 9 shown, it is applied to a host device, and the device includes:

[0204] A third acquisition module, configured to acquire video data and operation information sent by a first HMD; the operation information includes: head rotation information and / or operation instructions of the first HMD;

[0205] A third processing module, configured to determine a sensitive area according to the operation information; render the video data within the sensitive area to obtain first video data;

[0206] A third sending module, configured to send the first video data to the first HMD and / or the edge server; the edge server is configured to generate panoramic VR video data based on the first video data.

[0207] Specifically, the third sending module is further configured to send sensitive area information and operation information to the edge server.

[0208] It should be noted that: when the VR content live broadcast device provided in the above embodiment implements the corresponding VR content live broadcast method, only the above-mentioned division of each program module is used for illustration. In actual applications, the above-mentioned processing can be allocated to different program modules according to needs, that is, the internal structure of the host device is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be elaborated here.

[0209] Figure 10 This is a schematic structural diagram of still another VR content live broadcast device provided by an embodiment of the present invention, as Figure 10As shown, the device 100 includes: a processor 1001 and a memory 1002 for storing a computer program that can run on the processor;

[0210] When the device is applied to an edge server, when the processor 1001 runs the computer program, it performs: obtaining first video data and second video data; the first video data is video data of a sensitive area of a first HMD rendered by a host device; the second video data is video data of a non-sensitive area after rendering; performing image merging on the first video data and the second video data to generate panoramic VR video data; obtaining third video data, and generating fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host body video data sent by a shooting device; sending the panoramic VR video data and / or the fourth video data to a live server; the live server is used to send the corresponding video data to a second HMD. Specifically, the edge server can execute as Figure 1 shown in Figure 1 The VR content live broadcast method embodiment shown belongs to the same concept, and its specific implementation process is detailed in the method embodiment and will not be elaborated here.

[0211] When the device is applied to a shooting device, when the processor 1001 runs the computer program, it performs: obtaining host video data; identifying the host body contour in the host video data; cropping the video data of the area containing the host body contour in the host video data based on the identified host body contour; compressing the video data of the area containing the host body contour to obtain third video data, and sending the third video data to an edge server; the edge server is used to generate fourth video data based on the third video data and panoramic VR video data. Specifically, the shooting device can execute as Figure 2 shown in Figure 2 The VR content live broadcast method embodiment shown belongs to the same concept, and its specific implementation process is detailed in the method embodiment and will not be elaborated here.

[0212] When the device is applied to a host device, when the processor 1001 runs the computer program, it performs: obtaining video data and operation information sent by a first HMD; the operation information includes: head rotation information and / or operation instructions of the first HMD; determining a sensitive area according to the operation information; rendering the video data in the sensitive area to obtain first video data; sending the first video data to the first HMD and / or an edge server; the edge server is used to generate panoramic VR video data based on the first video data. Specifically, the host device can execute as Figure 3The method shown is the same as Figure 3 the embodiment of the VR content live broadcast method shown, belonging to the same concept. For the specific implementation process, refer to the method embodiment, which will not be elaborated here.

[0213] In practical applications, the device 100 may further include: at least one network interface 1003. Each component in the device 100 is coupled together through a bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 10 all kinds of buses are labeled as the bus system 1004. Among them, the number of the processors 1001 can be at least one. The network interface 1003 is used for the communication between the device 100 and other devices in a wired or wireless manner.

[0214] The memory 1002 in the embodiment of the present invention is used to store various types of data to support the operation of the device 100.

[0215] The method disclosed in the above embodiment of the present invention can be applied to the processor 1001 or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 1001 or the instructions in the form of software. The above-mentioned processor 1001 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiment of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. Combining the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module may be located in the storage medium, and this storage medium is located in the memory 1002. The processor 1001 reads the information in the memory 1002 and combines its hardware to complete the steps of the foregoing method.

[0216] In an exemplary embodiment, the device 100 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontroller units (MCUs), microprocessors, or other electronic components for performing the foregoing method.

[0217] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored;

[0218] When the computer-readable storage medium is applied to an edge server, when the computer program is run by a processor, it performs: obtaining first video data and second video data; the first video data is video data of a sensitive area of a first rendered HMD sent by a host device; the second video data is video data of a non-sensitive area after rendering; performing image merging on the first video data and the second video data to generate panoramic VR video data; obtaining third video data, and generating fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on host body video data sent by a shooting device; sending the panoramic VR video data and / or the fourth video data to a live server; the live server is configured to send the corresponding video data to a second HMD. Specifically, the edge server may execute as Figure 1 shown in the method, and Figure 1 the VR content live method embodiment shown belongs to the same concept, and the specific implementation process is detailed in the method embodiment and will not be repeated here.

[0219] When the computer-readable storage medium is applied to a shooting device, when the computer program is run by a processor, it performs: obtaining host video data; identifying a host body contour in the host video data; cropping video data of an area including the host body contour in the host video data based on the identified host body contour; compressing the video data of the area including the host body contour to obtain third video data, and sending the third video data to an edge server; the edge server is configured to generate fourth video data based on the third video data and panoramic VR video data. Specifically, the shooting device may execute as Figure 2The method shown is the same as that Figure 2 in the embodiment of the VR content live broadcast method shown. They belong to the same concept. For the specific implementation process, please refer to the method embodiment and will not be elaborated here.

[0220] When the computer-readable storage medium is applied to the host device, when the computer program is run by the processor, it performs: obtaining video data and operation information sent by the first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD; determining a sensitive area according to the operation information; rendering the video data within the sensitive area to obtain first video data; sending the first video data to the first HMD and / or the edge server; the edge server is used to generate panoramic VR video data based on the first video data. Specifically, the host device can execute as Figure 3 the method shown is the same as that Figure 3 in the embodiment of the VR content live broadcast method shown. They belong to the same concept. For the specific implementation process, please refer to the method embodiment and will not be elaborated here.

[0221] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0222] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, in each embodiment of the present invention, each functional unit can be all integrated in one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in one unit; the above integrated unit can be implemented in the form of hardware, or in the form of a hardware plus software functional unit.

[0224] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including those of the above method embodiments; and the foregoing storage medium includes: various media such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0225] Alternatively, if the above integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the foregoing storage medium includes: various media such as removable storage devices, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0226] It should be noted that: "first", "second", etc. are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0227] In addition, the technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

[0228] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A VR content live broadcast method, characterized in that, Applied to an edge server, the method includes: Obtain first video data and second video data; the first video data is the video data of the sensitive area of the rendered first HMD sent by the host device; the second video data is the video data of the non-sensitive area after rendering; wherein, the sensitive area includes the visible area corresponding to the field of view angle when the first HMD performs a VR service, and the non-sensitive area includes other areas except the sensitive area; Perform image merging on the first video data and the second video data to generate panoramic VR video data; Obtain third video data, and generate fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host body video data sent by the shooting device; Send the panoramic VR video data and / or the fourth video data to the live server; the live server is used to send the corresponding video data to the second HMD.

2. The method according to claim 1, characterized in that, The obtaining of the first video data and the second video data includes: Obtain the video data of the rendered sensitive area sent by the host device as the first video data; Obtain the sensitive area information sent by the host device, determine the other areas except the sensitive area as the non-sensitive area according to the sensitive area information; render the video data of the non-sensitive area to obtain the second video data.

3. The method according to claim 1, wherein The obtaining of the third video data and generating the fourth video data according to the third video data and the panoramic VR video data includes: Obtain the third video data and the human body position information sent by the shooting device; According to the human body position information, merge the third video data with the panoramic VR video data to obtain the fourth video data; wherein, the third video data is used as the foreground stream and the panoramic VR video data is used as the background stream.

4. The method according to claim 1, wherein In the case of sending the fourth video data to the live server, the sending of the fourth video data to the live server includes: Perform projection transformation on the fourth video data to obtain the fourth video data after projection transformation; Perform video compression on the fourth video data after projection transformation to obtain the first target video data; Send the first target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

5. The method according to claim 1 or 4, characterized in that, The sending of the panoramic VR video data to the live server includes: Perform projection transformation on the panoramic VR video data to obtain the panoramic VR video data after projection transformation; Perform video compression on the panoramic VR video data after projection transformation to obtain the second target video data; Send the second target video data to the live server; the live server is used to generate the third target video data based on the first target video data and the second target video data; the third target video data is used to be sent to the second HMD.

6. The method according to claim 1, characterized in that, The method further includes: Perform a projection transformation on the fourth video data to obtain the fourth video data after the projection transformation; perform video compression on the fourth video data after the projection transformation to obtain the first target video data; Perform a projection transformation on the panoramic VR video data to obtain the panoramic VR video data after the projection transformation; perform video compression on the panoramic VR video data after the projection transformation to obtain the second target video data; Merge the first target video data and the second target video data to generate the third target video data; Send the first target video data and / or the third target video data to the live server; the third target video data is used to be sent by the live server to the second HMD.

7. A VR content live broadcast method, characterized in that, Applied to a shooting device, the method includes: Obtain the host video data; Identify the host human body contour in the host video data; crop the video data in the area containing the host human body contour in the host video data based on the identified host human body contour; Compress the video data in the area containing the host human body contour to obtain the third video data, and send the third video data to the edge server; the edge server is used to generate the fourth video data based on the third video data and the panoramic VR video data, and the fourth video data can be sent to the second HMD.

8. A method for live streaming VR content, characterized in that Applied to the host device, the method includes: Obtain the video data and operation information sent by the first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD; Determine the sensitive area according to the operation information; render the video data in the sensitive area to obtain the first video data; wherein, the sensitive area includes the visible area corresponding to the field of view angle when the first HMD executes the VR service; Send the first video data to the first HMD and / or the edge server; the edge server is used to generate the panoramic VR video data based on the first video data and the second video data, the second video data is the video data of the non-sensitive area after rendering, the non-sensitive area includes other areas except the sensitive area, and the panoramic VR video data can be sent to the second HMD.

9. The method according to claim 8, characterized in that, The method further includes: Send the sensitive area information and operation information to the edge server.

10. A VR content live broadcast device, characterized in that, Applied to the terminal, including: A first acquisition module, configured to acquire the first video data and the second video data; the first video data is the video data of the sensitive area of the first HMD after rendering sent by the host device; the second video data is the video data of the non-sensitive area after rendering; wherein, the sensitive area includes the visible area corresponding to the field of view angle when the first HMD executes the VR service, and the non-sensitive area includes other areas except the sensitive area; A first merging module, configured to perform image merging on the first video data and the second video data to generate the panoramic VR video data; A first generation module, configured to obtain the third video data, and generate the fourth video data according to the third video data and the panoramic VR video data; the third video data is obtained based on the host human body video data sent by the shooting device; A first sending module, configured to send the panoramic VR video data and / or the fourth video data to a live server; the live server is configured to send the corresponding video data to a second HMD.

11. A VR content live broadcast device, characterized in that, Applied to a shooting device, including: A second obtaining module, configured to obtain the host video data; A first processing module, configured to identify the host human body contour in the host video data through a human portrait separation technology; crop the video data in the area including the host human body contour in the host video data based on the identified host human body contour; A second processing module, configured to compress the video data in the area including the host human body contour to obtain third video data; A second sending module, configured to send the third video data to an edge server; the edge server is configured to generate fourth video data based on the third video data and the panoramic VR video data, and the fourth video data can be sent to the second HMD.

12. A VR content live broadcast device, characterized in that, Applied to a host device, including: A third obtaining module, configured to obtain the video data and operation information sent by a first HMD; the operation information includes: the head rotation information and / or operation instructions of the first HMD; A third processing module, configured to determine a sensitive area according to the operation information; render the video data in the sensitive area to obtain first video data; wherein, the sensitive area includes the visible area corresponding to the field of view angle when the first HMD executes the VR service; A third sending module, configured to send the first video data to the first HMD and / or the edge server; the edge server is configured to generate panoramic VR video data based on the first video data and the second video data, the second video data is the video data of the non-sensitive area after rendering, the non-sensitive area includes other areas except the sensitive area, and the panoramic VR video data can be sent to the second HMD.

13. A VR content live broadcast device, characterized in that, Including: A processor and a memory for storing a computer program that can run on the processor, Wherein, when the processor is configured to run the computer program, it executes the steps of the method according to any one of claims 1 to 6; or, when the processor is configured to run the computer program, it executes the steps of the method according to claim 7; or, when the processor is configured to run the computer program, it executes the steps of the method according to claim 8 or 9.

14. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it realizes the steps of the method according to any one of claims 1 to 6; or, When the computer program is executed by the processor, it realizes the steps of the method according to claim 7; or, When the computer program is executed by the processor, it realizes the steps of the method according to claim 8 or 9.

Images