Live streaming method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to live streaming methods and apparatuses, an electronic device, and a storage medium. A method comprises: acquiring a source live video stream, and acquiring the video projection format of the source live video stream; when the video projection format of the source live video stream is a first projection format, converting the video projection format of the source live video stream into a second projection format to obtain a target live video stream, the first projection format being used for projecting a spherical image into a rectangular image, the second projection format being used for projecting a spherical image to each face of an unfolded cube, and the cube being circumscribed about the spherical image; and pushing the target live video stream to a terminal.

Description

Live streaming method and device, electronic device and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410437684.5, filed on April 11, 2024, entitled "Live streaming method and device, electronic device and storage medium", which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of data processing, and in particular, to a live streaming method and device, electronic device and storage medium. BACKGROUND

[0004] With the continuous development of technology, VR (Virtual Reality) live streaming has also developed to a great extent. Users can watch VR live streaming videos through VR devices. SUMMARY

[0005] The present disclosure provides a live streaming method and device, electronic device and storage medium.

[0006] According to a first aspect of the present disclosure, a live streaming method is provided, the method comprising:

[0007] obtaining a source live streaming video stream and obtaining a video projection format of the source live streaming video stream;

[0008] in a case where the video projection format of the source live streaming video stream is a first projection format, converting the video projection format of the source live streaming video stream to a second projection format to obtain a target live streaming video stream;

[0009] pushing the target live streaming video stream to a terminal.

[0010] According to a second aspect of the present disclosure, a live streaming method is provided, the method comprising:

[0011] obtaining a target live streaming video stream and obtaining a video projection format of the target live streaming video stream;

[0012] in a case where the video projection format of the target live streaming video stream is a second projection format, obtaining video layout information and a video type of the target live streaming video stream; wherein the second projection format is used to project a spherical image onto each face after unfolding a cube, and the cube circumscribes the spherical image;

[0013] based on the video layout information and the video type, splitting the target live streaming video stream into a plurality of sub-pictures, the plurality of sub-pictures comprising a plurality of perspectives;

[0014] map the plurality of sub-pictures to a target three-dimensional sphere respectively, and display a video picture on the target three-dimensional sphere on a terminal.

[0015] According to a third aspect of the present disclosure, a live streaming device is provided, the device comprising:

[0016] a video projection format obtaining module, configured to obtain a source live streaming video stream, and obtain a video projection format of the source live streaming video stream;

[0017] a video projection format converting module, configured to, in a case where the video projection format of the source live streaming video stream is a first projection format, convert the video projection format of the source live streaming video stream to a second projection format to obtain a target live streaming video stream; wherein the first projection format is used to project a spherical image to a rectangular image, and the second projection format is used to project the spherical image to each face of a cuboid after the cuboid is unfolded, the cuboid being circumscribed around the spherical image;

[0018] a video stream sending module, configured to push the target live streaming video stream to a terminal.

[0019] According to a fourth aspect of the present disclosure, a live streaming device is provided, the device comprising:

[0020] a format obtaining module, configured to obtain a target live streaming video stream, and obtain a video projection format of the target live streaming video stream;

[0021] a video information obtaining module, configured to, in a case where the video projection format of the target live streaming video stream is a second projection format, obtain video layout information and a video type of the target live streaming video stream; wherein the second projection format is used to project a spherical image to each face of a cuboid after the cuboid is unfolded, the cuboid being circumscribed around the spherical image;

[0022] a splitting module, configured to split the target live streaming video stream into a plurality of sub-pictures based on the video layout information and the video type, the plurality of sub-pictures comprising a plurality of perspectives;

[0023] a video display module, configured to map the plurality of sub-pictures to a target three-dimensional sphere respectively, and display a video picture on the target three-dimensional sphere on a terminal.

[0024] According to a fifth aspect of the present disclosure, an electronic device is provided. The electronic device comprises a memory and a processor, the memory having a computer program stored thereon, and the processor implementing the method as described above when executing the program.

[0025] According to a sixth aspect of the present disclosure, a computer readable storage medium is provided, the medium having a computer program stored thereon, the program being executed by a processor to implement the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0026] More details, features and advantages of the present disclosure will be disclosed in the following description of the exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] FIG. 1 is a schematic diagram of a VR live scene according to an exemplary embodiment of the present disclosure;

[0028] FIG. 2 is a schematic diagram of converting an image in ERP format into an image in EAC format according to an exemplary embodiment of the present disclosure;

[0029] FIG. 3 is a schematic diagram of converting an image in ERP format into an image in EAC format according to another exemplary embodiment of the present disclosure;

[0030] FIG. 4 is a schematic diagram of converting an image in ERP format into an image in EAC format according to yet another exemplary embodiment of the present disclosure;

[0031] FIG. 5 is a schematic diagram of converting an image in ERP format into an image in EAC format according to yet another exemplary embodiment of the present disclosure;

[0032] FIG. 6 is a schematic diagram of converting an image in ERP format into an image in EAC format according to yet another exemplary embodiment of the present disclosure;

[0033] FIG. 7 is a schematic diagram of a three-dimensional mesh according to an exemplary embodiment of the present disclosure;

[0034] FIG. 8 is a schematic diagram of mapping an image to a three-dimensional mesh according to an exemplary embodiment of the present disclosure;

[0035] FIG. 9 is a flowchart of a live method according to an exemplary embodiment of the present disclosure;

[0036] FIG. 10 is a schematic block diagram of functional modules of a live device according to an exemplary embodiment of the present disclosure;

[0037] FIG. 11 is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0038] FIG. 12 is a structural block diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0040] It should be understood that each step recited in the method embodiments of the present disclosure can be performed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect.

[0041] The term “comprising” and variations thereof as used herein are open-ended, that is, “comprising but not limited to.” The term “based on” means “based, at least in part, on.” The term “one embodiment” means “at least one embodiment.” The term “another embodiment” means “at least one additional embodiment.” The term “some embodiments” means “at least some embodiments.” Related definitions are given below. It should be noted that the concepts mentioned in the present disclosure are merely used to distinguish different apparatuses, modules or units, and are not intended to limit the order or interdependence of the functions performed by these apparatuses, modules or units.

[0042] It should be noted that the modification of “one” or “multiple” mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, it should be understood as “one or more”.

[0043] The names of the messages or information exchanged between the plurality of apparatuses in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0044] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in accordance with relevant laws and regulations.

[0045] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0046] As an optional but non-limiting implementation, in response to receiving the active request of the user, the manner of sending the prompt information to the user may be, for example, a pop-up window manner in which the prompt information may be presented in a text manner. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device. It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other manners that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0047] When a user watches a VR live video through a VR device, the user usually focuses attention on the middle part of the video. In the related art, a large amount of pixel redundancy is generated in the non-middle part of the VR live video, which causes waste of video coding resources and bandwidth in VR live, and has a negative impact on video transmission and playing. For example, a live video stream in an ERP (Equirectangular Projection, equirectangular projection) projection format is usually used in the related art. The video in the ERP projection format causes serious stretching of the top and bottom of the picture and a large amount of pixel redundancy. However, in VR live, the user's attention is mainly focused on the middle part of the VR picture. Compared with the center part of the live video stream in the ERP projection format, the top and bottom (i.e., the edges corresponding to the top and bottom of the VR picture) of the live video stream in the ERP projection format are allocated with denser pixels, which causes a large amount of waste of pixels, thereby causing waste of coding resources and bandwidth of CDN (Content Delivery Network, content delivery network) in VR live, and degrading the first frame time, second opening rate, lag rate, pull stream code rate or frame rate and other indicators of the VR live link, and further causing waste of video coding resources and bandwidth in VR live, and having a negative impact on video transmission and playing.

[0048] In the embodiments provided in the present disclosure, as shown in FIG. 1, the server can receive a VR live video stream sent by a host end. The VR live video stream can be obtained by splicing video images in different angles or by a panoramic camera to obtain a 360-degree or 180-degree panoramic image, and the host end sends the obtained VR live video stream to the server.

[0049] In the embodiments, the video images captured by multiple cameras can be spliced into a 360-degree or 180-degree panoramic image in an ERP format by a panoramic camera, the panoramic image in the ERP format is projected to a spherical coordinate to obtain a spherical image, and the VR live video stream can be obtained according to the spherical image.

[0050] In the embodiment, the anchor end can also convert the spherical image into a two-dimensional plane image in the ERP format, and then obtain the VR live video stream based on the two-dimensional plane image.

[0051] It should be noted that the VR live video stream sent by the anchor end can also be a VR live video stream in other projection formats, and the embodiments are not limited thereto.

[0052] In combination with FIG. 1, when the VR live video stream is received, the VR live video stream is first unpacked by the unpacking module during transcoding of the VR live video stream, and the audio stream is obtained by decrypting through the audio decoding module. The decoded live video stream is obtained by the video decoding module. The VR live video stream in the ERP format is converted into a VR live video stream in the EAC format by the projection format conversion module, and the VR live video stream is encoded by the video encoding module. The audio stream is encoded by the audio encoding module, and the encoded audio stream and video stream are encapsulated by the encapsulation module. The VR live video stream obtained after encapsulation is distributed to each edge node through the content distribution network.

[0053] The video decoding module, the projection format conversion module, and the video encoding module are located in the GPU (Graphics Processing Unit, graphics processing unit), and the video encoding and decoding and the projection format conversion are performed by the GPU.

[0054] Therefore, when the server receives the VR live video stream sent by the anchor end, it can first determine whether the VR live video stream is in the ERP format. If it is in the ERP format, the VR live video stream in the ERP format needs to be converted into a VR live video stream in the EAC format. Since the video in the EAC format can uniformly disperse the pixels, it can avoid a large number of pixel redundancies in the non-middle part of the VR live video.

[0055] Therefore, when the VR live video stream is received, it is detected that the projection format of the VR live video stream is in the ERP format. It can also be determined whether the VR live video stream is a spherical image or a two-dimensional plane image according to the video type of the VR live video stream, so as to convert the VR live video stream in the ERP format into a VR live video stream in the EAC format.

[0056] Specifically, when the VR live video stream is a two-dimensional planar image, the VR live video stream in the ERP format can be directly converted into the VR live video stream in the EAC format. When the VR live video stream is a spherical image, for example, a 360-degree spherical image, six pictures of a cube can be obtained by sampling in six directions from the center of the sphere, and then the six pictures of the cube are obtained by calculating each pixel of each picture according to the following formula (1), and the six pictures are the pictures of the EAC. When the VR live video stream is a 180-degree spherical image, five pictures in five directions can be obtained by sampling in five directions from the center of the sphere. As shown in FIG. 2, which is a schematic diagram of converting the image in the ERP format into the image in the EAC format, the left side of FIG. 2 is a face of the RPR cube, and the face has a pixel point p. The pixel point p is converted into the coordinate q of the pixel point by the coordinate transformation according to the formula (1), and the image in the EAC format can be obtained. The coordinates of the pixel point q (u, v) are as follows:

[0057] In the embodiment, the layout information of the VR live video stream can be obtained, and the VR live video stream in the ERP format is converted into the VR live video stream in the EAC format. The VR live video stream can be a video file in the FLV (FLASH VIDEO) format, and specifically can include four layouts: 2D180 degrees, that is, a 180-degree fov (field of view) monocular picture; 2D360 degrees, that is, a 360-degree fov monocular picture; 3D180 degrees, that is, a 180-degree fov binocular picture; and 3D360 degrees, that is, a 360-degree fov binocular picture.

[0058] As shown in FIG. 3, when the VR live video stream is converted from the ERP format into the EAC format, if the VR live video stream is 2D180 degrees, one frame of video in the ERP format can be converted into five sub-pictures in the EAC format.

[0059] As shown in FIG. 4, when the VR live video stream is converted from the ERP format into the EAC format, if the VR live video stream is 2D360 degrees, one frame of video in the ERP format can be converted into six sub-pictures in the EAC format.

[0060] As shown in FIG. 5, when the VR live video stream is converted from the ERP format into the EAC format, if the VR live video stream is 3D180 degrees, one frame of video in the ERP format of the left eye and the right eye can be converted into five sub-pictures in the EAC format respectively.

[0061] As shown in FIG. 6, if the VR live video stream is 3D 180 degrees, the ERP format video pictures of a frame of left eye and right eye video can be converted into six EAC format sub-pictures respectively.

[0062] In the embodiment, when the terminal receives the VR live video stream, the received two-dimensional planar image needs to be projected onto a spherical surface to generate a spherical video image, so that the user can watch the panoramic video through the VR terminal. Therefore, when converting the ERP format VR live video stream into the EAC format VR live video stream, an SEI (Supplemental Enhancement Information) can be added on the video frame, which can carry an EAC identifier for representing that the video frame or the VR live video stream is an EAC format video stream, so as to facilitate the VR terminal to map the EAC format VR live video stream to a spherical coordinate according to the EAC format of the VR live video stream, and obtain a panoramic spherical video image.

[0063] In the embodiment, the above-mentioned SEI information can be added on the starting frame of the generated EAC format VR live video stream, and the embodiment is not limited thereto.

[0064] In combination with FIG. 1, after the protocol layer encapsulation of the audio and video stream encoding poem is completed, the stream can be pushed to the CDN, and the field information of the projection format can be added in the metadata metadata, for representing that the transmitted VR live video stream is an EAC format video stream.

[0065] When the client on the VR terminal receives the VR live video stream pushed by the edge node, the VR live video stream can be processed by the decapsulation module to obtain and be decoded by the audio decoding module and the video decoding module, so as to obtain the audio data and the video data stream.

[0066] When it is detected that the video data stream is an EAC format video stream, the EAC sub-pictures of multiple perspectives can be cut out according to the VR live video stream EAC layout information and the SEI information, and part of the EAC sub-pictures can be rotated.

[0067] Specifically, as shown in FIG. 3, when the EAC layout information of the VR live video stream is 2D 180 degrees, the layout is cut into five sub-pictures of left, front, right, down and up, and the up and down two sub-pictures are counterclockwise rotated by 90 degrees.

[0068] As shown in FIG. 4, when the EAC layout information of the VR live video stream is 2D 360 degrees, the layout is divided into six sub-pictures, i.e., left, front, right, lower, rear and upper, the lower and upper two sub-pictures are rotated counterclockwise by 90 degrees, and the'rear' sub-picture is rotated clockwise by 90 degrees.

[0069] As shown in FIG. 5, when the EAC layout information of the VR live video stream is 3D 180 degrees, the layout is divided into ten sub-pictures, i.e., left eye left, left eye front, left eye right, left eye lower, left eye upper, right eye left, right eye front, right eye right, right eye lower and right eye upper, the left eye lower, left eye upper, right eye lower and right eye upper four sub-pictures are rotated counterclockwise by 90 degrees.

[0070] As shown in FIG. 6, when the EAC layout information of the VR live video stream is 3D 360 degrees, the layout is divided into twelve sub-pictures, i.e., left eye left, left eye front, left eye right, left eye lower, left eye rear, left eye upper, right eye left, right eye front, right eye right, right eye lower, right eye rear and right eye upper, the left eye lower, left eye upper, right eye lower and right eye upper four sub-pictures are rotated counterclockwise by 90 degrees, and the left eye rear and right eye rear two sub-pictures are rotated clockwise by 90 degrees.

[0071] In the embodiment, each EAC sub-picture obtained above can be mapped to a three-dimensional spherical surface, and a three-dimensional spherical surface picture is displayed on the VR terminal.

[0072] Specifically, taking displaying the EAC 3D 180-degree picture as an example, a three-dimensional mesh is regenerated for the hemisphere, since the EAC is in five directions, the mesh also includes five parts, which are spliced by five sub-meshes, as shown in FIG. 7, each sub-mesh respectively receives the EAC sub-picture in the corresponding direction, so that the EAC sub-pictures in the five directions are mapped to the five sub-meshes. It should be noted that if it is 360 degrees, the EAC will be a picture in six directions, and the corresponding spherical mesh will include six sub-meshes.

[0073] In the embodiment, since the sub-mesh receives a quadrilateral picture, but the edge of each sub-mesh in the three-dimensional space is curved, as shown in FIG. 7, the left upper sub-mesh and the left two edges of the leftmost mesh in FIG. 7 are arc-shaped. Therefore, the accuracy of the sub-mesh cannot be too low, otherwise the edge will appear an unsmooth arc. In the embodiment, the accuracy cannot be infinitely high for performance consideration, and can be set as MxN according to business requirements, i.e., the accuracy of each surface can be limited to MxN, which can be set according to requirements, and the embodiment is not limited thereto.

[0074] In the embodiments provided in the present disclosure, the mesh can not be split according to the traditional vertex of the spindle organization, but according to the picture distribution of the EAC. For example, according to the vertex of the spindle organization, if the six faces of the EAC sub-picture are attached to the spherical surface of the vertex of the spindle organization, the vertex density of the upper and lower two faces will be higher than that of the front, rear, left and right four faces, that is, the vertex density of the six faces is inconsistent. At the same time, the upper and lower two faces cannot be cut into regular regular polygons, so more performance will be consumed during rendering calculation. The embodiments of the present disclosure can ensure the uniformity of the density of the six faces by splitting according to the EAC distribution, and can produce less calculation amount.

[0075] In the embodiments, the UV (texture mapping coordinates) can be rotated and offset according to the layout of the EAC, and the arctan calculation of mapping the EAC to the spherical surface is also included, and the five faces (or six faces) are spliced together. Since the collinear parts each use their own UV, there will be no visible joint at the splicing position.

[0076] Specifically, in the embodiments, the result of the EAC calculation can be written on the vertex, and if it is a joint part, it can be written twice. For example, the joint between the upper and front faces in FIG. 8, the calculation value of the joint is written once when calculating the upper face, and the joint of the front face is written once. The conversion values of the same position are saved on both sides, and the respective calculations are performed during rendering to achieve the processing of the joint.

[0077] Based on the above embodiments, the present disclosure further provides a live streaming method, which can be applied to a server. As shown in FIG. 9, the method can include the following steps:

[0078] In step S110, a source live video stream is obtained, and a video projection format of the source live video stream is obtained.

[0079] In the embodiments, when the video projection format is ERP, a large number of pixel redundancies will be generated in the non-middle part of the video stream. For example, a large number of pixels are concentrated at the edges of the top and bottom of the VR picture, and the user's attention is generally concentrated on the middle part of the video, which will cause a large waste of pixels, and further cause the occupation of too many coding resources and bandwidth resources in the video transmission process. The source live video stream can be the VR live video stream in the above embodiments.

[0080] Therefore, when the source live video stream is obtained, the video projection format of the source live video stream needs to be obtained, and if it is an ERP format, format conversion needs to be performed to avoid the occupation of too many coding resources and bandwidth resources in the video transmission process.

[0081] In step S120, in a case where the video projection format of the source live video stream is the first projection format, the video projection format of the source live video stream is converted into the second projection format, to obtain a target live video stream.

[0082] The first projection format is used for projecting a spherical image into a rectangular image, and the second projection format is used for projecting the spherical image onto each face of a cube unfolded.

[0083] In the embodiments, the first projection format can be the ERP projection format in the above embodiments, and the second projection format can be the EAC projection format in the above embodiments, or can be a CMP (CubeMap, Cube Map projection) projection format. In the embodiments, the second projection format is taken as the EAC projection format as an example, and the embodiments are not limited thereto.

[0084] In step S130, the target live video stream is pushed to the terminal.

[0085] The live streaming method provided in the embodiments of the present disclosure can detect the video projection format of a source live video stream, and convert the video projection format of the source live video stream into an equiangular cube map projection in a case where the video projection format of the source live video stream is an equirectangular projection. The target video stream obtained in this way can reduce the occupation of too much coding and bandwidth resources when being transmitted, and further improve the transmission efficiency of the video.

[0086] In the embodiments provided in the present disclosure, the server can receive the pull streaming information sent by the terminal, and the pull streaming information can carry the position information of the terminal. When receiving the pull streaming information, the server can push the live video stream to the edge node closest to the terminal through the CDN according to the position information of the terminal, to improve the transmission efficiency of the video. Therefore, the server can send the target live video stream to the content distribution network; in a case where a play request sent by the terminal is received, the target live video stream is sent to the target edge node through the content distribution network, and the target live video stream is pushed to the terminal through the target edge node; wherein the target edge node is determined based on the current position of the terminal.

[0087] Based on the above embodiments, in another embodiment provided in the present disclosure, the method can further include the following steps:

[0088] In step S140, target supplementary enhancement information is obtained. The target supplementary enhancement information carries a video projection format identifier of the second projection format.

[0089] In step S150, the target supplementary enhancement information is added to the target video frame of the target live video stream.

[0090] In the embodiments, the target supplemental enhancement information can carry the video projection format identifier of the EAC projection format. The target supplemental enhancement information can be the SEI information in the above embodiments. The SEI information can be added in each video frame, or can be added in the target video frame. For example, the starting video frame of the target live video stream can be obtained, and the starting video frame is taken as the target video frame. In this way, the SEI information is added in the starting video frame, and the starting frame and the video frames after the starting frame are processed, for example, are divided into multiple sub-pictures and are respectively mapped to the spherical coordinates.

[0091] In the embodiments, the target supplemental enhancement information can also be added in the key frame of the target live video stream, and the embodiments are not limited thereto.

[0092] Based on the above embodiments, in combination with FIG. 1, the source live video stream includes audio data and video data; and the step S120 can further include the following steps.

[0093] In step S121, the audio data in the source live video stream is encoded, and the video data converted into the second projection format is encoded.

[0094] In step S122, the encoded audio data and the encoded video data are encapsulated to obtain encapsulated audio and video stream data, and the video projection format field information of the second projection format is added in the metadata of the encapsulated audio and video stream data to obtain the target live video stream.

[0095] In the embodiments, the second projection format is taken as the EAC projection format. By respectively encoding the audio data and the video data in the live video stream, and encapsulating the encoded audio data and the encoded video data, the encapsulated audio and video stream data can be obtained, and the video projection format field information of the EAC projection format is added in the metadata of the encapsulated audio and video stream data. In this way, when the terminal receives the target live video stream, the terminal can directly determine whether the target live video stream is the live video stream in the EAC projection format according to the video projection format field information carried in the metadata, so as to facilitate processing of the video according to the projection format of the live video stream, and to improve the processing efficiency of the video.

[0096] Based on the above embodiments, the source live video stream can include a panoramic video, and the step S120 can further include the following steps.

[0097] In step S123, the video type of the panoramic video is obtained, and the video projection format of the source live video stream is converted into the second projection format based on the video type.

[0098] In the embodiments, the video projection format of the source live video stream can be converted into a second projection format based on a video type, and the second projection format is still taken as an example of the EAC projection format. The video type can be a spherical image video or a two-dimensional plane image video. When the VR live video stream is a two-dimensional plane image, the VR live video stream in the ERP format can be directly converted into the VR live video stream in the EAC format. When the VR live video stream is a spherical image, a plurality of directional plane images can be obtained by sampling in a plurality of directions from the center of the sphere, and the plurality of directional plane images can be converted from the ERP format into the EAC format.

[0099] Based on the above embodiments, in another embodiment provided by the present disclosure, a live streaming method is further provided, which can be applied to a terminal such as a VR terminal. The method can include the following steps:

[0100] In step S210, a target live video stream is obtained, and a video projection format of the target live video stream is obtained.

[0101] In the embodiments, the terminal can send pull streaming information to a push streaming end, and the push streaming end pushes a target live video stream such as a VR live video stream to the terminal. In order to display the live video stream on a spherical surface, the video projection format of the target live video stream needs to be obtained.

[0102] In step S220, when the video projection format of the target live video stream is a second projection format, video layout information and a video type of the target live video stream are obtained. The second projection format is used to project a spherical image onto each face of a cuboid after the cuboid is unfolded, and the cuboid is circumscribed around the spherical image.

[0103] In the embodiments, the second projection format is taken as an example of the EAC projection format.

[0104] The projection format of the target live video stream is determined by obtaining target supplemental enhancement information carried by the target live video stream. The target supplemental enhancement information can be included in a target video frame. The target video frame can be a starting frame or a key frame of the live video stream. The target supplemental enhancement information can also be carried in each video frame, and the embodiments are not limited thereto.

[0105] Therefore, in the embodiments, a target video frame in a target live video stream can be acquired, the target video frame carrying target supplemental enhancement information; the target supplemental enhancement information carrying a video projection format identifier of an equal-angle cubemap projection (EAC); and a video projection format of the target live video stream is determined based on the video projection format identifier. In this way, by acquiring the target supplemental enhancement information carried in the target video frame and determining the video projection format of the target live video stream, the terminal can perform format conversion processing on the target live video stream based on the video projection format.

[0106] If the live video stream is in an EAC format, video layout information and a video type of the target live video stream can be acquired, and specific descriptions can be referred to the descriptions in the above embodiments. The video layout information can specifically include 2D 180 degrees, 2D 360 degrees, 3D 180 degrees, or 3D 360 degrees, and the video type can include a spherical image or a planar image.

[0107] In step S230, the target live video stream is split into a plurality of sub-pictures based on the video layout information and the video type, and the plurality of sub-pictures include a plurality of perspectives.

[0108] In the embodiments, the video in the ERP format can be converted into the video in the EAC format according to the video type. In combination with FIGS. 3-6, the image in 2D 180 degrees, 2D 360 degrees, 3D 180 degrees, or 3D 360 degrees can be converted into the corresponding image in the EAC format according to the video type of the live video stream, so that a plurality of sub-EAC images can be obtained.

[0109] In step S240, the plurality of sub-pictures are respectively mapped to a target three-dimensional sphere, and a video picture on the target three-dimensional sphere is displayed on the terminal.

[0110] By acquiring the video layout information and the video type of the target live video stream, the video in the ERP format is converted into the video in the EAC format, and the plurality of generated EAC sub-pictures are respectively mapped to the sphere according to the video layout information, so that the video picture on the target three-dimensional sphere is displayed. In this way, the problem that a large number of pixels at the top and bottom of the video in the ERP format causes the video transmission process to occupy too much coding and decoding resources and bandwidth resources can be avoided.

[0111] Based on the above embodiments, in another embodiment provided in the disclosure, step S240 can further include the following steps:

[0112] In step S241, a target three-dimensional sphere is generated based on the video type, and a target three-dimensional sphere grid is generated based on the target three-dimensional sphere. The target three-dimensional sphere grid includes a plurality of three-dimensional grid surfaces.

[0113] In step S242, the plurality of sub-pictures are respectively mapped to a plurality of three-dimensional mesh surfaces to obtain a target three-dimensional spherical surface containing the plurality of sub-pictures.

[0114] In an embodiment, a plurality of meshes can be generated according to the video layout information to display an EAC 3D 180-degree picture. Taking the EAC 3D 180-degree picture as an example, a three-dimensional mesh is regenerated for a hemisphere. Since the EAC is in five directions, the mesh also includes five parts and is spliced by five sub-meshes, as shown in FIG. 7. Each sub-mesh receives an EAC sub-picture in a corresponding direction. In this way, the EAC sub-pictures in the five directions can be mapped to the five sub-meshes. It should be noted that if it is a 360-degree picture, the EAC will be a picture in six directions. In this way, the corresponding spherical mesh will include six sub-meshes. In this way, the plurality of EAC sub-pictures can be respectively mapped to the corresponding sub-meshes of each part, and the sub-pictures in each mesh are spliced and jointed. A video picture on a target three-dimensional spherical surface can be obtained.

[0115] In the case of dividing each functional module according to each function, the live broadcast device provided in the embodiments of the present disclosure can be a server, a terminal or a chip applied to a server. FIG. 10 is a schematic block diagram of a functional module of a live broadcast device provided in an example embodiment of the present disclosure. As shown in FIG. 10, the live broadcast device includes:

[0116] A video projection format obtaining module 10, configured to obtain a source live video stream and obtain a video projection format of the source live video stream;

[0117] A video projection format converting module 20, configured to convert the video projection format of the source live video stream into a second projection format to obtain a target live video stream in a case where the video projection format of the source live video stream is a first projection format, wherein the first projection format is used to project a spherical image into a rectangular image, and the second projection format is used to project the spherical image onto each face of a cuboid after the cuboid is unfolded, and the cuboid is circumscribed on the spherical image;

[0118] A video stream sending module 30, configured to push the target live video stream to a terminal.

[0119] In another embodiment provided in the present disclosure, the device further includes:

[0120] A supplementary enhancement information obtaining module, configured to obtain target supplementary enhancement information, wherein the target supplementary enhancement information carries a video projection format identifier of the second projection format;

[0121] An information adding module, configured to add the target supplementary enhancement information in a target video frame of the target live video stream.

[0122] In a further embodiment provided by the present disclosure, the apparatus further comprises:

[0123] a target video frame determination module configured to obtain a starting video frame of the target live video stream and take the starting video frame as the target video frame.

[0124] In a further embodiment provided by the present disclosure, the source live video stream comprises audio data and video data; and the video projection format conversion module is specifically configured to:

[0125] encode the audio data in the source live video stream and encode the video data converted into the second projection format;

[0126] package the encoded audio data and the encoded video data to obtain packaged audio and video stream data, and add video projection format field information of the second projection format in metadata of the packaged audio and video stream data to obtain the target live video stream.

[0127] In a further embodiment provided by the present disclosure, the apparatus further comprises a target live video stream sending module configured to:

[0128] send the target live video stream to a content distribution network;

[0129] in a case where a play request sent by a terminal is received, send the target live video stream to a target edge node through the content distribution network, and push the target live video stream to the terminal through the target edge node; wherein the target edge node is determined based on a current location of the terminal.

[0130] In a further embodiment provided by the present disclosure, the source live video stream comprises a panoramic video.

[0131] In a further embodiment provided by the present disclosure, the video projection format conversion module is specifically configured to:

[0132] obtain a video type of the panoramic video, and convert a video projection format of the source live video stream into a second projection format based on the video type.

[0133] The apparatus part corresponds to the above method, and details are described in the description of the corresponding method, which will not be described here.

[0134] The live broadcast device provided by the embodiments of the present disclosure can detect the video projection format of a source live broadcast video stream, and convert the video projection format of the source live broadcast video stream into a second projection format when the video projection format of the source live broadcast video stream is a first projection format. The target video stream obtained in this way can reduce the occupation of too much coding and decoding resources and bandwidth resources when being transmitted, thereby improving the transmission efficiency of the video.

[0135] In the case of dividing each functional module according to each function, the embodiments of the present disclosure provide a live broadcast device, which can be a server, a terminal or a chip applied to a server. The live broadcast device comprises:

[0136] a format obtaining module, configured to obtain a target live broadcast video stream and obtain a video projection format of the target live broadcast video stream;

[0137] a video information obtaining module, configured to obtain video layout information and a video type of the target live broadcast video stream when the video projection format of the target live broadcast video stream is a second projection format; wherein the second projection format is used to project a spherical image onto each face of a cuboid after the cuboid is unfolded, and the cuboid is circumscribed on the spherical image;

[0138] a splitting module, configured to split the target live broadcast video stream into a plurality of sub-pictures based on the video layout information and the video type, wherein the plurality of sub-pictures comprise a plurality of perspectives;

[0139] a video display module, configured to map the plurality of sub-pictures to a target three-dimensional spherical surface respectively, and display a video picture on the target three-dimensional spherical surface on a terminal.

[0140] In another embodiment provided by the present disclosure, the video display module is specifically configured to:

[0141] generate a target three-dimensional spherical surface based on the video type, and generate a target three-dimensional spherical surface grid based on the target three-dimensional spherical surface; the target three-dimensional spherical surface grid comprises a plurality of three-dimensional grid faces;

[0142] map the plurality of sub-pictures to the plurality of three-dimensional grid faces respectively to obtain a target three-dimensional spherical surface containing the plurality of sub-pictures.

[0143] In another embodiment provided by the present disclosure, the format obtaining module is specifically configured to:

[0144] obtain a target video frame in the target live broadcast video stream, wherein the target video frame carries target supplemental enhancement information; the target supplemental enhancement information carries a video projection format identifier of the second projection format;

[0145] Determine a video projection format of the target live video stream based on the video projection format identification.

[0146] Regarding the device part, see the description of the corresponding method. Here, the description is not repeated.

[0147] In an embodiment, by obtaining the video layout information and the video type of the target live video stream, the video in the first projection format is converted into the video in the second projection format, and according to the video layout information, the generated multiple sub-pictures in the second projection format are respectively mapped to the sphere, and then the video picture is displayed on the target three-dimensional sphere. In this way, the problem of occupying too many coding and bandwidth resources in the video transmission process caused by the large number of pixels at the top and bottom of the video in the first projection format can be avoided.

[0148] The embodiments of the present disclosure also provide an electronic device, including: at least one processor; a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above-mentioned method disclosed by the embodiments of the present disclosure.

[0149] FIG. 11 is a structural schematic diagram of an electronic device provided by an example embodiment of the present disclosure. As shown in FIG. 11, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801, and the processor 1801 can execute the corresponding steps in the above-mentioned method disclosed by the embodiments of the present disclosure.

[0150] The processor 1801 can also be referred to as a central processing unit (CPU), which can be an integrated circuit for processing signals. Each of the steps of the methods disclosed in the embodiments of the present disclosure can be completed by an integrated logic circuit of hardware or an instruction in the form of software in the processor 1801. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in the memory 1802, such as a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, and other mature storage mediums in the art. The processor 1801 reads information in the memory 1802 and completes the steps of the above methods in combination with the hardware thereof.

[0151] In addition, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can install programs constituting the software from a storage medium or a network to a computer system having a dedicated hardware structure, such as the computer system 1900 shown in FIG. 12, which, when various programs are installed, can perform various functions, including functions such as those described above. FIG. 12 is a structural block diagram of a computer system according to an exemplary embodiment of the present disclosure.

[0152] The computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic devices can also be represented by various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0153] As shown in FIG. 12, the computer system 1900 includes a computing unit 1901 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded into a random access memory (RAM) 1903 from a storage unit 1908. Various programs and data required for the operation of the computer system 1900 can also be stored in the RAM 1903. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other through a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0154] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 can be any type of device that can input information to the computer system 1900, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 1907 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices through a network such as the Internet, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, for example, a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0155] The computing unit 1901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1901 performs various methods and processes described above. For example, in some embodiments, the above-described methods disclosed by embodiments of the present disclosure can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the above-described methods disclosed by embodiments of the present disclosure by any other appropriate means (e.g., by means of firmware).

[0156] The embodiment of the present disclosure further provides a computer readable storage medium, wherein when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above method disclosed by the embodiment of the present disclosure.

[0157] The computer readable storage medium in the embodiment of the present disclosure can be a tangible medium, which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The above computer readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specifically, the above computer readable storage medium can include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (ERPOM or flash memory), optical fibers, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0158] The above computer readable medium can be included in the above electronic device; or can exist separately without being assembled into the electronic device.

[0159] The embodiment of the present disclosure further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the above method disclosed by the embodiment of the present disclosure.

[0160] In the embodiments of the present disclosure, computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of the same, including object oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on a user computer, partially on the user computer and partially on a remote computer, as a separate software package, partially on the user computer and partially on a remote computer, or completely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer.

[0161] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0162] The modules, components or units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the module, component or unit does not constitute a limitation on the module, component or unit itself.

[0163] The functions described in this specification can be implemented in part or in whole by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0164] The above description is merely illustrative of the embodiments of the present disclosure and the principles of the technology involved. It is understood that the disclosure scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features thereof without departing from the above disclosure concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0165] Although some specific embodiments of the present disclosure have been described in detail by way of examples, one skilled in the art should understand that the above examples are only for illustration, and are not intended to limit the scope of the present disclosure. One skilled in the art should understand that the above embodiments can be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A live broadcast method, wherein the method comprises: Obtain a source live video stream and obtain a video projection format of the source live video stream; When the video projection format of the source live video stream is a first projection format, converting the video projection format of the source live video stream into a second projection format to obtain a target live video stream; wherein the first projection format is used to project a spherical image into a rectangular image, and the second projection format is used to project the spherical image onto each face of an unfolded cube, and the cube is circumscribed to the spherical image; Push the target live video stream to the terminal.

2. The method according to claim 1, further comprising: Acquire target supplementary enhancement information, where the target supplementary enhancement information carries a video projection format identifier of the second projection format; The target supplementary enhancement information is added to the target video frame of the target live video stream.

3. The method according to claim 2, wherein the method further comprises: Obtain a starting video frame of the target live video stream, and use the starting video frame as the target video frame.

4. The method according to claim 1, wherein the source live video stream includes audio data and video data; Obtaining the target live video stream includes: Encoding the audio data in the source live video stream and encoding the video data converted into the second projection format; The encoded audio data and the encoded video data are encapsulated to obtain encapsulated audio and video stream data, and video projection format field information of the second projection format is added to the metadata of the encapsulated audio and video stream data to obtain the target live video stream.

5. The method according to claim 1, further comprising: Sending the target live video stream to a content distribution network; Upon receiving a playback request from a terminal, the target live video stream is sent to a target edge node via the content distribution network, and the target live video stream is pushed to the terminal via the target edge node; wherein the target edge node is determined based on the current location of the terminal.

6. The method according to claim 1, wherein the source live video stream comprises a panoramic video; The converting the video projection format of the source live video stream into a second projection format comprises: The video type of the panoramic video is obtained, and based on the video type, the video projection format of the source live video stream is converted into a second projection format.

7. A live broadcast method, wherein the method comprises: Obtain a target live video stream and obtain a video projection format of the target live video stream; When the video projection format of the target live video stream is a second projection format, obtaining video layout information and a video type of the target live video stream; wherein the second projection format is used to project a spherical image onto each face of an unfolded cube, the cube being circumscribed to the spherical image; Based on the video layout information and the video type, the target live video stream is divided into a plurality of sub-pictures, where the plurality of sub-pictures include a plurality of viewing angles; The multiple sub-pictures are respectively mapped to a target three-dimensional sphere, and the video picture on the target three-dimensional sphere is displayed on the terminal.

8. The method according to claim 7, wherein mapping the plurality of sub-pictures to a target three-dimensional sphere comprises: generating a target three-dimensional spherical surface based on the video type, and generating a target three-dimensional spherical surface mesh based on the target three-dimensional spherical surface; The target three-dimensional spherical grid includes a plurality of three-dimensional grid surfaces; The multiple sub-pictures are mapped to the multiple three-dimensional mesh surfaces respectively to obtain a target three-dimensional spherical surface containing the multiple sub-pictures.

9. The method according to claim 7, wherein obtaining the video projection format of the target live video stream comprises: Acquire a target video frame in the target live video stream, where the target video frame carries target supplementary enhancement information; The target supplementary enhancement information carries a video projection format identifier of the second projection format; The video projection format of the target live video stream is determined based on the video projection format identifier.

10. A live broadcast device, wherein the device comprises: A video projection format acquisition module is used to acquire a source live video stream and obtain a video projection format of the source live video stream; a video projection format conversion module, configured to, when the video projection format of the source live video stream is a first projection format, convert the video projection format of the source live video stream into a second projection format to obtain a target live video stream; wherein the first projection format is used to project a spherical image into a rectangular image, and the second projection format is used to project the spherical image onto each face of an unfolded cube, the cube being circumscribed to the spherical image; The video stream sending module is used to push the target live video stream to the terminal.

11. A live broadcast device, wherein the device comprises: A format acquisition module is used to acquire a target live video stream and obtain a video projection format of the target live video stream; A video information acquisition module, configured to acquire video layout information and a video type of the target live video stream when the video projection format of the target live video stream is a second projection format; wherein the second projection format is used to project a spherical image onto each face of an unfolded cube, the cube being circumscribed to the spherical image; a segmentation module, configured to segment the target live video stream into a plurality of sub-pictures based on the video layout information and the video type, wherein the plurality of sub-pictures include a plurality of viewing angles; The video display module is used to map the multiple sub-pictures to a target three-dimensional sphere respectively, and display the video pictures on the target three-dimensional sphere on the terminal.

12. An electronic device, wherein the device comprises: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 9.

Images