Panoramic video playing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a panoramic video playing method and device, electronic equipment and storage medium. The method comprises downloading a target view cutting code stream corresponding to a target view slice in a panoramic view video code stream and a background code stream, the background code stream is a panoramic view video code stream, the definition of the target view cutting code stream is higher than that of the background code stream, merging the target view cutting code stream and the background code stream to obtain a merged code stream, and then decoding and rendering to realize panoramic video playing. By merging one high-definition local view cutting code stream and one low-definition background code stream, the transmission code rate of the panoramic video is reduced, and by the local high-definition code stream in the merged code stream, clear video can be seen within the viewing angle range, and by the background code stream, the panoramic video can be continuously presented. By code stream merging, one code stream can be formed, which can be normally decoded by a limited decoder, thereby reducing the decoding cost.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a panoramic video playback method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the increasing popularity of video applications, ultra-high resolution video (such as 4K and 8K video) and immersive video services are being widely used.

[0003] In related technologies, full-frame video is typically used for transmission to the terminal for decoding and playback. However, full-frame transmission has an excessively high bitrate, and hardware limitations result in significant decoding pressure, sometimes even making decoding impossible. Furthermore, in most cases, only a small portion of the full-frame image is actually seen by the user, for example, only about 25% of the image is visible, while the unseen portion wastes a significant amount of pixel transmission bitrate during transmission. Summary of the Invention

[0004] This disclosure provides a panoramic video playback method, apparatus, electronic device, and storage medium to reduce the waste of transmission bitrate during spherical video playback and reduce the decoding pressure of video playback.

[0005] In a first aspect, embodiments of this disclosure provide a panoramic video playback method, the method comprising:

[0006] Download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0007] The target view segmented bitstream and the background bitstream are merged to obtain a merged bitstream;

[0008] The merged bitstream is decoded and rendered for panoramic video playback.

[0009] Secondly, embodiments of this disclosure also provide a panoramic video playback device, the device comprising:

[0010] The download module is used to download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0011] The merging module is used to merge the target view segment bitstream and the background bitstream to obtain a merged bitstream;

[0012] The playback module is used to decode and render the merged bitstream for panoramic video playback.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic video playback method as described in any of the above embodiments.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the panoramic video playback method as described in any of the above embodiments.

[0018] In this embodiment, the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic video bitstream are downloaded. The background bitstream is the panoramic video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream. The target view segment bitstream and the background bitstream are merged to obtain a merged bitstream, which is then decoded and rendered to achieve panoramic video playback. By merging a high-resolution local view segment bitstream and a low-resolution background bitstream, the transmission bitrate of the panoramic video is reduced. Furthermore, the local high-resolution bitstream in the merged bitstream ensures that the video is clear within the viewing angle range, while the background bitstream ensures that the panoramic video is continuously displayed. In addition, the bitstream merging results in a single bitstream, which can be decoded normally with a limited decoder, reducing decoding costs.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 A flowchart illustrating a panoramic video playback method provided in this embodiment of the present disclosure;

[0022] Figure 2A schematic diagram of the overall architecture for panoramic video playback provided in the embodiments of this disclosure;

[0023] Figure 3 A flowchart illustrating another panoramic video playback method provided in this embodiment of the disclosure;

[0024] Figure 4 A schematic diagram of an applicable architecture for downloading bitstream during panoramic video playback, provided for embodiments of this disclosure;

[0025] Figure 5 A detailed schematic diagram illustrating the applicable bitstream download process during panoramic video playback, provided for embodiments of this disclosure;

[0026] Figure 6 A flowchart illustrating yet another panoramic video playback method provided in this disclosure embodiment;

[0027] Figure 7 A detailed schematic diagram illustrating the applicable method of merging bitstreams after downloading them during panoramic video playback, as provided in this embodiment of the disclosure.

[0028] Figure 8 A schematic diagram illustrating the interpolation process after bitstream merging during panoramic video playback, provided as an embodiment of this disclosure.

[0029] Figure 9 A schematic diagram illustrating the result of overlapping viewpoint segmentation during panoramic video playback, as provided in the embodiments of this disclosure;

[0030] Figure 10a A schematic diagram illustrating the principle of viewpoint re-arrangement during panoramic video playback, applicable to embodiments of this disclosure;

[0031] Figure 10b A schematic diagram illustrating the applicable video viewing angle switching during panoramic video playback, provided for embodiments of this disclosure;

[0032] Figure 11 A schematic diagram illustrating the applicable method for detecting purely random access frames during panoramic video playback, provided in an embodiment of this disclosure.

[0033] Figure 12 A schematic diagram illustrating misaligned keyframe streaming during panoramic video playback, provided as an embodiment of this disclosure.

[0034] Figure 13 A flowchart illustrating yet another panoramic video playback method provided in this disclosure embodiment;

[0035] Figure 14 A schematic diagram illustrating the applicable perspective rendering process during panoramic video playback, provided for embodiments of this disclosure;

[0036] Figure 15 A schematic diagram illustrating the positional mapping of viewpoint rendering during panoramic video playback, provided as an embodiment of this disclosure.

[0037] Figure 16 A structural block diagram of a panoramic video playback device provided in an embodiment of this disclosure;

[0038] Figure 17 A structural block diagram of an electronic device for implementing the panoramic video playback method of this embodiment. Detailed Implementation

[0039] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0040] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0043] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0044] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0045] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0046] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0047] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0048] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0049] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0050] Figure 1 This is a flowchart illustrating a panoramic video playback method provided in an embodiment of this disclosure. This embodiment is applicable to situations involving the playback of panoramic videos. The method can be executed by a panoramic video playback device, which can be implemented in software and / or hardware and is generally integrated into any electronic device with network communication capabilities. This electronic device may include, but is not limited to, mobile terminals, PCs, or servers. Figure 1 As shown, the panoramic video playback method of this embodiment may include steps S110-S130:

[0051] S110. Download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream; the background bitstream is the panoramic view video bitstream, and the clarity of the target view segment bitstream is higher than that of the background bitstream.

[0052] Full-frame transmission has a high bitrate, such as 25Mbps for on-demand and 50Mbps for live streaming, but only a portion of the video is actually viewed by the user. Furthermore, full-frame video playback typically involves decoding, and the limited decoding resources of the terminal chip increase the decoding load on the terminal, making it difficult to support very high resolutions.

[0053] See Figure 2 For panoramic video streams, instead of directly downloading the high-definition panoramic video stream for high-definition playback, a local viewpoint is selected from the panoramic viewpoint corresponding to the panoramic video stream, and this selected local viewpoint is used as the target viewpoint segment. Then, a segment of video stream corresponding to the target viewpoint segment can be downloaded from the panoramic video stream, thus obtaining the target viewpoint chunked stream. Each viewpoint in the panoramic video stream corresponds to one video stream, and the video streams from multiple video segments corresponding to different views constitute a complete panoramic video stream.

[0054] Meanwhile, panoramic video playback involves perspective switching. When this happens, acquiring the bitstream from the new perspective takes time, which can lead to a lack of bitstream from the new perspective, causing display interruptions and preventing normal display. Furthermore, the new perspective is difficult to predict. Even if it could be predicted accurately, transmitting both the original and new perspective video streams simultaneously would increase the transmission bitrate. Therefore, we choose to download a background bitstream with a lower resolution than the target perspective segment bitstream. This background bitstream is a full-view bitstream, and compared to the panoramic video stream, the video streams corresponding to each perspective in the background bitstream have lower resolution.

[0055] As an optional but non-restrictive implementation, see [link to relevant documentation]. Figure 2 To enable panoramic video playback across multiple terminal devices, a cloud server can be pre-deployed. This cloud server can store panoramic view video stream resources and is suitable for large-scale distribution of these streams. During panoramic video playback, different terminal devices can download the target view segment and background stream from the cloud server. Specifically, front-end devices can acquire the panoramic view video stream (including but not limited to video streams with a resolution of 7680x3840) to push it to the cloud server for storage.

[0056] For the above optional methods, cloud servers can be used to achieve large-scale video stream distribution. Transmitting video segments allows for selective transmission of the streams of interest, saving transmission bandwidth. Furthermore, background streams can supplement the viewing angle, enabling panoramic video playback.

[0057] As an optional but non-limiting implementation, the aforementioned terminal device can be a head-mounted virtual reality display device, such as a virtual reality headset, virtual reality glasses, etc. Of course, the virtual reality display device can also be any other playback device capable of playing panoramic videos.

[0058] S120. Merge the target view segment bitstream and the background bitstream to obtain a merged bitstream.

[0059] S130: Decode and render the merged bitstream for panoramic video playback.

[0060] See Figure 2 After downloading the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic video stream, the downloaded target view segment bitstream and background bitstream can be merged to obtain a merged bitstream. This bitstream merging allows for successful decoding and rendering using a finite decoder, reducing decoding costs. Simultaneously, the background bitstream, being a panoramic video stream, ensures continuous video streaming from all perspectives. Furthermore, the target view segment bitstream has higher resolution than the background bitstream, guaranteeing high-definition video within the user's viewing range and preserving the viewing experience.

[0061] It is understood that the "panoramic video playback" process in this embodiment may include, but is not limited to, video stream download, combination of video streams from different perspectives, merging and decoding, and playback of video frames within the stream.

[0062] According to the panoramic video playback scheme in this embodiment, the transmission bitrate of the panoramic video is reduced by merging a high-definition local viewpoint segmented bitstream and a low-definition background bitstream. The local high-definition bitstream in the merged bitstream ensures that the video is clear within the viewing angle range, while the background bitstream ensures that the panoramic video is continuously displayed. In addition, the bitstream merging can form a single bitstream, which can be decoded normally by a limited number of decoders, reducing decoding costs.

[0063] Figure 3This is a flowchart illustrating another panoramic video playback method provided by an embodiment of this disclosure. The technical solution of this embodiment further optimizes the process of downloading the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream, based on the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. For example... Figure 3 As shown, the panoramic video playback method of this embodiment may include the following steps S310-S350:

[0064] S310, Download the background stream corresponding to the panoramic video stream; the background stream is the panoramic video stream.

[0065] S320. Determine the target view segment from multiple view segments that have been pre-divided into panoramic view segments of the panoramic view video.

[0066] As an optional but non-restrictive implementation, see [link to relevant documentation]. Figure 2 On a pre-deployed cloud server, the panoramic view video stream can be divided into multiple view tiles. Each view tile corresponds to a video stream within a specific view range of the panoramic view video stream. This allows the target view tile for high-definition video stream playback to be selected from the pre-divided panoramic view tiles according to requirements.

[0067] As an optional but non-restrictive implementation, see [link to relevant documentation]. Figure 2 and Figure 4 For the multiple view segments pre-divided in the panoramic view of the panoramic video, the resolution of each view segment can be, but is not limited to, 448x448 or 640x640; the resolution of the background bitstream can be, but is not limited to, 1344x896.

[0068] S330. Download the target view segment bitstream corresponding to the target view segment from the panoramic view video stream in the playback order. The clarity of the target view segment bitstream is higher than that of the background bitstream. The target view segment bitstream is a video stream with a preset segment length.

[0069] See Figure 4Each panoramic view video stream can include multiple temporally consecutive panoramic view video frames, and correspondingly, the video stream under the viewpoint of each view segment can also include multiple temporally consecutive video frames. To achieve normal video playback, the video stream under the viewpoint corresponding to the target view segment can be determined from the panoramic view video stream, and a video stream of a preset segment length can be downloaded from the video stream under the viewpoint corresponding to the target view segment in playback order. The downloaded video stream of the preset segment length can then be used as the target view segment bitstream corresponding to the target view segment.

[0070] See Figure 4 The cloud server can divide and store the video streams of each viewpoint segment in the panoramic video stream according to viewpoint segments. Simultaneously, it stores the download links for each viewpoint segment and its corresponding video stream in a database file. It periodically updates and retrieves the download links for each viewpoint segment in the panoramic video stream. Based on the selected target viewpoint segment, it uses the corresponding download link and a media downloader to download the target viewpoint segment's corresponding cut-off video stream from the panoramic video stream in playback order.

[0071] As an optional but non-limiting implementation, determining the target view segment from multiple view segments pre-divided into panoramic view segments of the panoramic view video may include the following steps A1-A2:

[0072] Step A1: Based on the viewing posture of the panoramic video and the field of view parameters of the panoramic video display device, determine the target viewing angle in the panoramic view of the panoramic video stream.

[0073] See Figure 4 In response to a viewer's request to watch the panoramic video, a head tracker detects the viewer's head posture in real time and determines the viewer's viewing posture when watching the panoramic video. Simultaneously, the field of view parameters of the panoramic video display device can be obtained. In a display device system, the field of view can be the angle between the display edge of the device and the line connecting the observation point (e.g., the viewer's eyes). Furthermore, based on the video viewing posture and the field of view parameters of the panoramic video display device, the projection range of the viewer's line of sight within the panoramic view is calculated, thus obtaining the viewer's target viewing angle range within the panoramic view of the panoramic video stream.

[0074] For example, taking the panoramic view as a spherical region, when a viewer located at the center of the sphere watches the panoramic video, the viewing area of ​​the viewer's line of sight on the corresponding spherical region of the panoramic view is calculated. This viewing area is then projected onto a planar region, such as a latitude and longitude map, to determine the area covered by the viewer's viewing area in the panoramic view of the panoramic video stream. Thus, the target viewing angle range of the viewer in the panoramic view of the panoramic video stream can be determined.

[0075] Step A2: Determine the target view segment from multiple view segments pre-divided into panoramic view segments of the panoramic view video. The target view segment includes the pre-divided view segments covered by the target viewing view in the panoramic view.

[0076] See Figure 4 From multiple pre-divided viewpoint segments corresponding to the panoramic viewpoint of the panoramic video, the pre-divided viewpoint segment whose target viewing angle range is covered within the panoramic viewpoint is determined as the target viewing angle segment. The area occupied by the sum of all target viewing angle segments is greater than or equal to the target viewing angle range covered by the viewer in the panoramic viewpoint of the panoramic video stream. Using the above method, the target viewing angle segment can be adjusted in real time according to the video viewing posture and the field of view parameters of the panoramic video display device, thereby adapting the download of the high-definition video stream corresponding to the viewing angle range for playback.

[0077] As an optional but non-limiting implementation, the preset segmentation length includes a preset integer multiple of group of pictures (GOP), and each group of pictures includes a preset number of video frames or video frames of a preset duration. The target viewpoint segment bitstream corresponding to the target viewpoint slice can be obtained by cutting the video bitstream under the corresponding viewpoint of the panoramic viewpoint video bitstream according to the preset integer multiple of group of pictures as the bitstream segmentation interval.

[0078] S340. Merge the target view segment bitstream and the background bitstream to obtain a merged bitstream.

[0079] S350 decodes and renders the merged bitstream for panoramic video playback.

[0080] As an optional but non-limiting implementation, downloading the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream may include the following steps B1-B2:

[0081] Step B1: Download the background bitstream. The background bitstream is used to generate the merged bitstream when the target view segmented bitstream fails to download.

[0082] Step B2: Download the target view segment bitstream corresponding to the target view slice, prioritizing the download of the background bitstream over the target view segment bitstream.

[0083] See Figure 4 and Figure 5 After determining the target view segment's corresponding target view chunk bitstream and background bitstream that need to be downloaded in the panoramic view video stream, it is also necessary to determine the download priority of each target view segment's corresponding target view chunk bitstream and background bitstream. The background bitstream has the highest download priority, followed by the target view segment's corresponding target view chunk bitstream. Furthermore, for each target view segment's corresponding target view chunk bitstream, the closer the target view segment is to the center of the viewer's line of sight, the higher its download priority.

[0084] See Figure 5 In addition to downloading the target viewpoint segment bitstream and background bitstream, the downloader can also download the viewpoint segment bitstream corresponding to viewpoint segments outside the viewing viewpoint. These non-viewpoint segment segments include viewpoint segments other than the target viewpoint segment within the panoramic view, or viewpoint segments that were within the viewing viewpoint before the viewpoint switch but have moved out of the viewing viewpoint due to changes in the viewing posture. The download priority of the viewpoint segment bitstream corresponding to the non-viewpoint segment is lower than that of the target viewpoint segment bitstream and background bitstream.

[0085] In one optional example, while prioritizing the download of the background bitstream, the target view segment bitstream corresponding to the target view fragment is downloaded synchronously. In this case, the download stability of the background bitstream is higher than that of the target view segment bitstream corresponding to the target view fragment. This way, if the download of the target view segment bitstream fails, the background bitstream can be used directly to generate the merged bitstream.

[0086] In another alternative example, before downloading the target view segment bitstream corresponding to the target view segment, the background bitstream of the panoramic video is downloaded first. After the background bitstream is downloaded or a portion of the background bitstream has been downloaded and it is confirmed that it can continue to download stably, the download of the target view segment bitstream corresponding to the target view segment is started. This is so that if the download of the target view segment bitstream fails, the background bitstream can be used directly to generate the merged bitstream.

[0087] See Figure 5After downloading the target view segment corresponding to the target view block bitstream and the background bitstream at the (n+1)th second, the target view segment bitstream and the background bitstream at the (n+1)th second can be merged to obtain the merged bitstream at the (n+1)th second; after downloading the target view segment corresponding to the target view segment and the background bitstream at the (n+2)th second, the target view segment bitstream and the background bitstream at the (n+2)th second can be merged to obtain the merged bitstream at the (n+2)th second, and so on.

[0088] By using the above optional method, and by pre-configuring the priority of the background bitstream and the local high-definition video bitstream, it can be ensured that the background bitstream is downloaded first, so that the high-definition video bitstreams from each viewpoint can always be downloaded normally. In this way, when the target viewpoint segment bitstream fails to download, the background bitstream can be used directly to generate the merged bitstream. Even if the high-definition video bitstream fails to download later, the video frame corresponding to the background bitstream will always be present when the panoramic video is played, avoiding partial black screens and no picture due to the lack of video bitstreams from some viewpoints.

[0089] According to the panoramic video playback scheme in this embodiment, a suitable local high-definition video bitstream can be determined in real time to match the viewing angle and the matching high-definition video bitstream can be downloaded. The bitstream is merged by combining one high-definition local viewpoint segment bitstream and one low-definition background bitstream to reduce the transmission bitrate of the panoramic video. Furthermore, the local high-definition bitstream in the merged bitstream ensures that the video is clear within the viewing angle range, while the background bitstream ensures that the panoramic video is continuously displayed. In addition, the bitstream merging can form a single bitstream, which can be decoded normally by a limited decoder, reducing decoding costs.

[0090] Figure 6 This is a flowchart illustrating another panoramic video playback method provided in this disclosure. The technical solution of this embodiment further optimizes the process of merging the target viewpoint segmented bitstream and the background bitstream to obtain a merged bitstream, based on the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. For example... Figure 6 As shown, the panoramic video playback method in this embodiment may include the following steps S610-S640:

[0091] S610: Download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0092] S620. Merge the video frames in multiple target view slice streams that are played at the same time in the target view slice streams corresponding to each target view slice into the same video frame group, and each slice stream includes multiple video frames that are consecutive in the time domain.

[0093] For each target viewpoint segment, the corresponding target viewpoint block bitstream contains multiple temporally consecutive video frames within the target viewpoint block bitstream, with the same playback time for video frames in different target viewpoint block bitstreams. This allows us to iterate through the video frames within the target viewpoint block bitstreams corresponding to each target viewpoint segment, grouping video frames with the same playback time into the same video frame group, and then merging the video frames from each target viewpoint block bitstream within the same video frame group.

[0094] For example, see Figure 7 Taking the example that each of the target view segmented bitstream 1, target view segmented bitstream 2, and target view segmented bitstream 32 contains 30 consecutive video frames in the time domain, the first frame from each of the target view segmented bitstream 1, target view segmented bitstream 2, and target view segmented bitstream 32 is extracted and assigned to the first video frame group. The second frame from each of the target view segmented bitstream 1, target view segmented bitstream 2, and target view segmented bitstream 32 is extracted and assigned to the second video frame group, and so on, until the 30th frame from each of the target view segmented bitstream 1, target view segmented bitstream 2, and target view segmented bitstream 32 is extracted and assigned to the 30th video frame group.

[0095] As an optional but non-limiting implementation, when merging video frames in the target view slice bitstreams of different target view segments played at the same time within the same video frame group, the video frames in the target view slice bitstreams of different target view segments are merged sequentially according to the order of each target view segment in the panoramic view video stream.

[0096] S630. According to the playback order of video frames in the target viewpoint segment bitstream of each video frame group, merge each video frame group sequentially after the background bitstream to obtain a merged bitstream.

[0097] See Figure 7 When merging the bitstreams, the background bitstream is placed first. Considering the playback order of the video frames within each video frame group, the video frame groups need to be sorted according to the playback order of the video frames in the target viewpoint segment bitstream. Groups with earlier playback times are sorted first, and groups with later playback times are sorted last. The video frame groups are then merged after the background bitstream according to their sorting order. This ensures that the background bitstream is decoded first, resulting in the merged bitstream. For example, the first video frame group is placed after the background bitstream, then the second video frame group is merged after the first video frame group, and so on, to obtain the merged bitstream.

[0098] By using the Hong Kong-style method described above to group video frames in the segmented bitstream corresponding to different viewpoints, and placing video frames in the segmented bitstream at the same time together, the various high-definition segmented bitstreams can form a large, continuous bitstream in the time domain. This way, the decoder only needs to decode one bitstream, reducing decoding costs.

[0099] As an optional but non-limiting implementation, after merging the video frame groups sequentially after the background stream according to the playback order of the video frames in the target viewpoint segment bitstream in each video frame group to obtain the merged bitstream, the panoramic video playback method in this embodiment may further include the following steps:

[0100] When the merged bitstream does not have a preset resolution, the bitstream that has already been used is used to fill in the gaps after each video frame group in the sequential merging process, so that a merged bitstream of the preset resolution size can be obtained after filling in the gaps.

[0101] See Figure 8 The system detects whether the merged background bitstream and the video frames in each video frame group have been merged into a merged bitstream of a preset resolution. If they cannot be merged into a merged bitstream of the preset resolution, the system pads the end of each sequentially merged video frame group with the already used bitstream to obtain a merged bitstream of the preset resolution after padding. For example, ... Figure 8 As shown, if the number of target view segments is insufficient to stitch together a complete 4K image into a merged bitstream, then target view segments corresponding to repeated target view segments are used for padding. This can be understood as follows: if the resolution of a monocular 4K merged bitstream is 3840x1920, then for a panoramic 3D binocular 8K image, the merged bitstream resolution is 3840x3840, which is twice that of a monocular image.

[0102] Using the above method, the downloaded viewpoint segment bitstreams are merged into a new bitstream of one resolution and then sent to a decoder for decoding. Since the size of the transmitted bitstream is fixed, in order to match the bitstream decoding of the subsequent decoder, if there is insufficient number of selected viewpoint segments, resulting in insufficient number of selected viewpoint segments to stitch together a complete 4K image, then duplicate viewpoint segments are used to fill in the gaps.

[0103] S640 decodes and renders the merged bitstream for panoramic video playback.

[0104] According to the panoramic video playback scheme in this embodiment, a high-definition local viewpoint segmented bitstream and a low-definition background bitstream are merged to obtain a new bitstream, which is then fed into a single decoder for decoding. This not only reduces the transmission bitrate of the panoramic video, but also ensures clear video within the viewing angle by using the local high-definition bitstream in the merged bitstream, while the background bitstream ensures continuous display of the panoramic video. Furthermore, the bitstream merging creates a single bitstream that can be decoded normally with a limited number of decoders, reducing decoding costs.

[0105] As an optional but non-limiting implementation, downloading the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream may include steps C1-C2:

[0106] Step C1: When a video viewing perspective switching event is triggered, if there is an overlapping view segment between the previous view segment before the switch and the target view segment after the switch, download the newly added view segment bitstream corresponding to the newly added view segment in the panoramic view video bitstream. The newly added view segment includes the remaining view segments in the target view segment covered in the panoramic view after the video viewing perspective switch, excluding the overlapping view segments.

[0107] See Figure 9 When a video viewing perspective switching event is triggered, and the viewing perspective changes from the previous viewpoint segment (e.g., 0, 1, 2, 3, 4) at time t0 to the target viewpoint segment (e.g., 3, 4, 5, 6, 7) at time t1, it is easy to see that there are overlapping viewpoint segments (e.g., 3, 4) between the previous viewpoint segment and the target viewpoint segment. Before the viewing perspective switching, the overlapping viewpoint segment bitstream corresponding to the overlapping viewpoint segment has already been downloaded. Therefore, when the video viewing perspective switching occurs, the newly added viewpoint segment bitstream corresponding to the newly added viewpoint segment in the panoramic view video bitstream can be downloaded, and the overlapping viewpoint segment bitstream corresponding to the overlapping viewpoint segment will not be downloaded again, thus minimizing the download bitrate.

[0108] Step C2: Using the video frames in the newly added view segment corresponding to the newly added view segment, replace the video frames in the merged bitstream corresponding to the view segment to be replaced at the same time in the previous view segment to obtain the merged bitstream corresponding to the target view segment after the viewing view is switched. The view segments to be replaced include the remaining view segments in the previous view segment covered by the panoramic view before the video viewing view is switched, excluding the overlapping view segments.

[0109] See Figure 10aThe previous viewpoint segment's bitstream and the background bitstream have already been merged once. When the viewing perspective changes and a new viewpoint segment's bitstream is downloaded, the bitstream merging is not repeated. Instead, a flexible tile order algorithm is used. Video frames from the new viewpoint segment's bitstream directly replace the video frames in the previous viewpoint segment's merged bitstream at the same time. Video frames from overlapping viewpoint segments in the previous viewpoint segment's merged bitstream remain unchanged to obtain the merged bitstream after the viewing perspective change. The flexible tile order algorithm minimizes the download bitrate.

[0110] For example, see Figure 10a The video frames in the bitstream corresponding to the newly added viewpoint segments 5, 6, and 7 at time t1 are used to replace the video frames in the bitstream corresponding to the viewpoint segments 0, 1, and 2 at time t0. The video frames in the bitstream corresponding to the overlapping viewpoint segments 3 and 4 in the merged bitstream corresponding to the previous viewpoint segment remain unchanged, resulting in a merged bitstream after viewpoint segment rearrangement, including the bitstreams corresponding to the newly added viewpoint segments 5, 6, and 7 and the bitstreams corresponding to the overlapping viewpoint segments 3 and 4, showing the viewpoint switching.

[0111] Understandably, see Figure 10b Before the video viewing perspective switching event is triggered and the merged bitstream corresponding to the target view segment after the viewing perspective switch is obtained, the background bitstream can be temporarily used to replace the target view segment bitstream corresponding to the target view segment for panoramic video playback.

[0112] As an optional but non-limiting implementation, before merging the target view segmented bitstream and the background bitstream to obtain the merged bitstream, steps D1-D2 may also be included:

[0113] Step D1: When the video viewing perspective switching event is triggered, perform pure random access frame detection on the video frames in the target view slice bitstream corresponding to the target view slice.

[0114] Step D2: Replace the detected pure random access frames with breakpoint connection access frames.

[0115] See Figure 11Broken Link Access (BLA) frames can be considered a special case of Clean Random Access (CRA) frames during video stream switching. When switching from one video stream to another to continue decoding, the CRA frame is directly connected to a CRA frame in the other video stream, and the latter is the BLA.

[0116] If the decoder randomly accesses data from a CRA frame, subsequent frames in the display order will not be decoded due to the lack of a reference frame, and these images will be discarded by the decoder. Data not accessed from the current CRA frame, however, can be decoded and displayed normally. Because this data may be discarded, other images cannot reference it; otherwise, if these images are discarded, more images will be affected and unable to be decoded correctly.

[0117] See Figure 11 When a video viewing perspective switching event is triggered, if it is detected that the target viewpoint segment bitstream and the background bitstream need to be merged, CRA frame detection is performed on the video frames within the target viewpoint segment bitstream corresponding to the target viewpoint segment. If a CRA frame is detected within the target viewpoint segment bitstream, it can be directly replaced with a BLA frame. Using this method, CRA frames are detected and replaced during viewpoint merging. When the stream is not switched, CRA frames will not cause problems when sent to the decoder and do not need to be detected. Only when the stream is switched will CRA frames be replaced with BLA frames to avoid decoding errors.

[0118] As an optional but non-limiting implementation, downloading the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream may also include the following steps E1-E2:

[0119] Step E1: When the video viewing perspective switching event is triggered, and the target perspective segment is switched from the previous perspective segment to use for panoramic video viewing, the target perspective segment bitstream is downloaded from the multiple pre-deployed bitstream segments corresponding to the target perspective segment.

[0120] In this context, the keyframes included in the image frame group in the pre-deployed multiple block bitstreams corresponding to each view segment are misaligned.

[0121] Step E2: Merge the downloaded target view segment bitstream with the background bitstream.

[0122] See Figure 12The panoramic view of a video is pre-divided into multiple view segments. Each view segment in the panoramic video stream stores multiple keyframes with misaligned positions within its corresponding viewpoint. In other words, the target view segment corresponds to multiple pre-deployed keyframe-misaligned block streams. When a video viewing perspective switching event is triggered, and the viewing perspective is switched from the previous view segment to the target view segment for panoramic viewing, the principle of downloading the closest keyframe stream can be followed. The target view segment's keyframe with the closest keyframe to the switching point is downloaded from the pre-deployed block streams corresponding to the target view segment. This allows for a quick switching process without requiring a prolonged display of the target view segment's video footage using the background stream.

[0123] Figure 13 This is a flowchart illustrating another panoramic video playback method provided by this disclosure. The technical solution of this embodiment further optimizes the decoding and rendering process of the merged bitstream in the aforementioned embodiments based on the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 13 As shown, the panoramic video playback method provided in this embodiment may include the following steps S1310-S1350:

[0124] S1310. Download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the clarity of the target view segment bitstream is higher than that of the background bitstream.

[0125] S1320. Merge the target view segmented bitstream and the background bitstream to obtain a merged bitstream.

[0126] S1330. Decode the merged bitstream to obtain video frames in the background bitstream and video frames in the target view slice bitstream corresponding to the target view slice.

[0127] The merged bitstream is decoded by a single decoder, and the output is the video frames in the background bitstream and the video frames in the target view slice bitstream corresponding to each target view slice.

[0128] S1340. Upsample the video frames in the background bitstream to obtain the upsampled video frames in the background bitstream.

[0129] See Figure 14 The video frames within the decoded background bitstream are scaled proportionally to obtain scaled background bitstream video frames. These scaled background bitstream video frames are then subjected to super-resolution and upsampling processes to obtain super-resolution and upsampling background bitstream video frames.

[0130] S1350: Replace and render a portion of the video frames in the upsampled background bitstream with video frames from the target viewpoint bitstream at the same time for panoramic video playback.

[0131] See Figure 14 and Figure 15 This yields the video frames within the background bitstream after super-resolution and upsampling. These video frames within the background bitstream ( Figure 15 The background is rendered using the video frames corresponding to the dashed squares. Simultaneously, it can find video frames from the background bitstream that play at the same time as the video frames in the target viewpoint segmented bitstream, and use the video frames from the target viewpoint segmented bitstream (e.g., ...) Figure 15 The video frames corresponding to o1, o2, o3, and o4 in the video stream are replaced or rendered to cover them. This way, the video frames in the panoramic video stream can be obtained, where the high-definition video frame is near the viewing angle and the low-definition video frame is in other positions.

[0132] As an optional but non-limiting implementation, replacing and rendering a portion of the video frames in the upsampled background bitstream with video frames from the target viewpoint bitstream at the same time can include the following steps F1-F2:

[0133] Step F1: Based on the rendering mapping information obtained after decoding the merged bitstream, the video frames in the target viewpoint segment bitstream are mapped to the video frames in the background bitstream at the same time.

[0134] Step F2: Based on position mapping, replace some video frames in the original background video stream with video frames in the target viewpoint segmented bitstream for rendering. The rendering mapping information is used to describe the position mapping relationship between the video frames in the segmented bitstream and the video frames in the background bitstream.

[0135] See Figure 14 and Figure 15 After video frames are decoded, the video frames in the background bitstream and the video frames in the target viewpoint segment bitstream are rendered into a single source texture. Furthermore, after decoding, a set of mapping information corresponding to that video frame is generated. This mapping information includes the segmentation information of the background region and the viewpoint segment within the video frame. During rendering, the video frame locates the corresponding mapping information and renders the background and viewpoint segments accordingly.

[0136] See Figure 14 and Figure 15The mapping information includes the segmentation information of the decoded video frames to determine the region boundary information of each viewpoint segment in the source texture. Additionally, an index array indicates whether a certain block in the source texture needs to be rendered, and the index of the position to be rendered. The segmentation information of the background and viewpoint segments is used to determine the region information for rendering each block on the target texture. Based on the above information, each block in the source texture can be rendered to the specified position on the target texture. (Mapping method illustration follows.) Figure 14 As shown.

[0137] According to the panoramic video playback scheme in this embodiment, the transmission bitrate of the panoramic video is reduced by merging a high-definition local viewpoint segmented bitstream and a low-definition background bitstream. The local high-definition bitstream in the merged bitstream ensures that the video is clear within the viewing angle range, while the background bitstream ensures that the panoramic video is continuously displayed. In addition, the bitstream merging can form a single bitstream, which can be decoded normally by a limited number of decoders, reducing decoding costs.

[0138] Figure 16 This is a structural block diagram of a panoramic video playback device provided in an embodiment of this disclosure. This embodiment is applicable to situations involving the playback of panoramic videos. The device can be implemented in software and / or hardware and is generally integrated into any electronic device with network communication capabilities, including but not limited to mobile terminals, PCs, or servers. Figure 16 As shown, the panoramic video playback device of this embodiment may include: a download module 1610, a merging module 1620, and a playback module 1630. Wherein:

[0139] Download module 1610 is used to download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0140] The merging module 1620 is used to merge the target view segmented bitstream and the background bitstream to obtain a merged bitstream;

[0141] The playback module 1630 is used to decode and render the merged bitstream for panoramic video playback.

[0142] As an optional but non-limiting implementation, the target view segment bitstream corresponding to the target view segment in the downloaded panoramic view video bitstream includes:

[0143] The target view segment is determined from multiple view segments that have been pre-divided into panoramic view segments of the panoramic view video.

[0144] Download the target view segment bitstream corresponding to the target view segment from the panoramic view video stream in playback order. The target view segment bitstream is a video stream with a preset segment length.

[0145] As an optional but non-limiting implementation, determining the target view segment from multiple view segments pre-divided into panoramic view segments of the panoramic view video includes:

[0146] Based on the viewing posture of the panoramic video and the field of view parameters of the panoramic video display device, the target viewing angle in the panoramic view video stream is determined.

[0147] The target view segment is determined from multiple view segments pre-divided into panoramic view segments of the panoramic view video. The target view segment includes the pre-divided view segments covered by the target viewing angle in the panoramic view.

[0148] As an optional but non-limiting implementation, the target view segment bitstream and background bitstream corresponding to the target view segment in the downloaded panoramic view video bitstream include:

[0149] Download the background bitstream, which is used to generate the merged bitstream when the target view segment bitstream fails to be downloaded;

[0150] Download the target view segment bitstream corresponding to the target view segment, wherein the background bitstream is downloaded first compared to the target view segment bitstream corresponding to the target view segment.

[0151] As an optional but non-limiting implementation, the preset segmentation length includes a preset integer multiple of image frames.

[0152] As an optional but non-limiting implementation, merging the target view segment bitstream and the background bitstream to obtain a merged bitstream includes:

[0153] The video frames in multiple target view slice streams that are played at the same time in each target view slice are merged into the same video frame group. Each slice stream includes multiple video frames that are consecutive in the temporal domain.

[0154] According to the playback order of video frames in the target viewpoint segment bitstream of each video frame group, the video frame groups are sequentially merged into the background bitstream to obtain the merged bitstream.

[0155] As an optional but non-limiting implementation, after merging the video frame groups sequentially after the background bitstream according to the playback order of the video frames in the target viewpoint segment bitstream in each video frame group to obtain the merged bitstream, the method further includes:

[0156] When the merged bitstream does not have a preset resolution, the bitstream that has already been used is used to fill in the gaps after each video frame group in the sequential merging process, so that a merged bitstream of the preset resolution size can be obtained after filling in the gaps.

[0157] As an optional but non-limiting implementation, the target view segment bitstream corresponding to the target view segment in the downloaded panoramic view video bitstream includes:

[0158] When a video viewing perspective switching event is triggered, if there is an overlapping view segment between the previous view segment before the switch and the target view segment after the switch, the newly added view segment bitstream corresponding to the newly added view segment in the panoramic view video bitstream is downloaded. The newly added view segment includes the remaining view segments in the target view segment covered in the panoramic view after the video viewing perspective is switched, excluding the overlapping view segments.

[0159] The video frames in the newly added view segment bitstream corresponding to the newly added view segment are used to replace the video frames in the merged bitstream corresponding to the view segment to be replaced at the same time in the previous view segment, so as to obtain the merged bitstream corresponding to the target view segment after the viewing view is switched. The view segment to be replaced includes the remaining view segments in the previous view segment covered by the panoramic view before the video viewing view is switched, excluding the overlapping view segments.

[0160] As an optional but non-limiting implementation, before merging the target view segment bitstream and the background bitstream to obtain the merged bitstream, the method further includes:

[0161] When a video viewing perspective switching event is triggered, a purely random access frame detection is performed on the video frames in the target view slice bitstream corresponding to the target view slice.

[0162] Replace the detected pure random access frames with breakpoint connection access frames.

[0163] As an optional but non-limiting implementation, the download of the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream further includes:

[0164] When a video viewing perspective switching event is triggered, and the target viewing perspective segment is switched from the previous perspective segment to a panoramic viewing perspective segment, the target viewing perspective segment bitstream is downloaded from multiple pre-deployed bitstream segments corresponding to the target viewing perspective segment.

[0165] The downloaded target view segment bitstream is merged with the background bitstream;

[0166] In this context, the keyframes included in the image frame group in the pre-deployed multiple block bitstreams corresponding to each view segment are misaligned.

[0167] As an optional but non-limiting implementation, the decoding and rendering of the merged bitstream includes:

[0168] Decode the merged bitstream to obtain video frames in the background bitstream and video frames in the target view slice bitstream corresponding to the target view slice.

[0169] Upsample the video frames in the background bitstream to obtain the upsampled video frames in the background bitstream.

[0170] A portion of the video frames in the upsampled background bitstream are replaced and rendered using video frames from the target viewpoint bitstream at the same time.

[0171] As an optional but non-limiting implementation, replacing and rendering a portion of the video frames in the upsampled background bitstream with video frames from the target viewpoint segment bitstream at the same time includes:

[0172] Based on the rendering mapping information obtained after decoding the merged bitstream, the video frames in the target viewpoint segment bitstream are mapped to the video frames in the background bitstream at the same time.

[0173] Based on the position mapping, the video frames in the target viewpoint segmented bitstream are used to replace some video frames in the original background bitstream for rendering. The rendering mapping information is used to describe the position mapping relationship between the video frames in the segmented bitstream and the video frames in the background bitstream.

[0174] The panoramic video playback device provided in this embodiment can execute the panoramic video playback method provided in any of the above embodiments of this disclosure, and has the corresponding functions and beneficial effects of executing the panoramic video playback method. For details, please refer to the relevant operations of the panoramic video playback method in the foregoing embodiments.

[0175] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0176] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 17 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 17The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 17 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0177] like Figure 17 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0178] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 17 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0179] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the panoramic video playback method shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the panoramic video playback method of embodiments of this disclosure.

[0180] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0181] The electronic device provided in this embodiment and the panoramic video playback method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0182] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the panoramic video playback method provided in the above embodiments.

[0183] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0184] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0185] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0186] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0187] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream, wherein the background bitstream is a panoramic view video bitstream and the resolution of the target view segment bitstream is higher than that of the background bitstream; merge the target view segment bitstream and the background bitstream to obtain a merged bitstream; and decode and render the merged bitstream for panoramic video playback.

[0188] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0190] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0191] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0192] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0193] According to one or more embodiments of this disclosure, Example 1 provides a panoramic video playback method, the method comprising:

[0194] Download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0195] The target view segmented bitstream and the background bitstream are merged to obtain a merged bitstream;

[0196] The merged bitstream is decoded and rendered for panoramic video playback.

[0197] According to one or more embodiments of this disclosure, Example 2, based on the method described in Example 1, includes downloading the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream, comprising:

[0198] The target view segment is determined from multiple view segments that have been pre-divided into panoramic view segments of the panoramic view video.

[0199] Download the target view segment bitstream corresponding to the target view segment from the panoramic view video stream in playback order. The target view segment bitstream is a video stream with a preset segment length.

[0200] According to one or more embodiments of this disclosure, Example 3, based on the method of Example 2, includes determining the target view segment from a plurality of view segments pre-divided for the panoramic view video, comprising:

[0201] Based on the viewing posture of the panoramic video and the field of view parameters of the panoramic video display device, the target viewing angle in the panoramic view video stream is determined.

[0202] The target view segment is determined from multiple view segments pre-divided into panoramic view segments of the panoramic view video. The target view segment includes the pre-divided view segments covered by the target viewing angle in the panoramic view.

[0203] According to one or more embodiments of this disclosure, Example 4, based on any one of Examples 1-3, involves downloading the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream, including:

[0204] Download the background bitstream, which is used to generate the merged bitstream when the target view segment bitstream fails to be downloaded;

[0205] Download the target view segment bitstream corresponding to the target view segment, wherein the background bitstream is downloaded first compared to the target view segment bitstream corresponding to the target view segment.

[0206] According to one or more embodiments of this disclosure, Example 5 describes the method according to Example 2, wherein the preset segmentation length includes a preset integer multiple of image frame groups.

[0207] According to one or more embodiments of this disclosure, Example 6 describes the method described in Example 1, wherein merging the target view segmented bitstream and the background bitstream to obtain a merged bitstream includes:

[0208] The video frames in multiple target view slice streams that are played at the same time in each target view slice are merged into the same video frame group. Each slice stream includes multiple video frames that are consecutive in the temporal domain.

[0209] According to the playback order of video frames in the target viewpoint segment bitstream of each video frame group, the video frame groups are sequentially merged into the background bitstream to obtain the merged bitstream.

[0210] According to one or more embodiments of this disclosure, Example 7, based on the method described in Example 6, after sequentially merging the video frame groups after the background bitstream according to the playback order of the video frames in the target viewpoint segment bitstream in each video frame group to obtain a merged bitstream, the method further includes:

[0211] When the merged bitstream does not have a preset resolution, the bitstream that has already been used is used to fill in the gaps after each video frame group in the sequential merging process, so that a merged bitstream of the preset resolution size can be obtained after filling in the gaps.

[0212] According to one or more embodiments of this disclosure, Example 8, based on the method described in Example 6, involves downloading the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream, including:

[0213] When a video viewing perspective switching event is triggered, if there is an overlapping view segment between the previous view segment before the switch and the target view segment after the switch, the newly added view segment bitstream corresponding to the newly added view segment in the panoramic view video bitstream is downloaded. The newly added view segment includes the remaining view segments in the target view segment covered in the panoramic view after the video viewing perspective is switched, excluding the overlapping view segments.

[0214] The video frames in the newly added view segment bitstream corresponding to the newly added view segment are used to replace the video frames in the merged bitstream corresponding to the view segment to be replaced at the same time in the previous view segment, so as to obtain the merged bitstream corresponding to the target view segment after the viewing view is switched. The view segment to be replaced includes the remaining view segments in the previous view segment covered by the panoramic view before the video viewing view is switched, excluding the overlapping view segments.

[0215] According to one or more embodiments of this disclosure, Example 9, based on the method of Example 6, further includes, before merging the target view segmented bitstream and the background bitstream to obtain the merged bitstream:

[0216] When a video viewing perspective switching event is triggered, a purely random access frame detection is performed on the video frames in the target view slice bitstream corresponding to the target view slice.

[0217] Replace the detected pure random access frames with breakpoint connection access frames.

[0218] According to one or more embodiments of this disclosure, Example 10, based on the method described in Example 6, further includes downloading the target view segment bitstream corresponding to the target view segment in the panoramic view video bitstream, as follows:

[0219] When a video viewing perspective switching event is triggered, and the target viewing perspective segment is switched from the previous perspective segment to a panoramic viewing perspective segment, the target viewing perspective segment bitstream is downloaded from multiple pre-deployed bitstream segments corresponding to the target viewing perspective segment.

[0220] The downloaded target view segment bitstream is merged with the background bitstream;

[0221] In this context, the keyframes included in the image frame group in the pre-deployed multiple block bitstreams corresponding to each view segment are misaligned.

[0222] According to one or more embodiments of this disclosure, Example 11, based on the method described in Example 1, includes decoding and rendering the merged bitstream, comprising:

[0223] Decode the merged bitstream to obtain video frames in the background bitstream and video frames in the target view slice bitstream corresponding to the target view slice.

[0224] Upsample the video frames in the background bitstream to obtain the upsampled video frames in the background bitstream.

[0225] A portion of the video frames in the upsampled background bitstream are replaced and rendered using video frames from the target viewpoint bitstream at the same time.

[0226] According to one or more embodiments of this disclosure, Example 12, based on the method described in Example 11, includes replacing and rendering a portion of video frames in the upsampled background bitstream video frames with video frames in the target viewpoint segment bitstream at the same time, comprising:

[0227] Based on the rendering mapping information obtained after decoding the merged bitstream, the video frames in the target viewpoint segment bitstream are mapped to the video frames in the background bitstream at the same time.

[0228] Based on the position mapping, the video frames in the target viewpoint segmented bitstream are used to replace some video frames in the original background bitstream for rendering. The rendering mapping information is used to describe the position mapping relationship between the video frames in the segmented bitstream and the video frames in the background bitstream.

[0229] According to one or more embodiments of this disclosure, Example 13 provides a panoramic video playback device, the device comprising:

[0230] The download module is used to download the target view segment bitstream and background bitstream corresponding to the target view segment in the panoramic view video bitstream. The background bitstream is the panoramic view video bitstream, and the resolution of the target view segment bitstream is higher than that of the background bitstream.

[0231] The merging module is used to merge the target view segment bitstream and the background bitstream to obtain a merged bitstream;

[0232] The playback module is used to decode and render the merged bitstream for panoramic video playback.

[0233] According to one or more embodiments of this disclosure, Example 14 provides an electronic device, the electronic device comprising:

[0234] One or more processors;

[0235] Storage device for storing one or more programs.

[0236] When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic video playback method as described in any of Examples 1-12.

[0237] According to one or more embodiments of the present disclosure, Example 15 provides a storage medium containing computer-executable instructions that, when executed by a computer processor, are used to perform a panoramic video playback method as described in any of Examples 1-12.

[0238] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0239] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0240] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A panoramic video playing method, the method comprising: downloading a target view tile code stream corresponding to a target view slice in a panoramic view video code stream and a background code stream, the background code stream being a panoramic view video code stream, and the target view tile code stream having a higher definition than the background code stream; merging the target view tile code stream and the background code stream to obtain a merged code stream, wherein the merging of the target view tile code stream and the background code stream to obtain the merged code stream comprises: merging video frames in a plurality of target view tile code streams corresponding to the same time playing of each target view slice in the target view tile code stream into the same video frame packet, each tile code stream comprising a plurality of video frames that are continuous in time domain; and merging each video frame packet in sequence according to a playing order of the video frames in the target view tile code stream in each video frame packet to the rear side of the background code stream to obtain the merged code stream; decoding and rendering the merged code stream for panoramic video playing. 2.The method of claim 1, wherein the downloading of the target view tile code stream corresponding to the target view slice in the panoramic view video code stream comprises: determining a target view slice from a plurality of view slices pre-divided according to a panoramic view of the panoramic view video; downloading the target view tile code stream corresponding to the target view slice in sequence according to a playing order from the panoramic view video code stream, the target view tile code stream being a video code stream of a preset segmentation length. 3.The method of claim 2, wherein the determining of the target view slice from the plurality of view slices pre-divided according to the panoramic view of the panoramic view video comprises: determining a target viewing view in the panoramic view of the panoramic view video code stream according to a video viewing posture of the panoramic view video and a field of view angle parameter of a panoramic video display device; and determining the target view slice from the plurality of view slices pre-divided according to the panoramic view of the panoramic view video, the target view slice comprising a pre-divided view slice covered by the target viewing view in the panoramic view. 4.The method of any one of claims 1-3, wherein the downloading of the target view tile code stream corresponding to the target view slice in the panoramic view video code stream and the background code stream comprises: downloading the background code stream for generating the merged code stream when the downloading of the target view tile code stream fails; and downloading the target view tile code stream corresponding to the target view slice, wherein the background code stream is preferentially downloaded compared to the target view tile code stream corresponding to the target view slice. 5.The method of claim 2, wherein the preset segmentation length comprises a preset integer multiple of image groups of pictures. 6.The method of claim 1, after the merging of each video frame packet in sequence according to the playing order of the video frames in the target view tile code stream in each video frame packet to the rear side of the background code stream to obtain the merged code stream, the method further comprises: when the merged code stream does not have a preset resolution, using an already used code stream to fill in the rear side of each video frame packet merged in sequence to obtain the merged code stream of the preset resolution after the filling.

7. The method of claim 1, wherein the downloading the target view tile code stream corresponding to the target view slice in the panoramic view video code stream comprises: when a video viewing view switching event is triggered, and there is an overlapping view slice between a previous view slice before the switching and a target view slice after the switching, downloading a new view tile code stream corresponding to a new view slice in the panoramic view video code stream, the new view slice comprising remaining view slices in the target view slice covered in the panoramic view after the video viewing view switching, except for the overlapping view slice.

8. The method of claim 1, wherein the combining the target view tile code stream and the background code stream to obtain the combined code stream further comprises: when the video viewing view switching event is triggered, performing pure random access frame detection on video frames in the target view tile code stream corresponding to the target view slice.

9. The method of claim 1, wherein the downloading the target view tile code stream corresponding to the target view slice in the panoramic view video code stream further comprises: when the video viewing view switching event is triggered, and the panoramic view video is viewed using the target view slice from the previous view slice, downloading the target view tile code stream from a plurality of pre-deployed tile code streams corresponding to the target view slice.

10. The method of claim 1, wherein the decoding and rendering the combined code stream comprises: decoding the combined code stream to obtain video frames in the background code stream and video frames in the target view tile code stream corresponding to the target view slice.

11. The method of claim 10, wherein the replacing and rendering part of the video frames in the up-sampled background code stream with the video frames in the target view tile code stream at the same time comprises: mapping the video frames in the target view tile code stream to the video frames in the background code stream at the same time according to rendering mapping information obtained after the decoding of the combined code stream. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The video frames in the target view cutting code stream are used to replace part of the video frames in the original background code stream based on position mapping, and the rendering mapping information is used to describe the position mapping relationship between the video frames in the cutting code stream and the video frames in the background code stream.

12. A panoramic video playing device, the device comprising: a downloading module configured to download a target view cutting code stream corresponding to a target view slice in a panoramic view video code stream and a background code stream, the background code stream being a panoramic view video code stream, and the target view cutting code stream having a higher definition than the background code stream; a merging module configured to merge the target view cutting code stream and the background code stream to obtain a merged code stream, wherein the merging of the target view cutting code stream and the background code stream to obtain the merged code stream comprises: merging video frames in a plurality of target view cutting code streams corresponding to the same time playing of each target view slice in the target view cutting code stream into the same video frame packet, each cutting code stream comprising a plurality of video frames that are continuous in time domain; and sequentially merging each video frame packet to the rear side of the background code stream according to the playing order of the video frames in the target view cutting code stream in each video frame packet, to obtain the merged code stream; a playing module configured to decode and render the merged code stream to play the panoramic video.

13. An electronic device, the electronic device comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic video playing method according to any one of claims 1-11.

14. A storage medium containing computer executable instructions, when executed by a computer processor, for performing the panoramic video playing method according to any one of claims 1-11.

Images