A video playback method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application provide a video playback method, device, equipment, storage medium and program product, which are applied to the field of audio and video technology. The method includes: based on the viewing angle information corresponding to the original frame image, selecting N matching image blocks from the original frame image, and obtaining the compressed data of each of the N image blocks. Then, the compressed data of each of the N image blocks is merged to obtain merged compressed data; in this way, when decoding the video, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data, which greatly reduces the number of hardware decoders required, and correspondingly reduces the requirements for video playback devices, so that more types of devices can achieve high-resolution video playback and expand the video playback scenarios. When rendering the video, the image blocks are rendered and played according to the original position information of each image block in the original frame image, which ensures the accuracy of image block rendering.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio and video technology, and in particular to a video playback method, device, equipment, storage medium, and program product. Background Art

[0002] As virtual reality (VR) technology continues to update and iterate, the user experience of VR devices is also constantly improving. Among them, video playback is a common application scenario of VR devices. With the continuous update of video encoding and decoding technology and the gradual improvement of video resolution, higher requirements are also placed on the video playback performance of VR devices.

[0003] In related technologies, the server divides the frame image in the video into multiple image blocks, and then encodes each image block independently and transmits it to the VR device, and the VR device decodes and renders each image block independently.

[0004] Since there are a large number of image blocks obtained by segmentation, and each image block requires a separate hardware decoder for decoding, this places high requirements on VR devices. In addition, some VR devices have restrictions on the number of configured hardware decoders, so it is difficult to meet the requirement of assigning a separate hardware decoder to each image block for decoding. In this case, for image blocks that are not assigned a hardware decoder, software decoding can only be forced, and the efficiency of software decoding is low, which may cause freezes and other situations during video playback, thereby affecting the video playback effect. Summary of the invention

[0005] The embodiments of the present application provide a video playback method, apparatus, device, storage medium and program product for reducing the requirements of video playback on hardware devices, improving the efficiency of video decoding, and thereby improving the effect of video playback.

[0006] On the one hand, an embodiment of the present application provides a video playback method, the method comprising:

[0007] Based on viewing angle information corresponding to the original frame image, select N image blocks from the original frame image, and obtain compressed data of each of the N image blocks, where N is greater than 1;

[0008] Merging the compressed data of each of the N image blocks to obtain merged compressed data;

[0009] Decoding the combined compressed data using a hardware decoder to obtain combined original data; the combined original data includes: original data of each of the N image blocks;

[0010] For the N image blocks, the following operations are performed respectively: according to the original position information of the image block in the original frame image, the original data of the image block is rendered and played.

[0011] On the one hand, an embodiment of the present application provides a video playback device, the device comprising:

[0012] A selection module, configured to select N image blocks from the original frame image based on viewing angle information corresponding to the original frame image, and obtain compressed data of each of the N image blocks, where N is greater than 1;

[0013] A merging module, used for merging the compressed data of each of the N image blocks to obtain merged compressed data;

[0014] A decoding module, used to decode the combined compressed data using a hardware decoder to obtain combined original data; the combined original data includes: original data of each of the N image blocks;

[0015] The rendering module is used to perform the following operations respectively for the N image blocks: according to the original position information of the image block in the original frame image, render and play the original data of the image block.

[0016] Optionally, each image block in the original frame image corresponds to a first index in the original frame image;

[0017] The rendering module is also used for:

[0018] Based on a first index of an image block, the width and height of the original frame image, and the width and height of each image block in the original frame image, original position information of the image block in the original frame image is obtained.

[0019] Optionally, the rendering module is specifically used for:

[0020] Based on the first index of the image block, the width of the original frame image and the width of each image block, obtaining the starting horizontal coordinate and the ending horizontal coordinate of the image block in the original frame image;

[0021] Based on the first index of the image block, the height of the original frame image and the height of each image block, obtaining the starting ordinate and the ending ordinate of the image block in the original frame image;

[0022] Based on the starting horizontal coordinate, the ending horizontal coordinate, the starting vertical coordinate and the ending vertical coordinate, original position information of the one image block in the original frame image is obtained.

[0023] Optionally, the rendering module is specifically used for:

[0024] Acquire original position information of the one image block in the original frame image;

[0025] Adjusting target position information of the one image block in the merged image corresponding to the merged original data to the original position information;

[0026] According to the original position information, the original data of the image block is rendered and played.

[0027] Optionally, the merging module is further used for:

[0028] Merging the compressed data of each of the N image blocks to obtain the merged compressed data, reconstructing indexes for the N image blocks to obtain second indexes for each of the N image blocks, and recording an index mapping relationship between the first index and the second index corresponding to each image block;

[0029] The rendering module is specifically used for:

[0030] Based on the second index corresponding to the one image block and the index mapping relationship, obtaining a first index corresponding to the one image block;

[0031] Based on the first index corresponding to the one image block, original position information of the one image block in the original frame image is obtained.

[0032] Optionally, the rendering module is further used for:

[0033] Based on the second index of an image block, the width and height of the merged image, and the width and height of each image block in the merged image, target position information of the image block in the merged image is obtained.

[0034] Optionally, the merging module is specifically used for:

[0035] Based on a preset splicing template and the first indexes of the N image blocks, the compressed data of the N image blocks are sequentially spliced ​​to obtain merged compressed data.

[0036] Optionally, the selection module is specifically used for:

[0037] Downloading respective encapsulation format data of the N image blocks;

[0038] The encapsulated format data of the N image blocks are respectively decapsulated by using an audio / video separator to obtain compressed data of each of the N image blocks.

[0039] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video playback method when executing the computer program.

[0040] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned video playback method.

[0041] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned video playback method.

[0042] In an embodiment of the present application, based on the viewing angle information corresponding to the original frame image, N matching image blocks are selected from the original frame image, and the compressed data of each of the N image blocks is obtained. Then, the compressed data of each of the N image blocks is merged to obtain merged compressed data. In this way, when decoding the video, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data; compared to assigning a hardware decoder to each image block for decoding, the number of required hardware decoders is greatly reduced, and the requirements for video playback devices are correspondingly reduced, so that more types of devices can achieve high-resolution VR video playback and expand the scenarios of VR video playback.

[0043] Secondly, when the compressed data of N image blocks are combined and decoded, video decoding can be achieved through a hardware decoder, which effectively avoids the situation where software decoding is forced due to insufficient hardware decoder, thereby ensuring the efficiency of video decoding and further improving the effect of video playback; at the same time, during the video decoding process, the occupancy of device resources (such as processor resources, memory resources, etc.) and device temperature are reduced.

[0044] In addition, the present application combines and decodes the compressed data of N image blocks to obtain combined original data, wherein the combined original data includes: the original data of each of the N image blocks. When performing video rendering, the original position information of the image block in the original frame image is restored, and the original data of the image block is rendered and played according to the original position information, thereby ensuring the accuracy of image block rendering, thereby improving the effect of video playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0047] Figure 2 An example of an application scenario provided by the present application Figure 1 ;

[0048] Figure 3 An example of an application scenario provided by the present application Figure 2 ;

[0049] Figure 4 A schematic diagram of a video playback method provided in an embodiment of the present application Figure 1 ;

[0050] Figure 5 A schematic diagram of a process from video acquisition to video playback provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a method for matching image blocks provided in an embodiment of the present application Figure 1 ;

[0052] Figure 7 A schematic diagram of a method for matching image blocks provided in an embodiment of the present application Figure 2 ;

[0053] Figure 8 A schematic diagram of a flow chart of a method for merging compressed data of an image block provided in an embodiment of the present application;

[0054] Fig. 9 A flowchart of a method for building an index provided in an embodiment of the present application;

[0055] Fig.10 A schematic diagram of an original texture corresponding to an original frame image provided in an embodiment of the present application;

[0056] Fig.11 A schematic diagram of a target texture corresponding to a merged image provided in an embodiment of the present application;

[0057] Fig. 12A A schematic diagram of a process flow of a texture coordinate restoration method provided in an embodiment of the present application;

[0058] Fig. 12BA schematic diagram of a video playback method provided in an embodiment of the present application Figure 2 ;

[0059] Fig.13 A schematic diagram of the structure of a video playback device provided in an embodiment of the present application;

[0060] Fig.14 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0063] Virtual Reality (VR) technology is a technology that mainly uses computer technology, and utilizes and integrates multiple technologies such as 3D graphics technology, multimedia technology, simulation technology, and display technology. With the help of computers and other devices, it produces a virtual world with realistic 3D vision, touch, smell and other sensory experiences, so that people in the virtual world can feel as if they are in the real world. VR technology has been widely used, for example, in the fields of video playback and games. Taking VR video playback as an example, VR glasses usually have a built-in VR player, so after the viewer wears VR glasses, he can watch the VR video played by the VR player.

[0064] Video interaction: refers to a way of interacting with the video being played. When implementing video interaction, the content of the video being played will also change accordingly with the interaction process. The interactive action may include a change in the position of the interactive object. The change in the position of the interactive object will bring about a change in the viewing angle of the interactive object. Therefore, the change in video content may be the change in the viewing angle and the switching of the video content that can be viewed. In addition, the interactive object may use interactive props to perform certain actions or issue certain instructions, etc.

[0065] Posture information: The posture information involved in the embodiments of the present application includes object posture information of interactive objects and prop posture information of interactive props. The object posture information includes position information and posture information of the interactive objects. The posture information includes but is not limited to information such as the orientation, viewing angle, sight direction or body movement of the interactive objects. The object posture information can be mapped to the virtual scene, so as to select a matching local video screen (the local video screen includes multiple image blocks) from a panoramic video screen with the mapped object posture information; or, the prop posture information is similar to this, which can express the position information and posture information of the prop, so as to determine what kind of interactive effect the prop can produce, and then select the local video screen associated with the interactive effect from the panoramic video screen.

[0066] Frame image: also called video frame, is a still image in a video, and is the basic unit of a video. A video is composed of continuous video frames, each of which represents a complete image at a certain point in time. Video frames are created by continuously capturing or generating a series of consecutive image frames. During video playback, these video frames are played continuously at a certain rate, thus creating a dynamic video effect.

[0067] Hardware decoder: A dedicated decoding device, usually implemented by a chip or a specialized hardware circuit. It uses a dedicated algorithm and hardware accelerator to decode media files and convert them into a readable data format. The hardware decoder processes and decodes the input binary data through hardware circuits. It can read compressed data directly from the media file and decompress it into the original audio or video stream. The hardware decoder contains a dedicated decoder chip that implements a specific decoding algorithm to efficiently process media files.

[0068] Tile VR: A VR technology based on image blocks, that is, an image is divided into multiple image blocks, each of which is independently encoded, transmitted, decoded, and rendered.

[0069] Texture: Also called a map, it is an image.

[0070] Texture coordinate system: A spatial rectangular coordinate system, with the horizontal axis being the S axis, the vertical axis being the T axis, and the coordinate axis perpendicular to the screen being the R axis. In 2D textures, since there is no R axis, the horizontal axis can also be called the U axis and the vertical axis the V axis, which is also commonly known as the UV coordinate system. Pixels in a texture are usually addressed according to texture coordinates, and the range of 2D texture coordinates is [0, 1].

[0071] It is understandable that in the specific implementation of this application, data related to viewing angle information, posture information, etc. are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0072] Video playback is a common application scenario for VR devices. The Tile VR solution is used for VR video playback in related technologies. Specifically, the server divides the frame image in the video into multiple image blocks, and then encodes each image block independently and transmits it to the VR device, which decodes and renders each image block independently.

[0073] Since there are a large number of image blocks obtained by segmentation, and each image block requires a separate hardware decoder for decoding, this places high requirements on VR devices; and some VR devices have restrictions on the number of configured hardware decoders, so it is difficult to meet the requirement of assigning a separate hardware decoder to each image block for decoding. In this case, for image blocks that are not assigned a hardware decoder, software decoding can only be forced, and the efficiency of software decoding is low, which may cause freezes during video playback, thereby affecting the video playback effect.

[0074] Based on this, an embodiment of the present application provides a video playback method, in which N matching image blocks are first selected from the original frame image based on the viewing angle information corresponding to the original frame image, and the compressed data of each of the N image blocks is obtained. Then, the compressed data of each of the N image blocks is merged to obtain merged compressed data. In this way, when decoding the video, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data; compared to assigning a hardware decoder to each image block for decoding, the number of required hardware decoders is greatly reduced, and the requirements for video playback devices are correspondingly reduced, so that more types of devices can achieve high-resolution VR video playback and expand the scenarios of VR video playback.

[0075] Secondly, when the compressed data of N image blocks are combined and decoded, video decoding can be achieved through a hardware decoder, which effectively avoids the situation where software decoding is forced due to insufficient hardware decoder, thereby ensuring the efficiency of video decoding and further improving the effect of video playback; at the same time, during the video decoding process, the occupancy of device resources (such as processor resources, memory resources, etc.) and device temperature are reduced.

[0076] In addition, the present application combines and decodes the compressed data of N image blocks to obtain combined original data, wherein the combined original data includes: the original data of each of the N image blocks. When performing video rendering, the original position information of the image block in the original frame image is restored, and the original data of the image block is rendered and played according to the original position information, thereby ensuring the accuracy of image block rendering, thereby improving the effect of video playback.

[0077] The following is a brief introduction to the system architecture diagram applicable to the technical solution of the embodiment of the present application. It should be noted that the system architecture diagram introduced below is only used to illustrate the embodiment of the present application and is not a limitation.

[0078] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and servers 102.

[0079] The terminal device 101 has a video playback function, wherein the terminal device 101 can be an extended reality (XR) device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, an intelligent voice interaction device, an intelligent vehicle-mounted device, etc., but is not limited thereto.

[0080] Among them, XR includes various forms such as Augmented Reality (AR), VR, and Mixed Reality (MR); accordingly, XR devices can be AR devices, VR devices, MR devices, etc.

[0081] When the terminal device 101 is a VR device, the VR device can be worn by the wearable object to realize the VR function, and the VR device 101 is any device that can realize the VR function. For example, it can be one or a combination of VR glasses, VR handles, VR head-mounted display devices (referred to as VR helmets or VR head displays), etc.

[0082] Server 102 is a background server corresponding to the video playback function. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0083] The video method in the embodiment of the present application can be executed by the terminal device 101 or the server 102 alone, or can be executed by the terminal device 101 and the server 102 together.

[0084] The server 102 and the terminal device 101 may each include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 may also be configured with a database, which may be used for video data, etc. The memories of the server 102 and the terminal device 101 may also store program instructions required to be executed in the video playback method provided in the embodiment of the present application, and these program instructions may be used to implement the video playback process provided in the embodiment of the present application when executed by the processor.

[0085] It should be noted that when the video playback method provided in the embodiment of the present application is executed by the server 102 or the terminal device 101 alone, the system architecture of the present application may also include only a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may be considered to be the same device. Of course, in actual application, when the video playback method provided in the embodiment of the present application is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0086] In the embodiment of the present application, the terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more networks. The network may be a wired network or a wireless network, for example, the wireless network may be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, other possible networks, which are not limited in the embodiment of the present application.

[0087] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0088] The solution provided in the embodiment of the present application can be applied to video playback in various application scenarios, such as VR video playback, live video playback, game video playback, short video playback, etc. The solution can also be applied as a basic technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0089] The following is an illustrative description of the application scenarios to which the solution provided in this application is applicable.

[0090] Application scenario 1: VR video playback.

[0091] VR video playback generally maps each frame image to a spherical surface for display. The human eye is at the center of the sphere and can see a local area on the sphere surface each time. During the rotation, a 360-degree panoramic image (i.e., the entire frame image) on the sphere surface can be seen. Therefore, during video playback, for the local area visible to the human eye, a high-definition local image can be rendered, and for the area that the human eye cannot see, a low-definition local image can be rendered without rendering or with rendering.

[0092] Based on this, see Figure 2 The server saves the video to be played in advance. Specifically, for each original frame image in the video to be played, the server divides the original frame image into multiple image blocks, and independently encodes each image block and saves the compressed data obtained by encoding.

[0093] When the interactive object wearing the VR device performs video interaction, the VR device detects the object posture information of the interactive object and determines the viewing angle information of the interactive object based on the object posture information. Select multiple image blocks that match the viewing angle information from the multiple image blocks contained in the original frame image, that is, select the image blocks in the local area that the interactive object can currently see. The VR device obtains the compressed data of multiple image blocks from the server. Then, the compressed data of each of the multiple image blocks is merged to obtain merged compressed data; a hardware decoder is used to decode the merged compressed data to obtain merged original data. Finally, according to the original position information of each image block in the original frame image, the original data of the image block is rendered, and the rendered image block is mapped to the corresponding position on the surface of the sphere for display, so that the interactive object can see the local image composed of multiple image blocks.

[0094] Application scenario 2: Game video playback.

[0095] See also Figure 3 Specifically, for each original frame image in the game video, the server divides the original frame image into multiple image blocks, and independently encodes each image block and saves the compressed data obtained by encoding.

[0096] The terminal device responds to the operation of attacking the enemy behind the game character 302 triggered in the first game screen 301, and determines the viewing angle information of the game character 302, that is, the viewing angle after the game character 302 is turned, which includes the enemies that need to be attacked. Select N image blocks that match the viewing angle information from the multiple image blocks contained in the original frame image, and obtain the compressed data of the N image blocks from the server. Then merge the compressed data of each of the N image blocks to obtain merged compressed data. Use a hardware decoder to decode the merged compressed data to obtain merged original data. Finally, according to the original position information of each image block in the original frame image, render and display the original data of each image block, so that the game application can display the second game screen 303 after the game character is turned.

[0097] It should be noted that the embodiments of the present application are not limited to the several application scenarios exemplified above, but may also be other application scenarios, and the present application does not make any specific limitation on this.

[0098] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a process of a video playback method, such as Figure 4 As shown, the process of the method is executed by a computer device (also referred to as a device), which may be Figure 1 The terminal device 101 and / or server 102 shown includes the following steps:

[0099] Step 401 : based on the viewing angle information corresponding to the original frame image, select N image blocks from the original frame image, and obtain compressed data of each of the N image blocks, where N is greater than 1.

[0100] Specifically, a frame image is also called a video frame. A video is composed of multiple continuous video frames, each of which represents a complete image of the video at a certain time point. Multiple video frames in a video are played continuously at a certain rate, thereby producing a dynamic video effect.

[0101] According to the principle of persistence of vision, when a light pulse stimulates the human eye, the establishment and disappearance of vision requires a certain process, that is, it has a certain inertia. When the repetition frequency of the light pulse is not high enough, the human eye will produce a flickering feeling of alternating light and dark. Only when the continuous light pulse, that is, the image changes more than 24 frames per second, the human eye's flickering feeling is greatly reduced and it is difficult to distinguish a single static picture. It looks like a smooth and continuous video effect. Such a continuous image is a video.

[0102] The following describes the process from video acquisition to video playback. Figure 5 , including the following steps:

[0103] First, the original data of the video (including the original data of multiple frames of images in the video) is collected through cameras, camcorders, mobile phones and other devices, or the original data is imported from storage media. Then the original data is preprocessed to ensure the quality and integrity of the original data, where the preprocessing includes: denoising, color correction, stabilization and other operations. Then the original data is encoded according to a specific video coding standard to obtain the compressed data of the video (including the compressed data of multiple frames of images in the video) to reduce the size of the video file and improve the transmission efficiency. The video coding standards include: H.264, H.265, VP9, ​​etc.

[0104] The compressed data of the video is encapsulated with multiple media streams such as audio data and subtitles to obtain the encapsulation format data of the video (i.e., packaged into a complete container file), where the video encapsulation formats include MP4, AVI, MKV, etc. The processed encapsulation format data is saved to a storage medium or transmitted over a network so that it can be played and viewed on different devices.

[0105] The device (terminal device or server) decapsulates the encapsulated format data to obtain the compressed data of the video, audio data, subtitles and other data, and then decodes the compressed data of the video into the original data of the video and the audio data into audio samples for subsequent processing and playback. The decoded original data and audio samples are rendered and played.

[0106] In this application, for each original frame image in a high-resolution video, the original frame image is cut into multiple image blocks, and then each image block is encoded to obtain compressed data of each image block. The compressed data of each image block is then encapsulated with multiple media streams such as corresponding audio data and subtitles to obtain and save the encapsulation format data of the image block.

[0107] The device detects the object pose information of the interactive object, and / or the prop pose information of the interactive prop. The object pose information includes the position information and posture information of the interactive object, and the posture information includes but is not limited to the direction, perspective, line of sight direction or body movement information of the interactive object. The prop pose information is similar to this, which can express the position information and posture information of the prop. The object pose information and / or the prop pose information can be mapped to the virtual scene, so as to determine the viewing perspective information corresponding to the original frame image with the mapped object pose information and / or the prop pose information. Then, N image blocks that match the viewing perspective information are selected from the multiple image blocks contained in the original frame image, that is, the image blocks of the local area in the original frame image that the interactive object can currently see are selected.

[0108] In some embodiments, after selecting N image blocks matching the viewing angle information, the encapsulation format data of each of the N image blocks are downloaded, and the encapsulation format data of the N image blocks are respectively decapsulated using an audio / video separator to obtain compressed data of each of the N image blocks.

[0109] Specifically, the encapsulation format data of each image block corresponds to a download address, and the download address may be a uniform resource locator (URL). In addition, when the encapsulation format data of N image blocks are stored locally on the device, the encapsulation format data of each image block may be obtained according to the local storage path of the encapsulation format data of each image block.

[0110] The encapsulation format data corresponding to an image block not only includes the compressed data of the image block, but also includes other data such as audio data, subtitles, etc. corresponding to the image block. Therefore, an audio-visual separator (demuxer) is used to decapsulate the encapsulation format data of the image block to obtain the compressed data and other data such as audio data, subtitles, etc. of the image block. In practical applications, a separate audio-visual separator can be assigned to each image block, that is, N image blocks correspond to N audio-visual separators, and each audio-visual separator decapsulates the encapsulation format data of an image block separately to obtain the compressed data of each image block.

[0111] Of course, an audio / video separator may also be used to decapsulate the encapsulated format data of multiple image blocks to obtain the compressed data of each of the multiple image blocks. This application does not make any specific limitation on this.

[0112] In the embodiment of the present application, the original frame image in the high-resolution video is divided into multiple image blocks, and then each image block is encoded and encapsulated to obtain the encapsulation format data of each image block. Then, the encapsulation format data of each image block is independently transmitted and decapsulated, which reduces the complexity of video encoding and video encapsulation while ensuring the high resolution of the video.

[0113] Step 402: merge the compressed data of the N image blocks to obtain merged compressed data.

[0114] Specifically, each image block in the original frame image corresponds to a first index in the original frame image, and the first indexes of multiple image blocks in the original frame image can be constructed in a row-first manner; that is, firstly construct the first indexes of multiple image blocks in the first row of the original frame image in sequence, and then construct the first indexes of multiple image blocks in the second row in sequence, and so on, until the first indexes of all image blocks are obtained. Of course, the first indexes of multiple image blocks in the original frame image can also be constructed in a column-first manner, and this application does not make specific limitations on this.

[0115] The N image blocks in the original frame image that match the viewing angle information may be: N image blocks with consecutive indexes in the original frame image, or N image blocks with discontinuous indexes in the original frame image. The height and width of the multiple image blocks in the original frame image are the same.

[0116] For example, see Figure 6 , the segmentation results of the original frame image include: The image blocks (i.e., divided into 9 image blocks both horizontally and vertically) are constructed in a row-first manner, and the first indexes of each image block are 1 to 81. Nine image blocks matching the viewing angle information are determined from the original frame image, and the first indexes corresponding to the nine image blocks are 11, 14, 21, 22, 29, 30, 31, 40, and 41, respectively. The first indexes of these nine image blocks are not continuous.

[0117] For example, see Figure 7 , the segmentation results of the original frame image include: The image blocks corresponding to the first indexes are 1 to 81. Nine image blocks matching the viewing angle information are determined from the original frame image, and the first indexes corresponding to the nine image blocks are 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively. The first indexes of these nine image blocks are continuous.

[0118] In some embodiments, based on a preset splicing template and the first indexes of the N image blocks, the compressed data of the N image blocks are spliced ​​in sequence to obtain merged compressed data.

[0119] Specifically, multiple candidate templates can be set in advance, and the shapes and sizes of the multiple candidate templates can be set according to actual conditions. According to the number N of image blocks matching the viewing angle information, a splicing template is selected from the multiple candidate templates, wherein the selected splicing template can place at least N image blocks.

[0120] In the order of index from large to small, or index from small to large, the compressed data of each of the N image blocks are sequentially spliced ​​into the splicing template to obtain the merged compressed data, which is essentially the compressed data of the merged image obtained by merging the N image blocks. It should be noted that after the compressed data of each of the N image blocks are sequentially spliced ​​into the splicing template, the splicing template may still contain blank positions (for example, N=8, the size of the splicing template is: ), and may not contain blank positions (for example, N=9, the size of the splicing template is: ).

[0121] During the merging process, a local image area containing N image blocks is first determined from the original frame image. Specifically, the head image block (such as the data block with the smallest first index) among the multiple image blocks contained in the original frame image is first determined. For each image block in the N image blocks, the position offset of the image block compared to the head image block is calculated (including horizontal position offset and vertical position offset). The horizontal position offset refers to: the number of image blocks that are horizontally offset compared to the head image block; the vertical position offset refers to: the number of image blocks that are vertically offset compared to the head image block.

[0122] From the respective lateral position offsets of the N image blocks, the maximum lateral position offset and the minimum lateral position offset are obtained; and from the respective longitudinal position offsets of the N image blocks, the maximum longitudinal position offset and the minimum longitudinal position offset are obtained. The lateral offset range is determined based on the minimum lateral position offset and the maximum lateral position offset; the longitudinal offset range is determined based on the minimum longitudinal position offset and the maximum longitudinal position offset. Based on the lateral offset range and the longitudinal offset range, a local image area containing the N image blocks is determined from the original frame image. The local image area may contain only the selected N image blocks; or it may contain both the selected N image blocks and unselected image blocks (i.e., blank blocks).

[0123] Based on the local image area, for any two image blocks with adjacent indexes in the N image blocks, first calculate the position offset of each of the two image blocks compared to the head image block in the original frame image. Determine whether there is a blank block between the two image blocks based on the obtained position offset. If there is a blank block, filter out the blank block. Finally, according to the splicing template, splice the compressed data of each of the N image blocks retained in the local image area to obtain the merged compressed data.

[0124] For example, see Figure 8 , set the size of the splicing template to: The segmentation results of the original frame image include: The image blocks corresponding to the first indexes are 1 to 81, and the image block with the first index 1 is the head image block of the original frame image. Nine image blocks matching the viewing angle information are determined from the original frame image, and the first indexes corresponding to the nine image blocks are 11, 14, 21, 22, 29, 30, 31, 40, and 41, respectively. The indexes of the nine image blocks are not continuous.

[0125] Among the lateral position offsets of each of the 9 image blocks, the maximum lateral position offset is: the lateral position offset of the image block whose first index is 14 or 41 (i.e., 4 image blocks); the minimum lateral position offset is: the lateral position offset of the image block whose first index is 11 or 29 (i.e., 1 image block); the maximum longitudinal position offset is: the longitudinal position offset of the image block whose first index is 40 or 41 (i.e., 4 image blocks); the minimum longitudinal position offset is: the longitudinal position offset of the image block whose first index is 11 or 14 (i.e., 1 image block).

[0126] Therefore, the local image region 801 extracted from the original frame image is: the image blocks from the second column to the fifth column and the image blocks from the second row to the fifth row in the segmented original frame image, that is, the local image region 801 includes: image blocks.

[0127] Next, the position offset of the image block with the first index of 11 in the local image area 801 compared to the header image block is calculated; and the position offset of the image block with the first index of 14 compared to the header image block is calculated. Based on the position offsets of the two image blocks, it can be determined that there are two blank blocks between the two image blocks, and the two blank blocks are filtered out. The blank blocks at other positions in the local image area 801 can be filtered out in the same manner.

[0128] Then, according to the sequence of the splicing template and the first index from small to large, the compressed data of the 9 image blocks retained in the local image area 801 are spliced ​​in sequence to obtain the merged compressed data corresponding to the merged image 802 .

[0129] It should be noted that, in addition to splicing N image blocks in the order of index from large to small, or in the order of index from small to large, they can also be spliced ​​in other orders, and this application does not make any specific limitation on this.

[0130] In an embodiment of the present application, the compressed data of the N image blocks are sequentially spliced ​​together to obtain merged compressed data according to a preset splicing template and the first indexes of the N image blocks, thereby avoiding omission of compressed data and improving the efficiency of compressed data splicing.

[0131] In some embodiments, in the process of obtaining the merged compressed data, the index is reconstructed for the N image blocks, the second index of each of the N image blocks is obtained, and the index mapping relationship between the first index and the second index corresponding to each image block is recorded.

[0132] Specifically, the N image blocks can be re-indexed according to the splicing order of the N image blocks in the process of obtaining the merged compressed data to obtain the second index of each of the N image blocks; the N image blocks can be re-indexed according to their respective positions in the merged image to obtain the second index of each of the N image blocks; the N image blocks can also be re-indexed in sequence from small to large according to the corresponding first indexes to obtain the second index of each of the N image blocks; of course, the N image blocks can also be re-indexed according to other rules, and the present application does not make any specific limitations on this.

[0133] For example, in obtaining Figure 8 In the process of merging compressed data corresponding to the merged image 802 shown, the 9 image blocks are re-indexed in order from small to large first indexes. Specifically, for the image block with the first index of 11, the second index "1" is re-constructed; the first index "11" is associated with the second index "1" and recorded. For the image block with the first index of 14, the second index 2 is re-constructed; the first index "14" is associated with the second index "2" and recorded. And so on, until the second index of each of the 9 image blocks is obtained, and the first index and the second index of each image block are associated and recorded, as shown in FIG. Fig. 9 shown.

[0134] Step 403: Use a hardware decoder to decode the combined compressed data to obtain combined original data.

[0135] Specifically, the merged compressed data is essentially the compressed data of the merged image obtained by merging N image blocks. Therefore, a hardware decoder can be used to decode the merged compressed data to obtain the merged original data of the merged image, wherein the merged original data includes: the original data of each of the N image blocks. The merged original data is essentially the original data of the merged image obtained by merging N image blocks.

[0136] Step 404 , for each of the N image blocks, perform the following operations: according to the original position information of an image block in the original frame image, render and play the original data of the image block.

[0137] Specifically, before rendering each image block, the original texture corresponding to the original frame image is constructed first. The original position information of the image block in the original frame image refers to the texture coordinates corresponding to the image block on the original texture. According to the texture coordinates corresponding to the image block, the original data of the image block is rendered, and the rendering result is projected onto the screen for display. In practical applications, the original data of N image blocks can be rendered and displayed in parallel.

[0138] In the embodiment of the present application, based on the viewing angle information corresponding to the original frame image, N matching image blocks are selected from the original frame image, and the compressed data of each of the N image blocks is obtained. Then, the compressed data of each of the N image blocks is merged to obtain the merged compressed data; in this way, when performing video decoding, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data; compared with assigning a hardware decoder to each image block for decoding, the number of required hardware decoders is greatly reduced, and the requirements for video playback devices are correspondingly reduced, so that more types of devices can achieve high-resolution VR video playback and expand the scenarios of VR video playback.

[0139] Secondly, when the compressed data of N image blocks are combined and decoded, video decoding can be achieved through a hardware decoder, which effectively avoids the situation where software decoding is forced due to insufficient hardware decoder, thereby ensuring the efficiency of video decoding and further improving the effect of video playback; at the same time, during the video decoding process, the occupancy of device resources (such as processor resources, memory resources, etc.) and device temperature are reduced.

[0140] In addition, the present application combines and decodes the compressed data of N image blocks to obtain combined original data, wherein the combined original data includes: the original data of each of the N image blocks. When performing video rendering, the original position information of the image block in the original frame image is restored, and the original data of the image block is rendered and played according to the original position information, thereby ensuring the accuracy of image block rendering, thereby improving the effect of video playback.

[0141] In some embodiments, the original position information of multiple image blocks contained in the original frame image is obtained by respectively using the following operations: based on the first index of an image block, the width and height of the original frame image, and the width and height of each image block in the original frame image, the original position information of the image block in the original frame image is obtained.

[0142] Specifically, the original texture corresponding to the original frame image is first constructed, and the original texture corresponds to a texture coordinate system. The head image block among the multiple image blocks contained in the original frame image is first determined, and then a vertex of the head image block is used as the origin of the texture coordinate system.

[0143] The original position information of the image block in the original frame image refers to: the texture coordinates of the image block on the original texture corresponding to the original frame image, and the range of the texture coordinates is [0, 1]. The texture coordinates of the image block include: the coordinates of the four vertices on the image block, and the range of the coordinates of each vertex is [0, 1].

[0144] In some embodiments, the upper left corner of the header image block in the original frame image is used as the origin of the texture coordinate system of the original texture, and the first indexes of multiple image blocks in the original frame image are constructed in a row-first manner. At this time, the texture coordinates corresponding to each image block are obtained in the following manner:

[0145] Based on the first index of an image block, the width of the original frame image and the width of each image block, the starting horizontal coordinate and the ending horizontal coordinate of the image block in the original frame image are obtained.

[0146] Specifically, based on the first index of the image block and the number of image blocks into which the original frame image is longitudinally segmented, the lateral position offset of the image block is determined, and the lateral position offset refers to: the number of image blocks that are transversely offset compared to the head image block. Specifically, the lateral position offset of the image block can be obtained by taking the modulus of the first index of the image block and the number of image blocks into which the original frame image is longitudinally segmented. Based on the lateral position offset of the image block and the width of each image block, the lateral offset width of the image block is determined, and specifically, the lateral position offset of the image block can be multiplied by the width of each image block to obtain the lateral offset width of the image block.

[0147] The horizontal offset width of the image block is normalized by the width of the original frame image and then added to the starting horizontal coordinate of the head image block to obtain the starting horizontal coordinate of the image block, as shown in the following formula (1):

[0148]

[0149] in, represents the starting horizontal coordinate of the image block in the original texture, OI represents the first index, and m1 represents the number of image blocks divided vertically from the original frame image. represents the horizontal position offset of the image block, W represents the width of each image block, and I represents the width of the original frame image.

[0150] The horizontal offset width of the image block is normalized by the width of the original frame image and then added to the end point abscissa of the head image block to obtain the end point abscissa of the image block, as shown in the following formula (2):

[0151]

[0152] in, Indicates the horizontal coordinate of the end point of the image block in the original texture.

[0153] Next, based on the first index of the image block, the height of the original frame image and the height of each image block, the starting ordinate and the ending ordinate of the image block in the original frame image are obtained.

[0154] Specifically, based on the first index of the image block and the number of image blocks into which the original frame image is horizontally segmented, the longitudinal position offset of the image block is determined, and the longitudinal position offset refers to the number of image blocks that are longitudinally offset compared to the head image block. Specifically, the first index of the image block can be divided by the number of image blocks into which the original frame image is horizontally segmented and then rounded to obtain the longitudinal position offset of the image block. Based on the longitudinal position offset of the image block and the height of each image block, the longitudinal offset height of the image block is determined. Specifically, the longitudinal position offset of the image block can be multiplied by the height of each image block to obtain the longitudinal offset height of the image block.

[0155] The vertical offset height of the image block is normalized by the height of the original frame image and then added to the starting vertical coordinate of the head image block to obtain the starting vertical coordinate of the image block, as shown in the following formula (3):

[0156]

[0157] in, represents the starting ordinate of the image block in the original texture, OI represents the first index, n1 represents the number of image blocks that the original frame image is horizontally segmented into, represents the longitudinal position offset of the image block, H represents the height of each image block, and J represents the height of the original frame image.

[0158] The vertical offset height of the image block is normalized by the height of the original frame image and then added to the vertical coordinate of the end point of the head image block to obtain the vertical coordinate of the end point of the image block, as shown in the following formula (4):

[0159]

[0160] in, Indicates the vertical coordinate of the end point of the image block in the original texture.

[0161] Based on the obtained starting horizontal coordinate, ending horizontal coordinate, starting vertical coordinate and ending vertical coordinate, the original position information of the image block in the original frame image is obtained.

[0162] Specifically, the starting horizontal coordinate and the starting ordinate The coordinates of the upper left corner of the image block in the texture coordinate system . Set the starting horizontal coordinate and the end point ordinate The coordinates of the lower left corner of the image block in the texture coordinate system . Set the end point horizontal coordinate and the starting ordinate The coordinates of the upper right corner of the image block in the texture coordinate system . Set the end point horizontal coordinate and the end point ordinate The coordinates of the lower right corner of the image block in the texture coordinate system .

[0163] coordinate ,coordinate ,coordinate and coordinates It constitutes the texture coordinates of the image block in the original texture, that is, the original position information of the image block in the original frame image.

[0164] For example, see Fig.10 , the segmentation results corresponding to the original frame image include: The image blocks (i.e., divided into 9 image blocks both horizontally and vertically) have corresponding first indexes from 1 to 81, and each image block has a width of 1 and a height of 1. The width of the original frame image is 9 and the height is 9. The image block with the first index of 1 is the header image block in the original frame image. The original texture is constructed for the original frame image, and the upper left corner of the header image block is the origin (0,0) in the texture coordinate system of the original texture.

[0165] For the image block with the first index of 2, the first index "2" of the image block, the number of image blocks into which the original frame image is vertically divided "9", the width of each image block "1", and the width of the original frame image "9" are substituted into the above formula (1) to obtain the starting horizontal coordinate of the image block: 1 / 9.

[0166] Substituting the first index "2" of the image block, the number of image blocks into which the original frame image is vertically divided "9", the width of each image block "1", and the width of the original frame image "9" into the above formula (2), the abscissa of the end point of the image block is obtained as: 2 / 9.

[0167] Substituting the first index "2" of the image block, the number of image blocks into which the original frame image is horizontally divided "9", the height of each image block "1", and the height of the original frame image "9" into the above formula (3), the starting vertical coordinate of the image block is obtained as: 0.

[0168] Substituting the first index "2" of the image block, the number of image blocks into which the original frame image is horizontally divided "9", the height of each image block "1", and the height of the original frame image "9" into the above formula (4), the vertical coordinate of the end point of the image block is obtained as: 1 / 9.

[0169] Therefore, the texture coordinates of the image block with the first index 2 include: (1 / 9, 0), (2 / 9, 0), (1 / 9, 1 / 9), (2 / 9, 1 / 9).

[0170] For the image block with the first index of 11, the first index "11" of the image block, the number of image blocks into which the original frame image is vertically divided "9", the width of each image block "1", and the width of the original frame image "9" are substituted into the above formula (1) to obtain the starting horizontal coordinate of the image block: 1 / 9.

[0171] Substituting the first index "11" of the image block, the number of image blocks into which the original frame image is vertically divided "9", the width of each image block "1", and the width of the original frame image "9" into the above formula (2), the abscissa of the end point of the image block is obtained as: 2 / 9.

[0172] Substituting the first index "11" of the image block, the number of image blocks into which the original frame image is horizontally divided "9", the height of each image block "1", and the height of the original frame image "9" into the above formula (3), the starting vertical coordinate of the image block is obtained as: 1 / 9.

[0173] Substituting the first index "11" of the image block, the number of image blocks into which the original frame image is horizontally divided "9", the height of each image block "1", and the height of the original frame image "9" into the above formula (4), the vertical coordinate of the end point of the image block is obtained as: 2 / 9.

[0174] Therefore, the texture coordinates of the image block with the first index 11 in the original texture include: (1 / 9, 1 / 9), (1 / 9, 2 / 9), (2 / 9, 1 / 9), (2 / 9, 2 / 9).

[0175] The texture coordinates of any other image block in the original texture can be obtained in the same way, which will not be described here.

[0176] In an embodiment of the present application, based on the first index of each image block in the original frame image, the width and height of the original frame image, and the width and height of each image block in the original frame image, each image block in the original frame image is mapped to the texture coordinate system of the original texture, and the original position information of each image block in the original frame image (i.e., the texture coordinates in the original texture) is obtained, which provides an accurate rendering position reference for the subsequent rendering of the original data of each of the N selected image blocks, thereby ensuring the accuracy of image block rendering and improving the video playback effect.

[0177] It should be noted that the present application may also use other vertices of the head image block in the original frame image as the origin of the texture coordinate system. Similarly, the first indexes of multiple image blocks in the original frame image may also be constructed in a column dimension priority manner. When the selected vertices change or the method of constructing the index changes, obtaining the original position information of each image block in the original frame image is the same as the implementation method described above, and will not be repeated here.

[0178] In some embodiments, the target position information of each of the N image blocks in the merged image is obtained by respectively performing the following operations:

[0179] Based on the second index of the image block, the width and height of the merged image, and the width and height of each image block in the merged image, target position information of the image block in the merged image is obtained.

[0180] Specifically, a target texture corresponding to the merged image is first constructed, and the target texture corresponds to a texture coordinate system. The head image block among the N image blocks contained in the merged image is first determined. The upper left corner of the head image block in the merged image is used as the origin of the texture coordinate system of the target texture. The second indexes of the multiple image blocks in the merged image are constructed in a row dimension priority manner.

[0181] Based on the second index of an image block, the width of the merged image and the width of each image block, the starting horizontal coordinate and the ending horizontal coordinate of the image block in the merged image are obtained.

[0182] Specifically, based on the second index of the image block and the number of image blocks into which the merged image is longitudinally segmented, the lateral position offset of the image block is determined. Specifically, the lateral position offset of the image block can be obtained by taking the modulus of the second index of the image block and the number of image blocks into which the merged image is longitudinally segmented. Based on the lateral position offset of the image block and the width of each image block, the lateral offset width of the image block is determined. Specifically, the lateral position offset of the image block can be multiplied by the width of each image block to obtain the lateral offset width of the image block.

[0183] The horizontal offset width of the image block is normalized by the width of the merged image and then added to the starting horizontal coordinate of the head image block to obtain the starting horizontal coordinate of the image block, as shown in the following formula (5):

[0184]

[0185] in, represents the starting horizontal coordinate of the image block in the target texture, CI represents the second index, m2 represents the number of image blocks divided vertically by the merged image, Indicates the horizontal position offset of the image block. W represents the width of each image block, and K represents the width of the merged image.

[0186] The horizontal offset width of the image block is normalized by the width of the merged image and then added to the end point abscissa of the head image block to obtain the end point abscissa of the image block, as shown in the following formula (6):

[0187]

[0188] in, Indicates the horizontal coordinate of the end point of the image block in the target texture.

[0189] Next, based on the second index of the image block, the height of the merged image and the height of each image block, the start ordinate and the end ordinate of the image block in the merged image are obtained.

[0190] Specifically, the longitudinal position offset of the image block is determined based on the second index of the image block and the number of image blocks into which the merged image is horizontally sliced. Specifically, the longitudinal position offset of the image block can be obtained by dividing the second index of the image block by the number of image blocks into which the merged image is horizontally sliced ​​and then rounding the resultant. Based on the longitudinal position offset of the image block and the height of each image block, the longitudinal offset height of the image block is determined. Specifically, the longitudinal position offset of the image block can be multiplied by the height of each image block to obtain the longitudinal offset height of the image block.

[0191] The vertical offset height of the image block is normalized by the height of the merged image and then added to the starting vertical coordinate of the head image block to obtain the starting vertical coordinate of the image block, as shown in the following formula (7):

[0192]

[0193] in, represents the starting ordinate of the image block in the target texture, CI represents the second index, n2 represents the number of image blocks that the merged image is horizontally segmented into, represents the vertical position offset of the image block, H represents the height of each image block, and P represents the height of the merged image.

[0194] The vertical offset height of the image block is normalized by the height of the merged image and then added to the vertical coordinate of the end point of the head image block to obtain the vertical coordinate of the end point of the image block, as shown in the following formula (8):

[0195]

[0196] in, Indicates the ordinate of the end point of the image block in the target texture.

[0197] Based on the obtained starting abscissa, ending abscissa, starting ordinate and ending ordinate, target position information of the image block in the merged image is obtained.

[0198] Specifically, the starting horizontal coordinate and the starting ordinate The coordinates of the upper left corner of the image block in the texture coordinate system . Set the starting horizontal coordinate and the end point ordinate The coordinates of the lower left corner of the image block in the texture coordinate system . Set the end point horizontal coordinate and the starting ordinate The coordinates of the upper right corner of the image block in the texture coordinate system . Set the end point horizontal coordinate and the end point ordinate The coordinates of the lower right corner of the image block in the texture coordinate system .

[0199] coordinate ,coordinate ,coordinate and coordinates It constitutes the texture coordinates of the image block in the target texture, that is, the target position information of the image block in the merged image.

[0200] For example, see Fig.11 , the segmentation results corresponding to the merged image include The image blocks (i.e., divided into 3 image blocks both horizontally and vertically) have corresponding second indexes of 1 to 9, and the width and height of each image block are 1. The width and height of the merged image are 3 and 3, respectively. The image block with the second index of 1 is the head image block in the merged image. The target texture is constructed for the merged image, and the upper left corner of the head image block is the origin (0,0) in the texture coordinate system of the target texture.

[0201] For the image block with the second index of 1, the second index of the image block "1", the number of image blocks into which the merged image is vertically divided "3", the width of each image block "1", and the width of the merged image "3" are substituted into the above formula (5) to obtain the starting horizontal coordinate of the image block: 0.

[0202] Substituting the second index "1" of the image block, the number of image blocks into which the merged image is vertically divided "3", the width of each image block "1", and the width of the merged image "3" into the above formula (6), the starting horizontal coordinate of the image block is obtained as: 1 / 3.

[0203] Substituting the second index "1" of the image block, the number of image blocks into which the merged image is horizontally divided "3", the height of each image block "1", and the height of the merged image "3" into the above formula (7), the starting vertical coordinate of the image block is obtained as: 0.

[0204] Substituting the second index "1" of the image block, the number of image blocks into which the merged image is horizontally divided "3", the height of each image block "1", and the height of the merged image "3" into the above formula (8), the vertical coordinate of the end point of the image block is obtained as: 1 / 3.

[0205] Therefore, the texture coordinates of the image block with the second index “1” in the target texture include: (0, 0), (0, 1 / 3), (1 / 3, 0), and (1 / 3, 1 / 3).

[0206] The texture coordinates of any other image block in the original texture can be obtained in the same way, which will not be described here.

[0207] In an embodiment of the present application, based on the second index of each image block in the merged image, the width and height of the merged image, and the width and height of each image block in the merged image, each image block is mapped to the texture coordinate system of the target texture, and the texture coordinates of each image block in the target texture (i.e., the target position information in the merged image) are obtained. This unifies the position representation method of the image block in the original frame image and in the merged image, which facilitates the restoration of the original position information of the image block in the original frame image during the image block rendering stage, while improving the accuracy of position restoration.

[0208] It should be noted that the present application may also use other vertices of the head image block in the merged image as the origin of the texture coordinate system. Similarly, the second index of multiple image blocks in the merged image may also be constructed in a column dimension priority manner. When the selected vertices change or the method of constructing the index changes, obtaining the target position information of each image block in the merged image is the same as the implementation method described above, and will not be repeated here.

[0209] In some embodiments, during the rendering process, the original position information of the image block in the original frame image is obtained, the target position information of the image block in the merged image corresponding to the merged original data is adjusted to the original position information, and the original data of the image block is rendered and played according to the original position information.

[0210] Specifically, in the process of obtaining the merged compressed data, the indexes are reconstructed for the N image blocks, the second indexes of the N image blocks are obtained, and the index mapping relationship between the first index and the second index corresponding to each image block is recorded.

[0211] Therefore, before rendering, the first index corresponding to the image block can be obtained based on the second index corresponding to the image block and the index mapping relationship. Then, based on the first index corresponding to the image block, the original position information of the image block in the original frame image, that is, the texture coordinates of the image block in the original texture, is obtained. Then, the texture coordinates of the image block in the target texture corresponding to the merged image are adjusted to the texture coordinates in the original texture. According to the texture coordinates of the image block in the original texture, the original data of the decoded image block is rendered, and then the rendering result is projected onto the screen for playback.

[0212] For example, see Fig. 12A, the merged image corresponding to the merged original data includes: The corresponding second indexes of the image blocks are 1 to 9. For the image block with the second index of 1, the texture coordinates of the image block in the target texture (corresponding to the merged image) are: (0, 0), (0, 1 / 3), (1 / 3, 0), (1 / 3, 1 / 3).

[0213] Based on the recorded index mapping relationship, it can be known that the first index corresponding to the image block in the original frame image is 11, and the corresponding texture coordinates of the image block in the original texture (corresponding to the original frame image) are: (1 / 9, 1 / 9), (1 / 9, 2 / 9), (2 / 9, 1 / 9), (2 / 9, 2 / 9).

[0214] The texture coordinates of the image block in the target texture are mapped to the texture coordinates of the image block in the original texture. In the same way, the texture coordinates of other image blocks in the merged image in the target texture can be mapped to the texture coordinates of the image block in the original texture.

[0215] According to the texture coordinates of each image block in the original texture, the decoded original data of the image block is rendered, and then the rendering result is projected onto the screen for playback.

[0216] In an embodiment of the present application, in the process of merging compressed data of image blocks, the second index is reconstructed for each of the N image blocks, and the index mapping relationship between the first index and the second index corresponding to each image block is recorded. Therefore, before video rendering, the texture coordinates of the image block in the target texture can be quickly restored to the texture coordinates in the original texture based on the index mapping relationship, thereby improving the efficiency and accuracy of coordinate restoration.

[0217] Secondly, when performing video rendering, the texture coordinates of the image block in the target texture are restored to the texture coordinates in the original texture, so as to restore the target position information of the image block in the merged image to the original position information of the image block in the original frame image, and render and play the original data of the image block according to the original position information, thereby ensuring the accuracy of image block rendering and improving the effect of video playback.

[0218] In order to better explain the embodiment of the present application, a video playback method provided by the embodiment of the present application is introduced below in combination with the implementation scenario of VR. The process of the method can be interactively executed by the VR device and the server, including the following steps: Fig. 12B As shown:

[0219] The server saves the video to be played in advance, and for each original frame image in the video to be played, the server divides the original frame image into multiple image blocks, each image block corresponds to a first index. Each image block is independently encoded to obtain corresponding compressed data. The compressed data of each image block is encapsulated to obtain corresponding encapsulation format data, and the encapsulation format data of each image block is saved.

[0220] When an interactive object wearing a VR device performs video interaction, the VR device detects the posture information of the interactive object, determines the viewing angle information of the interactive object based on the posture information, and then sends a video download request carrying the viewing angle information to the server.

[0221] The server selects multiple original frame images that need to be played from the video to be played. Taking one original frame image as an example, 9 image blocks that match the viewing angle information are selected from the multiple image blocks contained in the original frame image, and then the encapsulation format data of each of the 9 image blocks is sent to the VR device.

[0222] The VR device decapsulates the encapsulated format data of the nine image blocks respectively to obtain compressed data of the nine image blocks. Then, the compressed data of the nine image blocks are merged to obtain compressed data of a merged image (obtained by merging the nine image blocks). The second index constructed in the merged image is reconstructed for the nine image blocks, and the index mapping relationship between the first index and the second index is recorded.

[0223] The VR device uses a hardware decoder to decode the compressed data of the merged image to obtain the original data of the merged image. The original texture corresponding to the original frame image is constructed, and then the texture coordinates of each block in the original frame image in the original texture are calculated. The VR device constructs the target texture corresponding to the merged image, and then calculates the texture coordinates of each block in the merged image in the target texture. According to the index mapping relationship between the first index and the second index, the texture coordinates of each image block in the merged image in the target texture are restored to the texture coordinates in the original texture.

[0224] Rendering is performed according to the texture coordinates in the original texture, and the rendered image blocks are mapped to the corresponding positions on the surface of the sphere for display.

[0225] In the embodiment of the present application, based on the viewing angle information corresponding to the original frame image, N matching image blocks are selected from the original frame image, and the compressed data of each of the N image blocks is obtained. Then, the compressed data of each of the N image blocks is merged to obtain the merged compressed data; in this way, when performing video decoding, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data; compared with assigning a hardware decoder to each image block for decoding, the number of required hardware decoders is greatly reduced, and the requirements for video playback devices are correspondingly reduced, so that more types of devices can achieve high-resolution VR video playback and expand the scenarios of VR video playback.

[0226] Secondly, when the compressed data of N image blocks are combined and decoded, video decoding can be achieved through a hardware decoder, which effectively avoids the situation where software decoding is forced due to insufficient hardware decoder, thereby ensuring the efficiency of video decoding and further improving the effect of video playback; at the same time, during the video decoding process, the occupancy of device resources (such as processor resources, memory resources, etc.) and device temperature are reduced.

[0227] In addition, the present application combines and decodes the compressed data of N image blocks to obtain combined original data, wherein the combined original data includes: the original data of each of the N image blocks. When performing video rendering, the original position information of the image block in the original frame image is restored, and the original data of the image block is rendered and played according to the original position information, thereby ensuring the accuracy of image block rendering, thereby improving the effect of video playback.

[0228] Based on the same technical concept, the present application embodiment provides a structural diagram of a video playback device, such as Fig.13 As shown, the video playback device 1300 includes:

[0229] A selection module 1301 is used to select N image blocks from the original frame image based on viewing angle information corresponding to the original frame image, and obtain compressed data of each of the N image blocks, where N is greater than 1;

[0230] A merging module 1302 is used to merge the compressed data of each of the N image blocks to obtain merged compressed data;

[0231] The decoding module 1303 is used to decode the combined compressed data using a hardware decoder to obtain combined original data; the combined original data includes: the original data of each of the N image blocks;

[0232] The rendering module 1304 is used to perform the following operations respectively for the N image blocks: according to the original position information of the image block in the original frame image, render and play the original data of the image block.

[0233] Optionally, each image block in the original frame image corresponds to a first index in the original frame image;

[0234] The rendering module 1304 is also used for:

[0235] Based on a first index of an image block, the width and height of the original frame image, and the width and height of each image block in the original frame image, original position information of the image block in the original frame image is obtained.

[0236] Optionally, the rendering module 1304 is specifically used for:

[0237] Based on the first index of the image block, the width of the original frame image and the width of each image block, obtaining the starting horizontal coordinate and the ending horizontal coordinate of the image block in the original frame image;

[0238] Based on the first index of the image block, the height of the original frame image and the height of each image block, obtaining the starting ordinate and the ending ordinate of the image block in the original frame image;

[0239] Based on the starting horizontal coordinate, the ending horizontal coordinate, the starting vertical coordinate and the ending vertical coordinate, original position information of the one image block in the original frame image is obtained.

[0240] Optionally, the rendering module 1304 is specifically used for:

[0241] Acquire original position information of the one image block in the original frame image;

[0242] Adjusting target position information of the one image block in the merged image corresponding to the merged original data to the original position information;

[0243] According to the original position information, the original data of the image block is rendered and played.

[0244] Optionally, the merging module 1302 is further used for:

[0245] Merging the compressed data of each of the N image blocks to obtain the merged compressed data, reconstructing indexes for the N image blocks to obtain second indexes for each of the N image blocks, and recording an index mapping relationship between the first index and the second index corresponding to each image block;

[0246] The rendering module 1304 is specifically used for:

[0247] Based on the second index corresponding to the one image block and the index mapping relationship, obtaining a first index corresponding to the one image block;

[0248] Based on the first index corresponding to the one image block, original position information of the one image block in the original frame image is obtained.

[0249] Optionally, the rendering module 1304 is further used for:

[0250] Based on the second index of an image block, the width and height of the merged image, and the width and height of each image block in the merged image, target position information of the image block in the merged image is obtained.

[0251] Optionally, the merging module 1302 is specifically used for:

[0252] Based on a preset splicing template and the first indexes of the N image blocks, the compressed data of the N image blocks are sequentially spliced ​​to obtain merged compressed data.

[0253] Optionally, the selection module 1301 is specifically used for:

[0254] Downloading respective encapsulation format data of the N image blocks;

[0255] The encapsulated format data of the N image blocks are respectively decapsulated by using an audio / video separator to obtain compressed data of each of the N image blocks.

[0256] In the embodiment of the present application, based on the viewing angle information corresponding to the original frame image, N matching image blocks are selected from the original frame image, and the compressed data of each of the N image blocks is obtained. Then, the compressed data of each of the N image blocks is merged to obtain the merged compressed data; in this way, when performing video decoding, only one hardware decoder is needed to decode the merged compressed data to obtain the merged original data; compared with assigning a hardware decoder to each image block for decoding, the number of required hardware decoders is greatly reduced, and the requirements for video playback devices are correspondingly reduced, so that more types of devices can achieve high-resolution VR video playback and expand the scenarios of VR video playback.

[0257] Secondly, when the compressed data of N image blocks are combined and decoded, video decoding can be achieved through a hardware decoder, which effectively avoids the situation where software decoding is forced due to insufficient hardware decoder, thereby ensuring the efficiency of video decoding and further improving the effect of video playback; at the same time, during the video decoding process, the occupancy of device resources (such as processor resources, memory resources, etc.) and device temperature are reduced.

[0258] In addition, the present application combines and decodes the compressed data of N image blocks to obtain combined original data, wherein the combined original data includes: the original data of each of the N image blocks. When performing video rendering, the original position information of the image block in the original frame image is restored, and the original data of the image block is rendered and played according to the original position information, thereby ensuring the accuracy of image block rendering, thereby improving the effect of video playback.

[0259] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0260] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Fig.14 As shown, it includes at least one processor 1401 and a memory 1402 connected to the at least one processor. The specific connection medium between the processor 1401 and the memory 1402 is not limited in the embodiment of the present application. Fig.14 For example, the processor 1401 and the memory 1402 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0261] In the embodiment of the present application, the memory 1402 stores instructions that can be executed by at least one processor 1401, and the at least one processor 1401 can perform the steps of the above-mentioned video playback method by executing the instructions stored in the memory 1402.

[0262] Among them, the processor 1401 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and realize video playback by running or executing instructions stored in the memory 1402 and calling data stored in the memory 1402. Optionally, the processor 1401 may include one or more processing units, and the processor 1401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1401. In some embodiments, the processor 1401 and the memory 1402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0263] Processor 1401 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0264] The memory 1402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1402 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory 1402 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 1402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, used to store program instructions and / or data.

[0265] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned video playback method.

[0266] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned video playback method.

[0267] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0268] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0269] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0270] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0271] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0272] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A video playback method, characterized in that: include: Based on viewing angle information corresponding to the original frame image, select N image blocks from the original frame image, and obtain compressed data of each of the N image blocks from a server, where N is greater than 1; Merging the compressed data of each of the N image blocks to obtain merged compressed data, where the merged compressed data is compressed data of a merged image obtained by merging the N image blocks; Using a hardware decoder to decode the combined compressed data to obtain combined original data; The merged original data includes: original data of each of the N image blocks, and the merged original data is original data of the merged image; For the N image blocks, the following operations are performed respectively: according to the original position information of the image block in the original frame image, the original data of the image block is rendered and played.

2. The method according to claim 1, characterized in that Each image block in the original frame image corresponds to a first index in the original frame image; original position information of multiple image blocks contained in the original frame image is obtained by respectively performing the following operations: Based on a first index of an image block, the width and height of the original frame image, and the width and height of each image block in the original frame image, original position information of the image block in the original frame image is obtained.

3. The method according to claim 2, characterized in that The obtaining, based on a first index of an image block, the width and height of the original frame image, and the width and height of each image block in the original frame image, original position information of the image block in the original frame image includes: Based on the first index of the image block, the width of the original frame image and the width of each image block, obtaining the starting horizontal coordinate and the ending horizontal coordinate of the image block in the original frame image; Based on the first index of the image block, the height of the original frame image and the height of each image block, obtaining the starting ordinate and the ending ordinate of the image block in the original frame image; Based on the starting horizontal coordinate, the ending horizontal coordinate, the starting vertical coordinate and the ending vertical coordinate, original position information of the one image block in the original frame image is obtained.

4. The method according to claim 1, characterized in that The rendering and playing of the original data of the image block according to the original position information of the image block in the original frame image includes: Acquire original position information of the one image block in the original frame image; Adjusting target position information of the one image block in the merged image corresponding to the merged original data to the original position information; According to the original position information, the original data of the image block is rendered and played.

5. The method according to claim 4, characterized in that After merging the compressed data of the N image blocks to obtain the merged compressed data, the method further includes: Reconstructing indexes for the N image blocks, obtaining second indexes for the N image blocks, and recording an index mapping relationship between the first index and the second index corresponding to each image block; The obtaining original position information of the image block in the original frame image includes: Based on the second index corresponding to the one image block and the index mapping relationship, obtaining a first index corresponding to the one image block; Based on the first index corresponding to the one image block, original position information of the one image block in the original frame image is obtained.

6. The method according to claim 5, characterized in that The target position information of each of the N image blocks in the merged image is obtained by respectively performing the following operations: Based on the second index of an image block, the width and height of the merged image, and the width and height of each image block in the merged image, target position information of the image block in the merged image is obtained.

7. The method according to any one of claims 2 to 6, characterized in that: The step of merging the compressed data of each of the N image blocks to obtain merged compressed data includes: Based on a preset splicing template and the first indexes of the N image blocks, the compressed data of the N image blocks are sequentially spliced ​​to obtain merged compressed data.

8. The method according to any one of claims 1 to 6, characterized in that: The obtaining compressed data of each of the N image blocks from the server includes: Downloading the packaging format data of each of the N image blocks from the server; The encapsulated format data of the N image blocks are respectively decapsulated by using an audio / video separator to obtain compressed data of each of the N image blocks.

9. A video playback device, characterized in that: include: A selection module, configured to select N image blocks from the original frame image based on viewing angle information corresponding to the original frame image, and obtain compressed data of each of the N image blocks from a server, where N is greater than 1; a merging module, used to merge the compressed data of each of the N image blocks to obtain merged compressed data, where the merged compressed data is compressed data of a merged image obtained by merging the N image blocks; A decoding module, used for decoding the combined compressed data using a hardware decoder to obtain combined original data; The merged original data includes: original data of each of the N image blocks, and the merged original data is original data of the merged image; The rendering module is used to perform the following operations respectively for the N image blocks: according to the original position information of the image block in the original frame image, render and play the original data of the image block.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of any one of the methods of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 8.

Images