Space video generation method and device, space video playing method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a space video generation method and device, a space video playing method and device, electronic equipment, a storage medium and a program product. The space video generation method comprises the steps that a first frame queue is shot through a first camera, a second frame queue is shot through a second camera, the first frame queue comprises at least one first-eye video frame, and the second frame queue comprises at least one second-eye video frame; encoding the first video frame and the second video frame to obtain media data of a target space video; and generating a target video file of the target space video according to the media data and coding and decoding feature data of the target space video, the coding and decoding feature data including first-view coding and decoding feature data and second-view coding and decoding feature data. According to the embodiment of the invention, the generation and / or playing of the MV-HEVC space video are / is realized in the Android system by utilizing the technical scheme.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for generating and playing spatial video, an apparatus, an electronic device, a storage medium, and a program product. Background Technology

[0002] For spatial video, especially multi-view stitched spatial video, the Joint Collaborative Team on 3D Video Coding Extensions (JCT-3V) was established, and in 2014, it published MV-HEVC, a High Efficiency Video Coding standard extension suitable for spatial multi-view video coding. MV-HEVC stores information using a difference between the main view and auxiliary views, which improves storage efficiency and coding performance.

[0003] However, the Android system currently lacks the capability to generate and play MV-HEVC spatial videos. Summary of the Invention

[0004] This disclosure provides a method for generating and playing spatial video, an apparatus, an electronic device, a storage medium, and a program product to realize the generation and / or playback of MV-HEVC spatial video in an Android system.

[0005] In a first aspect, embodiments of this disclosure provide a method for generating spatial video, comprising:

[0006] A first frame queue is captured by a first camera, and a second frame queue is captured by a second camera, wherein the first frame queue contains at least one first-view video frame, and the second frame queue contains at least one second-view video frame.

[0007] The first and second target video frames are encoded to obtain media data of the target space video.

[0008] A target video file for the target spatial video is generated based on the media data and the encoding / decoding feature data of the target spatial video, wherein the encoding / decoding feature data includes first-order encoding / decoding feature data and second-order encoding / decoding feature data.

[0009] Secondly, embodiments of this disclosure also provide a method for playing spatial video, including:

[0010] Obtain codec feature data and media data from a target video file, wherein the target video file is a video file of a target spatial video, and the codec feature data includes first-order codec feature data and second-order codec feature data;

[0011] The media data is decoded according to the encoding and decoding feature data to obtain a first video stream and a second video stream. The timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other.

[0012] The first video stream is played on a first display screen, and the second video stream is played on a second display screen, wherein the first video stream and the second video stream are played synchronously.

[0013] Thirdly, embodiments of this disclosure also provide a spatial video generation apparatus, comprising:

[0014] The shooting module is used to capture a first frame queue through a first camera and a second frame queue through a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame.

[0015] The encoding module is used to encode the first target video frame and the second target video frame to obtain media data of the target space video.

[0016] The generation module is used to generate a target video file of the target spatial video based on the media data and the codec feature data of the target spatial video, wherein the codec feature data includes first codec feature data and second codec feature data.

[0017] Fourthly, embodiments of this disclosure also provide a spatial video playback device, comprising:

[0018] The acquisition module is used to acquire the codec feature data and media data in the target video file, wherein the target video file is a video file of the target spatial video, and the codec feature data includes first codec feature data and second codec feature data.

[0019] The decoding module is used to decode the media data according to the encoding and decoding feature data to obtain a first video stream and a second video stream, wherein the timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other.

[0020] The display module is used to play the first video stream on a first display screen and the second video stream on a second display screen, wherein the first video stream and the second video stream are played synchronously.

[0021] Fifthly, embodiments of this disclosure also provide an electronic device, including:

[0022] One or more processors;

[0023] Memory, used to store one or more programs.

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the spatial video generation method or spatial video playback method as described in the embodiments of this disclosure.

[0025] Sixthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the spatial video generation method or spatial video playback method as described in embodiments of this disclosure.

[0026] In a seventh aspect, embodiments of this disclosure also provide a computer program product that, when executed by a computer, enables the computer to implement the spatial video generation method or the spatial video playback method as described in embodiments of this disclosure.

[0027] The spatial video generation method, playback method, apparatus, electronic device, storage medium, and program product provided in this disclosure, when generating spatial video, respectively store the first and second octet codec feature data of the spatial video into the video file of the target spatial video. The first octet codec feature data indicates the codec features of the first octet video frame of the spatial video, and the second octet codec feature data indicates the codec features of the second octet video frame of the spatial video. And / or, when playing spatial video, the media data in the video file is decoded based on the first and second octet codec feature data in the video file of the spatial video. This enables the generation and / or playback of spatial video in the Android system, enriching the generation and / or playback methods of spatial video. Attached Figure Description

[0028] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0029] Figure 1 A schematic flowchart illustrating a method for generating spatial video according to an embodiment of this disclosure;

[0030] Figure 2 A schematic flowchart illustrating another method for generating spatial video provided in this embodiment of the disclosure;

[0031] Figure 3 A schematic diagram illustrating the generation process of spatial video according to an embodiment of this disclosure;

[0032] Figure 4 A schematic flowchart illustrating a method for playing spatial video according to an embodiment of this disclosure;

[0033] Figure 5 A schematic diagram illustrating the playback process of spatial video provided in an embodiment of this disclosure;

[0034] Figure 6 A structural block diagram of a spatial video generation apparatus provided in an embodiment of this disclosure;

[0035] Figure 7 A structural block diagram of a spatial video playback device provided in an embodiment of this disclosure;

[0036] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0037] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0038] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0039] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0040] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0041] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0042] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0043] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0044] Figure 1 This is a flowchart illustrating a method for generating spatial video according to an embodiment of this disclosure. The method can be executed by a spatial video generation device, which can be implemented in software and / or hardware and can be configured in an electronic device, typically a virtual reality (VR) device, mobile phone, or tablet computer. The spatial video generation method provided in this disclosure is applicable to scenarios involving shooting spatial video, such as shooting spatial video using an electronic device with an Android operating system. Figure 1 As shown, the spatial video generation method provided in this embodiment may include:

[0045] S101. Capture a first frame queue using a first camera and a second frame queue using a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame.

[0046] The first camera and the second camera can be two different cameras. The first camera can be used to capture a first-view video frame of the target space video, and the second camera can be used to capture a second-view video frame of the target space video. The first and second cameras can be configured in the electronic device on which the current application is installed, such as by simultaneously capturing the first and second-view video frames of the target space video through different cameras of the electronic device on which the current application is installed. Alternatively, the first and second cameras can also be configured independently of the electronic device on which the current application is installed, such as by capturing the first and second-view video frames of the target space video through a shooting device that has established a communication connection with the electronic device on which the current application is installed. The shooting device can be configured with at least two cameras, including a first camera and a second camera, and different cameras can be used to capture images from different perspectives. For example, the first camera can be used to simulate the vision of one eye (left and right) for image acquisition, and the second camera can be used to simulate the vision of the other eye (left and right) for image acquisition, in order to capture and generate a target space video with stereoscopic visual effects.

[0047] The first frame queue can be a frame queue consisting of first-view video frames, which may contain at least one first-view video frame, and the first-view video frames are arranged in the order they were captured. The second frame queue can be a frame queue consisting of second-view video frames, which may contain at least one second-view video frame, and the second-view video frames are arranged in the order they were captured.

[0048] The first video frame can be a video frame captured by a first camera simulating the perspective of one eye. During subsequent video playback, the first video frame can be displayed on the screen corresponding to this eye. The second video frame can be a video frame captured by a second camera simulating the perspective of the other eye. For example, the first video frame can be a left-eye video frame, and the second video frame can be a right-eye video frame; or, the first video frame can be a right-eye video frame, and the second video frame can be a left-eye video frame. The specific configuration can be set as needed. The following explanation uses the example of the first video frame being a left-eye video frame and the second video frame being a right-eye video frame. In this case, the first camera can be a left-eye camera, the second camera can be a right-eye camera, the first frame queue can be a left-eye frame queue, and the second frame queue can be a right-eye frame queue.

[0049] In this embodiment, when generating the target space video, the first frame queue of the target space video can be captured by the first camera, and the second frame queue of the target space video can be captured by the second camera.

[0050] For example, when it is detected that the current user is performing a spatial video shooting operation, in response to the shooting operation, the left eye camera and the right eye camera can be used to shoot simultaneously. Specifically, the left eye camera is used to shoot each left eye video frame of the target spatial video in sequence, and the right eye camera is used to shoot each right eye video frame of the target spatial video in sequence.

[0051] S102. Encode the first target video frame and the second target video frame to obtain the media data of the target space video.

[0052] The target spatial video can be understood as the spatial video to be generated, or in other words, the spatial video currently being filmed. The type of target spatial video is not limited; for example, it can be MV-HEVC spatial video. MV-HEVC spatial video is a type of 3D video encoded using the MV-HEVC spatial video encoding format. It can simulate human stereoscopic vision, allowing viewers to clearly perceive the depth and distance of the video, thus providing an immersive viewing experience. MV-HEVC allows multiple image information to be included in the same video frame data. For example, each frame of MV-HEVC spatial video can contain image information for the left eye (e.g., left-eye video frame data) and image information for the right eye (e.g., right-eye video frame data). Unlike traditional side-by-side display methods, MV-HEVC stores information through the difference between the main view and the auxiliary view, allowing it to be played as ordinary HEVC video even on devices that do not support 3D viewing.

[0053] For example, the first and second video frames can be encoded. For instance, they can be encoded according to the shooting order of each video frame (including the first and second video frames). Specifically, first and second video frames with matching timestamps can be encoded simultaneously; alternatively, they can be encoded sequentially, such as in adjacent orders. The specific encoding method used for the first and second video frames is not limited.

[0054] Understandably, before encoding the first and second video frames, they can be rendered. For example, the queue of first frames captured by the first camera and the queue of second frames captured by the second camera can be alternately drawn onto a surface created by the video encoder. The rendered first and second video frames can then be obtained through a Graphic Buffer Source. Here, the Surface can be an object pointing to video memory, used to draw image content to display it on the screen, thus achieving a smooth display. The Graphic Buffer typically refers to a memory buffer that contains image data, such as a data queue, that will be displayed on the screen.

[0055] S103. Generate a target video file for the target spatial video based on the media data and the encoding / decoding feature data of the target spatial video, wherein the encoding / decoding feature data includes first encoding / decoding feature data and second encoding / decoding feature data.

[0056] Codec Specific Data (CSD) can be understood as data used to describe codec characteristics, and it can be a set of sequence parameters of metadata. For example, codec specific data includes first-view codec specific data and second-view codec specific data. First-view codec specific data can be used to describe the codec characteristics of a first-view video frame, and second-view codec specific data can be used to describe the codec characteristics of a second-view video frame. When the first-view video frame is a left-view video frame and the second-view video frame is a right-view video frame, the first-view codec specific data can be left-view codec specific data, and the second-view codec specific data can be right-view codec specific data.

[0057] Encoding and decoding feature data can be generated during the encoding of video frames. For example, the first-view encoding and decoding feature data of the target spatial video can be determined and generated before, after, or during the encoding of the first-view video frame of the target spatial video; and the second-view encoding and decoding feature data of the target spatial video can be determined and generated before, after, or during the encoding of the second-view video frame of the target spatial video.

[0058] For example, the first-view codec feature data may include the video parameter set (VPS) of the target spatial video, the sequence parameter set (SPS) of the first-view video frame, and the picture parameter set (PPS) of the first-view video frame. The second-view codec feature data may include the sequence parameter set of the second-view video frame and the picture parameter set of the second-view video frame. Since the first-view video frame and the second-view video frame can share the same video parameter set, when the first-view codec feature data contains the video parameter set, the second-view codec feature data may not contain the video parameter set, thereby reducing the storage space occupied by the codec feature data.

[0059] The video parameter set can be used to transmit video classification information, such as describing the overall structure of the encoded video sequence, including temporal sub-layer dependencies. The sequence parameter set can describe the configuration information of the video sequence; for example, it can store a set of global parameters for the encoded video sequence, which may include encoding level and / or resolution. An encoded video sequence can be understood as a sequence composed of encoded frames from the original video. The image parameter set can describe the encoding parameters of the video frames; these parameters can be understood as parameters related to image processing and encoding, such as image size, frame rate, and / or color space.

[0060] The target video file can be understood as a video file of the target space video. For example, the target video file can be a Moving Picture Experts Group 4 (MP4) file.

[0061] Specifically, a target video file for the target spatial video can be generated based on the media data and encoding / decoding feature data of the target spatial video.

[0062] Taking an MP4 file as the target video file, after encoding each video frame, the encoded video frame data (including the first and second video frame data) can be stored in real time in the media data chunk of the target video file; and the encoding and decoding feature data (including the first and second encoding and decoding feature data) of the target video can be stored in the metadata chunk (MOOV) of the target video file, thereby encapsulating the target video file.

[0063] The spatial video generation method provided in this embodiment involves capturing a first frame queue using a first camera and a second frame queue using a second camera. The first frame queue contains at least one first-view video frame, and the second frame queue contains at least one second-view video frame. The first and second video frames are encoded to obtain media data for the target spatial video. A target video file for the target spatial video is generated based on the media data and the codec feature data of the target spatial video. This codec feature data includes first-view and second-view codec feature data. This embodiment utilizes the above technical solution to store the first-view and second-view codec feature data of the spatial video into the target spatial video file during spatial video generation. The first-view codec feature data indicates the codec features of the first-view video frame, and the second-view codec feature data indicates the codec features of the second-view video frame. This enables spatial video generation in the Android system, enriching the methods for generating spatial videos.

[0064] Figure 2 This is a flowchart illustrating another method for generating spatial video according to an embodiment of this disclosure. The solution in this embodiment can be combined with one or more optional solutions in the above embodiments. Optionally, generating the target video file of the target spatial video based on the media data and the codec feature data of the target spatial video includes: storing the media data in the media data block of the target video file; and, in response to the completion of media data storage, storing the codec feature data in the metadata block of the target video file to obtain the target video file of the target spatial video.

[0065] Correspondingly, such as Figure 2 As shown, the spatial video generation method provided in this embodiment may include:

[0066] S201. Capture a first frame queue using a first camera and a second frame queue using a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame.

[0067] S202. Encode the first target video frame and the second target video frame to obtain the media data of the target space video.

[0068] S203. Store the media data in the media data block of the target video file.

[0069] In this embodiment, after encoding the media data of the target spatial video, the media data of the target spatial video can be stored in the media data chunk of the target video file.

[0070] For example, the media data of the target spatial video can be stored in real time into the media data block of the target video file. For instance, during the shooting process, the captured video frames of the target spatial video can be encoded, and after encoding a specific video frame data of the target spatial video, that video frame data can be written into the Chunk of the target video file. The specific video frame data of the target spatial video may include first and second video frame data corresponding to the timestamp of that video frame data.

[0071] S204. In response to the completion of media data storage, the encoding and decoding feature data is stored in the metadata block of the target video file to obtain the target video file of the target spatial video, wherein the encoding and decoding feature data includes first-order encoding and decoding feature data and second-order encoding and decoding feature data.

[0072] Specifically, once the media data of the target video is stored, the codec feature data of the target video can be saved to the metadata block of the target video file. For example, the metadata block of the target video file can be nested with a configuration information data block HVC1, and the codec feature data of the target video file can be stored in the HVC1 of the target video file.

[0073] In this embodiment, the encoding and decoding feature data of the target spatial video may include the first-order encoding and decoding feature data and the second-order encoding and decoding feature data of the target spatial video.

[0074] Taking the storage of codec feature data of target spatial video in HVC1 of target video file as an example, the first and second codec feature data of target spatial video can be stored in the same or different sub-data blocks (Boxes) of HVC1.

[0075] In some examples, a first target identifier can be added to the first-view codec feature data of the target spatial video, and a second target identifier can be added to the second-view codec feature data of the target spatial video. The first-view codec feature data with the added first target identifier and the second-view codec feature data with the added second target identifier are then stored in the same sub-data block of HVC1, such as in the hvcC box of HVC1. After storage, the stored first-view codec feature data and second-view codec feature data can be distinguished by the first and second target identifiers.

[0076] In other examples, the first and second category codec feature data of the target spatial video can be stored in different sub-data blocks of HVC1, i.e., in different configuration information sub-data blocks. Optionally, the first and second category codec feature data are stored in different configuration information sub-data blocks of the metadata block. For example, HVC1 may include an hvcC box, and an evcC box can be customized within HVC1. Thus, the first category codec feature data of the target spatial video can be stored in the hvcC box of HVC1, and the second category codec feature data of the target spatial video can be stored in the evcC box of HVC1. After storage, the first and second category codec feature data of the target spatial video can be distinguished by the different stored sub-data block boxes. The customization method of the evcC box is not limited; for example, except for the different codec feature data used for storage, the parameters of the evcC box can be the same as those of the hvcC box.

[0077] In some embodiments, the media data includes continuous video frame data. Encoding the first and second video frames to obtain media data for the target spatial video includes: determining the first video frame currently to be encoded in the first frame queue and the second video frame currently to be encoded in the second frame queue, wherein the timestamps of the first and second video frames to be encoded match; encoding the first video frame to be encoded to obtain first video frame data, and encoding the second video frame to be encoded to obtain second video frame data; storing the media data in the media data block of the target video file includes: packaging the first and second video frame data into current video frame data of the target spatial video, and storing the current video frame data in the media data block of the target video file.

[0078] In the above implementation, the first and second video frames with matching timestamps can be used as the left and right video frames that need to be displayed synchronously in the target space video, and they can be encoded and stored together.

[0079] In this context, timestamp matching can be understood as timestamps being the closest. The first video frame to be encoded can be understood as the first video frame that needs encoding processing. The second video frame to be encoded can be understood as the second video frame that needs encoding processing. The first video frame data can be the video frame data obtained by encoding the aforementioned first video frame to be encoded. The second video frame data can be the video frame data obtained by encoding the aforementioned second video frame to be encoded. The current video frame data can be understood as the video frame data resulting from packaging the first and second video frame data; that is, the packaged data of the first and second video frame data.

[0080] For example, each first video frame and each second video frame can be encoded sequentially according to their arrangement order in the first frame queue.

[0081] Specifically, based on the order of each first video frame in the first frame queue, the first video frame that needs to be encoded in the first frame queue can be determined as the current first video frame to be encoded, and the second video frame whose timestamp is closest to the timestamp of the current first video frame to be encoded can be obtained from the second frame queue as the current second video frame to be encoded.

[0082] Next, the first video frame to be encoded can be encoded to obtain the first video frame data, and the second video frame to be encoded can be encoded to obtain the second video frame data. The first and second video frames with matching timestamps can be encoded simultaneously, or one of the matching video frames can be encoded first, and then the other video frame can be encoded after the first frame is encoded. The specific configuration can be flexibly set according to needs.

[0083] After obtaining the first and second video frame data, these data can be packaged together to form a single video frame of the target space video. That is, each frame of the target space video can include both the first and second video frame data. After packaging, this packaged video frame data can be stored as a single video frame in the target space video's media data chunk. This process can be repeated to achieve the encoding and storage of the target space video's media data.

[0084] In some implementations, the first and second video frame data within the video frame data can be distinguished by their packing order. For example, the first and second video frame data can be packed according to a pre-set packing order. Therefore, the first and second video frame data within each video frame data of the target spatial video can be distinguished subsequently based on this packing order.

[0085] In some implementations, the first video frame may carry a first target identifier, and the second video frame may carry a second target identifier. Thus, after encoding / decoding and / or packaging, the first and second video frame data can be distinguished according to their respective identifiers. Optionally, before determining the first video frame currently to be encoded in the first frame queue and the second video frame currently to be encoded in the second frame queue, the method further includes: adding a first target identifier to the first video frame in the first frame queue and adding a second target identifier to the second video frame in the second frame queue. The first target identifier can be used to indicate that the corresponding data corresponds to the first target, and the second target identifier can be used to indicate that the corresponding data corresponds to the second target.

[0086] Figure 3 This is a schematic diagram illustrating a spatial video generation process provided in an embodiment of this disclosure. In some optional embodiments, such as... Figure 3 As shown, the process of generating spatial video can be described as follows:

[0087] A1. After detecting that the user has performed a recording operation, such as after detecting that the user has triggered the recording control, control the stereo camera to take pictures and obtain the left eye frame queue captured by the left eye camera and the right eye frame queue captured by the right eye camera.

[0088] A2. The renderer alternately draws the image frames of the left and right eye frame queues onto the Surface created by the video encoder (Codec), and obtains the rendered left and right eye video frames through the Graphic Buffer Source, that is, obtains the rendered left and right eye frames.

[0089] A3. Add target identifiers to the left and right frames. For example, add a left target identifier view-id: 0 to the left frame and a right target identifier view-id: 1 to the right frame.

[0090] A4. Store the left and right frames in an encoding work example C2Work and send it to the video encoder android codec2 for encoding.

[0091] A5. After encoding, the video encoder outputs the encoded left-eye video frame data (i.e., left-eye video data), right-eye video frame data (i.e., right-eye video data), and codec feature data (CSD). The left-eye CSD includes VPS, SPS, and PPS, while the right-eye CSD includes SPS and PPS. At this point, the left-eye and right-eye CSDs can be transmitted separately to the mediamuxer; and the left-eye and right-eye video data with matching timestamps can be packaged together and transmitted to the mediamuxer.

[0092] For example, when encoding the first left eye frame and the first right eye frame, the video encoder can output the encoded first left eye video frame data, the first right eye video frame data, and the CSD; when encoding the left eye frame and the right eye frame other than the first left eye frame and the first right eye frame, it can output the encoded left eye video frame data and the right eye video frame data, in which case there is no need to output the CSD.

[0093] Accordingly, after receiving the first frame left video data, the first frame right video data, and the CSD from the video encoder, the current application can separate the first frame video data and the CSD, and first send the CSD to the mediamuxer, and then send the first frame video data to the mediamuxer. When the current application receives other video data from the video encoder besides the first frame video data, it can send them to the mediamuxer.

[0094] A6. After receiving the CSDs of the left and right destinations, MediaMuxer packages and stores the two CSDs into the hvcC box and the custom evcC box in the hvc1 of the MP4 file, respectively, and writes the received video data into the chunk of the MP4 file.

[0095] Specifically, the hvc1 of an MP4 file can include an hvcC box and a custom evcC box. After receiving the CSD, MediaMuxer can store it in the cache; after receiving video data, MediaMuxer can write it into the MP4 file's chunks in real time. After all video data has been stored, the left video CSD can be written to the hvcC box in hvc1, and the right video CSD data can be written to the custom evcC box.

[0096] In addition, such as Figure 3As shown, the audio data collected by the microphone is encoded by the audio encoder and then sent to MediaMuxer, which stores it into an MP4 file.

[0097] Therefore, in the above embodiment, during encoding, time-stamped left and right video data can be continuously read from the Graphic Buffer Source. This time-stamped left and right video data is then sent together to the video encoder for encoding. The video encoder first outputs the video configuration data (CSD data). Because it's MV-HEVC, it contains two sets of CSDs: left and right video CSDs. After transmitting these two sets of CSDs to the MediaMuxer, the left video data is written to the hvcCbox, and the right video data is written to a custom evcCbox. The encoded, time-stamped left and right video data are then packaged together and transmitted to the mediaMuxer for storage.

[0098] The spatial video generation method provided in this embodiment can generate spatial videos in the Android system, enriching the ways in which spatial videos can be generated.

[0099] Figure 4 This is a flowchart illustrating a method for playing spatial video according to an embodiment of this disclosure. The method can be executed by a spatial video playback device, which can be implemented in software and / or hardware and can be configured in an electronic device, typically in a VR device. The spatial video playback method provided in this disclosure is applicable to scenarios involving playing spatial video, such as playing spatial video using a VR device with an Android operating system. Figure 4 As shown, the spatial video playback method provided in this embodiment may include:

[0100] S301. Obtain the encoding / decoding feature data and media data from the target video file, wherein the target video file is a video file of the target spatial video, and the encoding / decoding feature data includes first-order encoding / decoding feature data and second-order encoding / decoding feature data.

[0101] Specifically, when a playback operation for a target spatial video is received, the target video file of the target spatial video can be obtained; the target video file can be parsed to obtain the encoding and decoding feature data of the target spatial video, and the media data of the target spatial video can be obtained from the target video file.

[0102] In some implementations, the codec feature data of the target spatial video can be stored in the metadata block of the target video file, and the media data of the target spatial video can be stored in the media data block of the target video file. Therefore, the codec feature data of the target spatial video can be parsed from the metadata block of the target video file, and the media data of the target spatial video can be parsed from the media data block of the target video file. Optionally, obtaining the codec feature data and media data from the target video file includes: parsing the metadata block of the target video file to obtain the codec feature data of the target spatial video; and parsing the media data block of the target video file to obtain the media data of the target spatial video.

[0103] In some implementations, the codec feature data of the target spatial video may include first-view codec feature data (such as left-view codec feature data) and second-view codec feature data (such as right-view codec feature data). The first-view and second-view codec feature data of the target spatial video may be stored in a configuration information data block HVC1 nested within the metadata. For example, the first-view and second-view codec feature data of the target spatial video may be stored in the same or different sub-data blocks of HVC1.

[0104] Optionally, obtaining the codec feature data of the target spatial video includes: obtaining the first target codec feature data of the target spatial video from the first configuration information sub-data block of the metadata block; and obtaining the second target codec feature data of the target spatial video from the second configuration information sub-data block of the metadata block.

[0105] The first configuration information sub-data block can be understood as a sub-data block used to store first-end codec feature data, such as the sub-data block used to store first-end codec feature data in HVC1. The second configuration information sub-data block can be understood as a sub-data block used to store second-end codec feature data, such as the sub-data block used to store second-end codec feature data in HVC1. For example, the first configuration information sub-data block can be an hvcC box in HVC1, and the second configuration information sub-data block can be a custom evcC box in HVC1.

[0106] For example, the first-view codec feature data of the target spatial video can be stored in the first configuration information sub-data block, and the second-view codec feature data of the target spatial video can be stored in the second configuration information sub-data block. Therefore, the first-view codec feature data of the target spatial video can be obtained from the first configuration information sub-data block, and the second-view codec feature data of the target spatial video can be obtained from the second configuration information sub-data block. In other words, the codec feature data stored in the first configuration information sub-data block can be determined as the first-view codec feature data of the target spatial video, and the codec feature data stored in the second configuration information sub-data block can be determined as the second-view codec feature data of the target spatial video.

[0107] S302. Decode the media data according to the encoding and decoding feature data to obtain a first video stream and a second video stream, wherein the timestamps of video frames at the same position in the first video stream and the second video stream are matched.

[0108] Specifically, after obtaining the encoding and decoding feature data of the target spatial video, the media data of the target spatial video can be decoded based on the encoding and decoding feature data of the target spatial video, and the first video frame data after decoding can be rendered sequentially by a renderer to obtain the first video stream of the target spatial video, and the second video frame data after decoding can be rendered sequentially by a renderer to obtain the second video stream of the target spatial video.

[0109] In some implementations, the codec feature data of the target spatial video can be obtained first by parsing, and the decoder (such as a video decoder) can be configured based on this codec feature data, such as configuring the corresponding parameters of the decoder. After configuration, the media data of the target spatial video can be parsed, and the configured decoder can be used to decode the media data of the target spatial video. After decoding, the first and second video streams of the target spatial video can be rendered. Optionally, the step of decoding the media data according to the codec feature data to obtain the first and second video streams includes: configuring the decoder according to the codec feature data; and after the decoding configuration is completed, decoding the media data through the decoder to obtain the first and second video streams.

[0110] In some embodiments, the media data includes continuous video frame data. The step of decoding the media data using the decoder to obtain a first video stream and a second video stream includes: obtaining the current video frame data to be decoded according to the arrangement order of the video frame data; splitting the current video frame data to be decoded to obtain a first video frame data and a second video frame data to be decoded; decoding the first video frame data to be decoded using the decoder to obtain a first video frame in the first video stream; and decoding the second video frame data to be decoded using the decoder to obtain a second video frame in the second video stream.

[0111] The video frame data to be decoded can be understood as the video frame data that needs to be decoded. It can be the video frame data with the earliest timestamp that has not yet been decoded in the video file of the target space video.

[0112] In the above embodiments, the media data of the target spatial video may include continuous video frame data of the target spatial video. Each video frame data includes first video frame data and second video frame data. That is, the video frame data of the first and second video frames with matching timestamps of the target spatial video can be packaged and stored in the target video file of the target spatial video.

[0113] During decoding, for example, the current video frame data to be decoded can be determined based on the order of the video frame data.

[0114] After determining the current video frame data to be decoded, this data can be split into a first video frame and a second video frame. For example, the first video frame may carry a first target identifier, and the second video frame may carry a second target identifier. Thus, the first video frame carrying the first target identifier can be obtained, and the second video frame carrying the second target identifier can also be obtained.

[0115] After acquiring the first and second video frame data to be decoded, the decoder can process the first video frame data and render it to obtain the first video frame; similarly, the decoder can process the second video frame data and render it to obtain the second video frame. The decoder can process the first video frame data based on the first video frame's encoding / decoding feature data, and it can also process the second video frame data based on the second video frame's encoding / decoding feature data.

[0116] Therefore, as the decoder continues decoding and the renderer continues rendering, the first and second video streams of the target space video can be obtained.

[0117] S303. Play the first video stream on the first display screen and play the second video stream on the second display screen, wherein the first video stream and the second video stream are played synchronously.

[0118] The VR device can be equipped with at least two displays: a first-view display and a second-view display. The first-view display can be understood as the display on the VR device used to play the first-view video stream, and the second-view display can be understood as the display on the VR device used to play the second-view video stream.

[0119] In this embodiment, when playing the target space video, the first-view video stream of the target space video can be played simultaneously on the first-view display screen and the second-view video stream of the target space video can be played simultaneously on the second-view display screen. Taking the first-view video stream as the left-view video stream and the second-view video stream as the right-view video stream as an example, each left-view video frame of the target space video can be displayed sequentially on the left-view display screen (i.e., the first-view display screen), and each right-view video frame of the target space video can be displayed sequentially on the right-view display screen (i.e., the second-view display screen), thereby realizing the playback of the target space video. Among them, the left-view video frames and right-view video frames with matching timestamps are displayed synchronously.

[0120] Figure 5 This disclosure provides a schematic diagram of a spatial video playback process according to an embodiment. In some optional embodiments, such as... Figure 5 As shown, the playback process of spatial video can be described as follows:

[0121] B1. Using ffmpeg as the depacker, the MP4 file of the video is parsed by ffmpeg. If the evcC box is parsed, it means that the video is an MV-HEVC space video. The left target identifier is added to the parsed hvcc (i.e., left CSD), and the right target identifier is added to the parsed evcC (i.e., right CSD). The video is then packaged into a CSD and transmitted to the video decoder.

[0122] B2. After receiving the spatial video CSD, the video decoder can split it into left-eye CSD and right-eye CSD based on the left-eye and right-eye CSDs, and configure the decoding based on these left-eye and right-eye CSDs.

[0123] B3. After the video decoder is configured, the video data of spatial video can be read using ffmpeg.

[0124] B4. After reading the spatial video data, the data can be split into left-view and right-view video data. These split data are then placed into c2work and sent to the video decoder for decoding. Correspondingly, the video decoder can distinguish the received left-view and right-view video data based on the left-view identifier (view-id: 0) and the right-view identifier (view-id: 1), and decode the left-view video data according to the left-view CSD, and the right-view video data according to the right-view CSD.

[0125] B5. After the video decoder decodes the left and right video data, the video renderer can render the data based on the decoded data to obtain the left and right frames. If the decoded left and right video data are output to the surface, the display module can obtain the left and right frames and display them on the corresponding display screens.

[0126] In addition, such as Figure 5 As shown, while decoding and rendering video frame data, the audio decoder also decodes the audio data of the target space video. The decoded audio data can be sent to the audio renderer for rendering and playback.

[0127] Therefore, in the above embodiment, during decoding, ffmpeg can parse the MP4 file. When parsing the hvc1 box, if the evcC box is parsed, the hvcC data and evcC data are sent together to the decoding module. The decoding module reads and separates the VPS, SPS, and PPS of the left-eye data (i.e., left-eye CSD) and the SPS and PPS of the right-eye data (i.e., right-eye CSD), and then sends them to the video decoder for configuration. When reading video data, ffmpeg reads a sample (containing left and right eye data). The video decoder splits the left and right eye data into two parts and then sends them together to the video decoder for decoding.

[0128] This embodiment provides a method for playing spatial video. It acquires codec feature data and media data from a target video file, where the target video file is a target spatial video file. The codec feature data includes first-view codec feature data and second-view codec feature data. Based on this codec feature data, the media data is decoded to obtain a first-view video stream and a second-view video stream. The timestamps of video frames at the same position in the first-view and second-view video streams are matched. The first-view video stream is played on a first-view display screen, and the second-view video stream is played on a second-view display screen, with the first-view and second-view video streams played synchronously. This embodiment utilizes the above technical solution to decode the media data in the video file based on the first-view and second-view codec feature data when playing spatial video, enabling spatial video playback in the Android system and enriching the playback methods for spatial video.

[0129] Figure 6 This is a structural block diagram of a spatial video generation apparatus provided in an embodiment of this disclosure. The apparatus can be implemented by software and / or hardware and can be configured in an electronic device, typically a virtual reality (VR) device, mobile phone, or tablet computer. It can capture spatial video by executing a spatial video generation method, such as capturing spatial video using an electronic device with an Android operating system. Figure 6 As shown, the spatial video generation device provided in this embodiment may include: a shooting module 601, an encoding module 602, and a generation module 603, wherein,

[0130] The shooting module 601 is used to shoot a first frame queue through a first camera and a second frame queue through a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame.

[0131] The encoding module 602 is used to encode the first target video frame and the second target video frame to obtain media data of the target space video.

[0132] The generation module 603 is used to generate a target video file of the target spatial video based on the media data and the encoding and decoding feature data of the target spatial video, wherein the encoding and decoding feature data includes first encoding and decoding feature data and second encoding and decoding feature data.

[0133] The spatial video generation apparatus provided in this embodiment uses a shooting module 601 to capture a first frame queue using a first camera and a second frame queue using a second camera. The first frame queue contains at least one first-view video frame, and the second frame queue contains at least one second-view video frame. An encoding module encodes the first and second video frames to obtain media data for the target spatial video. A generation module generates a target video file for the target spatial video based on the media data and the encoding / decoding feature data of the target spatial video. This encoding / decoding feature data includes first-view and second-view encoding / decoding feature data. This embodiment utilizes the above technical solution to store the first and second-view encoding / decoding feature data of the spatial video into the target spatial video file during spatial video generation. The first-view encoding / decoding feature data indicates the encoding / decoding features of the first-view video frame, and the second-view encoding / decoding feature data indicates the encoding / decoding features of the second-view video frame. This enables spatial video generation in the Android system, enriching the methods for generating spatial videos.

[0134] Optionally, the generation module 603 includes: a media data storage unit, used to store the media data in the media data block of the target video file; and a feature data storage unit, used to store the encoding and decoding feature data in the metadata block of the target video file in response to the completion of media data storage, to obtain the target video file of the target spatial video.

[0135] Optionally, the first target encoding / decoding feature data and the second target encoding / decoding feature data are stored in different configuration information sub-data blocks of the metadata block.

[0136] Optionally, the media data includes continuous video frame data, and the encoding module 602 includes: a video frame determination unit, used to determine the first video frame to be encoded in the first frame queue and the second video frame to be encoded in the second frame queue, wherein the timestamps of the first video frame to be encoded and the second video frame to be encoded are matched; an encoding unit, used to encode the first video frame to be encoded to obtain first video frame data, and to encode the second video frame to be encoded to obtain second video frame data; the media data storage unit may be specifically used to: package the first video frame data and the second video frame data into the current video frame data of the target spatial video, and store the current video frame data in the media data block of the target video file.

[0137] Furthermore, the spatial video generation apparatus may further include: an identifier adding module, configured to add a first identifier to the first video frame in the first frame queue and a second identifier to the second video frame in the second frame queue before determining the first video frame to be encoded in the first frame queue and the second video frame to be encoded in the second frame queue.

[0138] The spatial video generation apparatus provided in this disclosure can execute the spatial video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the spatial video generation method. Technical details not described in detail in this embodiment can be found in the spatial video generation method provided in any embodiment of this disclosure.

[0139] Figure 7 This is a structural block diagram of a spatial video playback device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware, and can be configured in an electronic device, typically in a VR device. It can play spatial video scenes by executing a spatial video playback method, such as playing spatial video on a VR device with an Android operating system. Figure 7 As shown, the spatial video playback device provided in this embodiment may include: an acquisition module 701, a decoding module 702, and a display module 703, wherein,

[0140] The acquisition module 701 is used to acquire encoding and decoding feature data and media data in a target video file, wherein the target video file is a video file of a target spatial video, and the encoding and decoding feature data includes first-order encoding and decoding feature data and second-order encoding and decoding feature data.

[0141] Decoding module 702 is used to decode the media data according to the encoding and decoding feature data to obtain a first video stream and a second video stream, wherein the timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other.

[0142] Display module 703 is used to play the first video stream on a first display screen and the second video stream on a second display screen, wherein the first video stream and the second video stream are played synchronously.

[0143] The spatial video playback device provided in this embodiment acquires codec feature data and media data from a target video file through an acquisition module. This target video file is a video file containing a target spatial video. The codec feature data includes first-view codec feature data and second-view codec feature data. A decoding module decodes the media data based on this codec feature data to obtain a first-view video stream and a second-view video stream. The timestamps of video frames at the same position in the first-view and second-view video streams are matched. A display module plays the first-view video stream on a first-view display screen and the second-view video stream on a second-view display screen, with the first-view and second-view video streams played synchronously. This embodiment utilizes the above technical solution to decode the media data in the video file based on the first-view and second-view codec feature data when playing spatial video, enabling spatial video playback in the Android system and enriching the playback methods for spatial video.

[0144] Optionally, the acquisition module 701 includes: a feature data acquisition unit, used to parse the metadata block of the target video file to acquire the encoding and decoding feature data of the target spatial video; and a media data acquisition unit, used to parse the media data block of the target video file to acquire the media data of the target spatial video.

[0145] Optionally, the feature data acquisition unit is specifically configured to: acquire first-order codec feature data of the target spatial video from the first configuration information sub-data block of the metadata block; and acquire second-order codec feature data of the target spatial video from the second configuration information sub-data block of the metadata block.

[0146] Optionally, the decoding module 702 includes: a decoder configuration unit, configured to configure the decoder according to the encoding and decoding feature data; and a decoding unit, configured to decode the media data through the decoder after the decoding configuration is completed, to obtain a first video stream and a second video stream.

[0147] Optionally, the decoding unit is specifically configured to: obtain the current video frame data to be decoded according to the arrangement order of the video frame data; split the current video frame data to be decoded to obtain the current first video frame data to be decoded and the current second video frame data to be decoded; decode the current first video frame data to be decoded using the decoder to obtain a first video frame in the first video stream; and decode the second video frame data to be decoded using the decoder to obtain a second video frame in the second video stream.

[0148] The spatial video playback device provided in this disclosure can execute the spatial video playback method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the spatial video playback method. Technical details not described in detail in this embodiment can be found in the spatial video playback method provided in any embodiment of this disclosure.

[0149] The following is for reference. Figure 8 This illustration shows a structural diagram of an electronic device (e.g., a terminal device) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0151] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0152] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0153] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0154] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0155] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0156] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: capture a first frame queue using a first camera and a second frame queue using a second camera, wherein the first frame queue contains at least one first-view video frame and the second frame queue contains at least one second-view video frame; encode the first and second-view video frames to obtain media data of the target spatial video; and generate a target video file of the target spatial video based on the media data and the codec feature data of the target spatial video, wherein the codec feature data includes first-view codec feature data and second-view codec feature data. Alternatively

[0157] The codec feature data and media data in the target video file are obtained. The target video file is a video file of target spatial video. The codec feature data includes first-view codec feature data and second-view codec feature data. The media data is decoded according to the codec feature data to obtain a first-view video stream and a second-view video stream. The timestamps of video frames at the same position in the first-view video stream and the second-view video stream are matched. The first-view video stream is played on a first-view display screen, and the second-view video stream is played on a second-view display screen, wherein the first-view video stream and the second-view video stream are played synchronously.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of modules do not, in some cases, constitute a limitation on the unit itself.

[0161] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] According to one or more embodiments of this disclosure, Example 1 provides a method for generating spatial video, including:

[0164] A first frame queue is captured by a first camera, and a second frame queue is captured by a second camera, wherein the first frame queue contains at least one first-view video frame, and the second frame queue contains at least one second-view video frame.

[0165] The first and second target video frames are encoded to obtain media data of the target space video.

[0166] A target video file for the target spatial video is generated based on the media data and the encoding / decoding feature data of the target spatial video, wherein the encoding / decoding feature data includes first-order encoding / decoding feature data and second-order encoding / decoding feature data.

[0167] According to one or more embodiments of this disclosure, Example 2, based on the method described in Example 1, the step of generating a target video file for the target spatial video based on the media data and the codec feature data of the target spatial video includes:

[0168] The media data is stored in the media data block of the target video file;

[0169] In response to the completion of media data storage, the encoding and decoding feature data is stored in the metadata block of the target video file to obtain the target video file of the target spatial video.

[0170] According to one or more embodiments of this disclosure, Example 3, based on the method described in Example 2, stores the first target encoding / decoding feature data and the second target encoding / decoding feature data in different configuration information sub-data blocks of the metadata block.

[0171] According to one or more embodiments of this disclosure, Example 4 describes the method according to Example 2, wherein the media data includes continuous video frame data, and the encoding process of the first target video frame and the second target video frame to obtain media data of the target spatial video includes:

[0172] The first video frame to be encoded in the first frame queue and the second video frame to be encoded in the second frame queue are determined, wherein the timestamps of the first video frame to be encoded and the second video frame to be encoded are matched.

[0173] The first video frame to be encoded is encoded to obtain first video frame data, and the second video frame to be encoded is encoded to obtain second video frame data.

[0174] The step of storing the media data into the media data block of the target video file includes:

[0175] The first and second target video frame data are packaged into the current video frame data of the target spatial video, and the current video frame data is stored in the media data block of the target video file.

[0176] According to one or more embodiments of this disclosure, Example 5, based on the method of Example 4, further includes, before determining the first video frame currently to be encoded in the first frame queue and the second video frame currently to be encoded in the second frame queue:

[0177] Add a first target identifier to the first target video frame in the first frame queue, and add a second target identifier to the second target video frame in the second frame queue.

[0178] According to one or more embodiments of this disclosure, Example 6 provides a method for playing spatial video, including:

[0179] Obtain codec feature data and media data from a target video file, wherein the target video file is a video file of a target spatial video, and the codec feature data includes first-order codec feature data and second-order codec feature data;

[0180] The media data is decoded according to the encoding and decoding feature data to obtain a first video stream and a second video stream. The timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other.

[0181] The first video stream is played on a first display screen, and the second video stream is played on a second display screen, wherein the first video stream and the second video stream are played synchronously.

[0182] According to one or more embodiments of this disclosure, Example 7, based on the method described in Example 6, includes obtaining codec feature data and media data in the target video file, comprising:

[0183] The metadata blocks of the target video file are parsed to obtain the encoding and decoding feature data of the target spatial video; and

[0184] The media data blocks of the target video file are parsed to obtain the media data of the target spatial video.

[0185] According to one or more embodiments of this disclosure, Example 8 describes the method described in Example 7, wherein obtaining the codec feature data of the target spatial video includes:

[0186] Obtain the first target spatial video encoding / decoding feature data from the first configuration information sub-data block of the metadata block; and

[0187] The second target spatial video encoding and decoding feature data is obtained from the second configuration information sub-data block of the metadata block.

[0188] According to one or more embodiments of this disclosure, Example 9 describes the method according to any one of Examples 6-8, wherein decoding the media data based on the encoding / decoding feature data to obtain a first video stream and a second video stream includes:

[0189] Configure the decoder based on the encoding / decoding feature data;

[0190] After the decoding configuration is completed, the media data is decoded by the decoder to obtain the first video stream and the second video stream.

[0191] According to one or more embodiments of this disclosure, Example 10 describes the method according to Example 9, wherein the media data includes continuous video frame data, and the step of decoding the media data through the decoder to obtain a first video stream and a second video stream includes:

[0192] Based on the order of the video frame data, obtain the current video frame data to be decoded;

[0193] The current video frame data to be decoded is split to obtain the first video frame data to be decoded and the second video frame data to be decoded.

[0194] The decoder decodes the first video frame data to be decoded to obtain a first video frame in the first video stream, and the decoder decodes the second video frame data to be decoded to obtain a second video frame in the second video stream.

[0195] According to one or more embodiments of this disclosure, Example 11 provides an apparatus for generating spatial video, comprising:

[0196] The shooting module is used to capture a first frame queue through a first camera and a second frame queue through a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame.

[0197] The encoding module is used to encode the first target video frame and the second target video frame to obtain media data of the target space video.

[0198] The generation module is used to generate a target video file of the target spatial video based on the media data and the codec feature data of the target spatial video, wherein the codec feature data includes first codec feature data and second codec feature data.

[0199] According to one or more embodiments of this disclosure, Example 12 provides a spatial video playback device, comprising:

[0200] The acquisition module is used to acquire the codec feature data and media data in the target video file, wherein the target video file is a video file of the target spatial video, and the codec feature data includes first codec feature data and second codec feature data.

[0201] The decoding module is used to decode the media data according to the encoding and decoding feature data to obtain a first video stream and a second video stream, wherein the timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other.

[0202] The display module is used to play the first video stream on a first display screen and the second video stream on a second display screen, wherein the first video stream and the second video stream are played synchronously.

[0203] According to one or more embodiments of this disclosure, Example 13 provides an electronic device comprising:

[0204] One or more processors;

[0205] Memory, used to store one or more programs.

[0206] When the one or more programs are executed by the one or more processors, the one or more processors implement the spatial video generation method as described in any one of Examples 1-5 or the spatial video playback method as described in any one of Examples 6-10.

[0207] According to one or more embodiments of the present disclosure, Example 14 provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a method for generating spatial video as described in any one of Examples 1-5 or a method for playing spatial video as described in any one of Examples 6-10.

[0208] According to one or more embodiments of this disclosure, Example 15 provides a computer program product that, when executed by a computer, causes the computer to implement a spatial video generation method as described in any one of Examples 1-5 or a spatial video playback method as described in any one of Examples 6-10.

[0209] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0210] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0211] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for generating spatial video, characterized in that, include: A first frame queue is captured by a first camera, and a second frame queue is captured by a second camera, wherein the first frame queue contains at least one first-view video frame, and the second frame queue contains at least one second-view video frame. The first and second target video frames are encoded to obtain media data of the target space video. A target video file for the target spatial video is generated based on the media data and the encoding / decoding feature data of the target spatial video, wherein the encoding / decoding feature data includes first-order encoding / decoding feature data and second-order encoding / decoding feature data.

2. The method according to claim 1, characterized in that, The step of generating the target video file of the target spatial video based on the media data and the encoding / decoding feature data of the target spatial video includes: The media data is stored in the media data block of the target video file; In response to the completion of media data storage, the encoding and decoding feature data is stored in the metadata block of the target video file to obtain the target video file of the target spatial video.

3. The method according to claim 2, characterized in that, The first and second target encoding / decoding feature data are stored in different configuration information sub-data blocks of the metadata block.

4. The method according to claim 2, characterized in that, The media data includes continuous video frame data. The encoding process of the first and second target video frames to obtain the media data of the target spatial video includes: The first video frame to be encoded in the first frame queue and the second video frame to be encoded in the second frame queue are determined, wherein the timestamps of the first video frame to be encoded and the second video frame to be encoded are matched. The first video frame to be encoded is encoded to obtain first video frame data, and the second video frame to be encoded is encoded to obtain second video frame data. The step of storing the media data into the media data block of the target video file includes: The first and second target video frame data are packaged into the current video frame data of the target spatial video, and the current video frame data is stored in the media data block of the target video file.

5. The method according to claim 4, characterized in that, Before determining the first video frame to be encoded in the first frame queue and the second video frame to be encoded in the second frame queue, the method further includes: Add a first target identifier to the first target video frame in the first frame queue, and add a second target identifier to the second target video frame in the second frame queue.

6. A method for playing spatial video, characterized in that, include: Obtain codec feature data and media data from a target video file, wherein the target video file is a video file of a target spatial video, and the codec feature data includes first-order codec feature data and second-order codec feature data; The media data is decoded according to the encoding and decoding feature data to obtain a first video stream and a second video stream. The timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other. The first video stream is played on a first display screen, and the second video stream is played on a second display screen, wherein the first video stream and the second video stream are played synchronously.

7. The method according to claim 6, characterized in that, The acquisition of codec feature data and media data from the target video file includes: The metadata blocks of the target video file are parsed to obtain the encoding and decoding feature data of the target spatial video; and The media data blocks of the target video file are parsed to obtain the media data of the target spatial video.

8. The method according to claim 7, characterized in that, The acquisition of the encoding and decoding feature data of the target spatial video includes: Obtain the first target spatial video encoding / decoding feature data from the first configuration information sub-data block of the metadata block; and The second target spatial video encoding and decoding feature data is obtained from the second configuration information sub-data block of the metadata block.

9. The method according to any one of claims 6-8, characterized in that, The step of decoding the media data according to the encoding / decoding feature data to obtain the first video stream and the second video stream includes: Configure the decoder based on the encoding / decoding feature data; After the decoding configuration is completed, the media data is decoded by the decoder to obtain the first video stream and the second video stream.

10. The method according to claim 9, characterized in that, The media data includes continuous video frame data. The decoding process of the media data using the decoder to obtain the first video stream and the second video stream includes: Based on the order of the video frame data, obtain the current video frame data to be decoded; The current video frame data to be decoded is split to obtain the first video frame data to be decoded and the second video frame data to be decoded. The decoder decodes the first video frame data to be decoded to obtain a first video frame in the first video stream, and the decoder decodes the second video frame data to be decoded to obtain a second video frame in the second video stream.

11. A spatial video generation apparatus, characterized in that, include: The shooting module is used to capture a first frame queue through a first camera and a second frame queue through a second camera, wherein the first frame queue contains at least one first video frame and the second frame queue contains at least one second video frame. The encoding module is used to encode the first target video frame and the second target video frame to obtain media data of the target space video. The generation module is used to generate a target video file of the target spatial video based on the media data and the codec feature data of the target spatial video, wherein the codec feature data includes first codec feature data and second codec feature data.

12. A spatial video playback device, characterized in that, include: The acquisition module is used to acquire the codec feature data and media data in the target video file, wherein the target video file is a video file of the target spatial video, and the codec feature data includes first codec feature data and second codec feature data. The decoding module is used to decode the media data according to the encoding and decoding feature data to obtain a first video stream and a second video stream, wherein the timestamps of video frames at the same position in the first video stream and the second video stream are matched with each other. The display module is used to play the first video stream on a first display screen and the second video stream on a second display screen, wherein the first video stream and the second video stream are played synchronously.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which is executed by the at least one processor to enable the at least one processor to perform the spatial video generation method according to any one of claims 1-5 or the spatial video playback method according to any one of claims 6-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for generating spatial video according to any one of claims 1-5 or the method for playing spatial video according to any one of claims 6-10.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for generating spatial video according to any one of claims 1-5 or the method for playing spatial video according to any one of claims 6-10.

Images