Space video processing method, processing device, processing system and storage medium

Abstract

The invention discloses a space video processing method, a processing device, a processing system and a computer readable storage medium. The space video processing method comprises the following steps: simultaneously shooting different visual angles of the same scene through a left-eye camera and a right-eye camera to obtain a left-eye video stream and a right-eye video stream; respectively encoding the left-eye video stream and the right-eye video stream to obtain a left-eye encoded video stream and a right-eye encoded video stream; transmitting the left-eye coded video stream and the right-eye coded video stream to a server in real time, the server being used for performing video processing on the left-eye coded video stream and the right-eye coded video stream to obtain a first video stream to be played; sending the first video stream to be played to the client; and the client is used for performing video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing. Therefore, the real-time online playing of the space video is realized, and the latest space video content can be watched without recording the space video in advance.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a spatial video processing method, processing device, processing system, and computer-readable storage medium. Background Technology

[0002] Spatial video shooting is based on the principle of human binocular stereoscopic vision. It creates a stereoscopic image in the mind, incorporating parallax, and subjectively generates the perceived depth of the viewed object in space. Spatial video can provide users with an immersive 3D video experience, allowing them to observe the content from different angles and distances. Currently, how to achieve real-time playback of spatial video is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a spatial video processing method, processing apparatus, processing system, and computer-readable storage medium to solve at least one of the aforementioned technical problems.

[0004] The spatial video processing method of this application is applied to a video acquisition device, which includes a left-eye camera and a right-eye camera. The spatial video processing method includes:

[0005] By simultaneously capturing the same scene from different perspectives using the left and right cameras, a left-eye video stream and a right-eye video stream are obtained.

[0006] The left-eye video stream and the right-eye video stream are encoded respectively to obtain a left-eye encoded video stream and a right-eye encoded video stream;

[0007] The left-eye encoded video stream and the right-eye encoded video stream are transmitted to the server in real time. The server processes the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played. The first video stream to be played is sent to the client. The client processes the first video stream to be played to obtain a second video stream to be played, so as to play it in real time.

[0008] In some implementations, the encoding formats of the left-eye encoded video stream and the right-eye encoded video stream are AVC or HEVC formats.

[0009] In some implementations, the left-eye encoded video stream and the right-eye encoded video stream are transmitted in real time via RTMP.

[0010] The spatial video processing method of this application, applied to a server, includes:

[0011] The system receives left-eye encoded video streams and right-eye encoded video streams transmitted in real time from a video acquisition device. The left-eye encoded video streams and right-eye encoded video streams originate from: the video acquisition device simultaneously capturing images of the same scene from different perspectives using a left-eye camera and a right-eye camera, obtaining the left-eye video streams and right-eye video streams respectively; and the left-eye video streams and right-eye video streams are encoded separately to obtain the left-eye encoded video streams and right-eye encoded video streams.

[0012] Video processing is performed on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played;

[0013] The first video stream to be played is sent to the client, which processes the first video stream to obtain a second video stream to be played in real time.

[0014] In some embodiments, the step of performing video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played includes:

[0015] The left-eye encoded video stream and the right-eye encoded video stream are time-synchronized and merged to obtain a merged video stream;

[0016] The merged video stream is split and decoded to obtain a left-view video stream and a right-view video stream;

[0017] The corresponding video frames in the left-view video stream and the right-view video stream are rendered to the corresponding display areas of the output screen and merged into a merged screen.

[0018] Each frame of the merged image is encoded to obtain the first video stream to be played.

[0019] In some implementations, the step of time-synchronizing and merging the left-eye encoded video stream and the right-eye encoded video stream to obtain a merged video stream includes:

[0020] Obtain the video frame rate of a monocular encoded video stream, wherein the monocular encoded video stream is either the left-eye encoded video stream or the right-eye encoded video stream;

[0021] Obtain the system timestamp of the server;

[0022] Obtain the timestamp of the video frame in the monocular coded video stream that is closest to the system timestamp, and the corresponding monocular coded video frame;

[0023] Based on the system timestamp, the video frame timestamp, and the video frame rate, determine whether the monocular encoded video stream has dropped frames at the system timestamp;

[0024] When a frame is dropped in the monocular encoded video stream, a monocular predicted video frame is generated based on the convolutional long short-term memory network model, which serves as the synchronous video frame under the system timestamp for video stream merging.

[0025] When the monocular encoded video stream has no dropped frames, the monocular encoded video frames are used as synchronous video frames under the system timestamp for video stream merging.

[0026] In some implementations, generating monocular predicted video frames based on a convolutional long short-term memory network model as synchronized video frames under the system timestamp includes:

[0027] Obtain multiple monocular encoded video frames preceding the timestamp of the video frame;

[0028] Multiple monocular coded video frames are preprocessed to obtain a five-dimensional tensor, wherein the preprocessing includes normalization and format conversion.

[0029] Spatiotemporal features are extracted from the five-dimensional tensor based on the convolutional long short-term memory network model.

[0030] Based on the extracted spatiotemporal features, the monocular predicted video frame is generated and used as the synchronized video frame under the system timestamp.

[0031] In some implementations, the merged screen is a side-by-side format screen, and the first video stream to be played is a side-by-side format video stream.

[0032] In some implementations, the first video stream to be played is encoded in HEVC format.

[0033] In some implementations, the client includes multiple head-mounted display devices, and sending the first video stream to be played to the client includes:

[0034] The first video stream to be played is sent to the multiple head-mounted display devices via parallel forwarding.

[0035] The spatial video processing method of this application, applied to a client, includes:

[0036] The server receives a first video stream to be played, which originates from: a video capture device simultaneously capturing the same scene from different perspectives using a left-eye camera and a right-eye camera, obtaining a left-eye video stream and a right-eye video stream; the left-eye video stream and the right-eye video stream are encoded respectively to obtain a left-eye encoded video stream and a right-eye encoded video stream; the server receives the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video capture device; and performs video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played.

[0037] The first video stream to be played is processed to obtain a second video stream to be played in real time.

[0038] In some implementations, the client includes multiple head-mounted display devices, and the first video stream to be played, sent by the receiving server, includes:

[0039] The first video stream to be played is received in parallel by the plurality of head-mounted display devices.

[0040] In some implementations, the step of performing video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback includes:

[0041] The first video stream to be played is decoded and parsed to obtain a decoded video stream;

[0042] Spatial reconstruction and rendering are performed on the decoded video stream to obtain the second video stream to be played in real time.

[0043] The spatial video processing apparatus of this application is applied to a video acquisition device, the video acquisition device including a left eye camera and a right eye camera, and the spatial video processing apparatus includes:

[0044] The shooting module is used to simultaneously capture images of the same scene from different perspectives using the left eye camera and the right eye camera, thereby obtaining a left eye video stream and a right eye video stream;

[0045] The encoding module is used to encode the left-eye video stream and the right-eye video stream respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream;

[0046] The transmission module is used to transmit the left-eye encoded video stream and the right-eye encoded video stream to the server in real time. The server is used to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played. The first video stream to be played is sent to the client. The client is used to perform video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0047] The spatial video processing apparatus of this application embodiment is applied to a server, and the spatial video processing apparatus includes:

[0048] The first receiving module is used to receive the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device. The left-eye encoded video stream and the right-eye encoded video stream are obtained by the video acquisition device simultaneously capturing the same scene from different perspectives using the left-eye camera and the right-eye camera to obtain the left-eye video stream and the right-eye video stream; the left-eye video stream and the right-eye video stream are encoded respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream.

[0049] The first processing module is used to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played.

[0050] The sending module is used to send the first video stream to be played to the client. The client is used to perform video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0051] The spatial video processing apparatus of this application embodiment is applied to a client, and the spatial video processing apparatus includes:

[0052] The second receiving module is used to receive a first video stream to be played sent by the server. The first video stream to be played originates from: a video acquisition device simultaneously capturing the same scene from different perspectives using a left-eye camera and a right-eye camera, obtaining a left-eye video stream and a right-eye video stream; encoding the left-eye video stream and the right-eye video stream respectively, obtaining a left-eye encoded video stream and a right-eye encoded video stream; the server receives the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device; and performs video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played.

[0053] The second processing module is used to process the first video stream to be played to obtain a second video stream to be played in real time.

[0054] The spatial video processing system of this application includes one or more processors and a memory. The memory stores a computer program, which, when executed by the processor, implements the spatial video processing method of any of the above embodiments.

[0055] The computer-readable storage medium of the present application embodiments stores a computer program thereon, which, when executed by a processor, implements the spatial video processing method of any of the above embodiments.

[0056] In the spatial video processing method, processing apparatus, processing system, and computer-readable storage medium of this application, the left-eye video stream and right-eye video stream captured by the video acquisition device are encoded to obtain a left-eye encoded video stream and a right-eye encoded video stream, respectively. These can be transmitted in real time to a server for video processing to obtain a first video stream to be played. The first video stream to be played can be sent to a client, and after video processing, a second video stream to be played can be obtained. The second video stream to be played can then be played online in real time on the client. Thus, real-time online playback of spatial video is achieved, allowing users to view the latest spatial video content without the need for pre-recording.

[0057] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0058] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0059] Figure 1 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0060] Figure 2 This is a schematic diagram illustrating an application scenario of the spatial video processing method according to certain embodiments of this application;

[0061] Figure 3 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0062] Figure 4 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0063] Figure 5 This is a schematic diagram illustrating the distribution of single-frame video information in a merged video stream according to certain embodiments of this application;

[0064] Figure 6 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0065] Figure 7 This is a schematic diagram illustrating the time synchronization of the left-eye encoded video stream and the right-eye encoded video stream according to certain embodiments of this application.

[0066] Figure 8 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0067] Figure 9This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0068] Figure 10 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0069] Figure 11 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0070] Figure 12 This is a schematic flowchart of a spatial video processing method according to certain embodiments of this application;

[0071] Figure 13 This is a schematic diagram of a spatial video processing apparatus according to certain embodiments of this application;

[0072] Figure 14 This is a schematic diagram of a spatial video processing apparatus according to certain embodiments of this application;

[0073] Figure 15 This is a schematic diagram of a spatial video processing apparatus according to certain embodiments of this application;

[0074] Figure 16 This is a schematic diagram of the modules of a spatial video processing system according to certain embodiments of this application;

[0075] Figure 17 This is a schematic diagram illustrating the connection state between a computer-readable storage medium and a processor according to certain embodiments of this application. Detailed Implementation

[0076] The embodiments of this application will be further described below with reference to the accompanying drawings. The same or similar reference numerals in the drawings denote the same or similar elements or elements having the same or similar functions throughout. Furthermore, the embodiments of this application described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting this application.

[0077] Please see Figure 1 and Figure 2 This application provides a spatial video processing method applied to a video acquisition device 100, which includes a left eye camera 10 and a right eye camera 20. The spatial video processing method includes:

[0078] S101: Simultaneously capture the same scene from different perspectives using the left eye camera 10 and the right eye camera 20 to obtain the left eye video stream and the right eye video stream;

[0079] S102: Encode the left-eye video stream and the right-eye video stream separately to obtain the left-eye encoded video stream and the right-eye encoded video stream;

[0080] S103: The left-eye encoded video stream and the right-eye encoded video stream are transmitted to the server 200 in real time. The server 200 performs video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played. The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0081] In this embodiment, the left-eye and right-eye video streams captured by the video acquisition device 100 are encoded to obtain left-eye encoded video streams and right-eye encoded video streams, respectively. These encoded video streams can then be transmitted in real-time to the server 200 for video processing to obtain a first video stream to be played. This first video stream can be sent to the client 300, where it undergoes further processing to obtain a second video stream to be played. This second video stream can then be played online in real-time on the client 300. Thus, real-time online playback of spatial video is achieved, allowing users to view the latest spatial video content without the need for pre-recording.

[0082] Specifically, the video acquisition device 100 can be any binocular camera device. The video acquisition device 100 includes a left-eye camera 10 and a right-eye camera 20, which are separated by a predetermined distance to have different viewing angles. The predetermined distance is used to simulate the interpupillary distance of human eyes; in one example, the predetermined distance can be 67mm. By simultaneously capturing images of the same scene from different perspectives using the left-eye camera 10 and the right-eye camera 20, the way human eyes observe the world is simulated, resulting in a left-eye video stream and a right-eye video stream. There is parallax between the left-eye video stream and the right-eye video stream. It should be noted that, for ease of subsequent processing, the resolution of the left-eye video stream and the right-eye video stream must remain the same.

[0083] The left and right video streams are encoded separately to obtain the left-eye encoded video stream and the right-eye encoded video stream. These two streams are transmitted in real-time to server 200 for subsequent processing. It should be noted that this streaming transmission method effectively reduces latency during subsequent real-time playback.

[0084] Server 200 can process the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played, and send the first video stream to be played to client 300. Client 300 can be any device capable of playing spatial video, such as a head-mounted display device. Client 300 can process the first video stream to be played to obtain a second video stream to be played, and play the second video stream online in real time.

[0085] In related technologies, spatial videos need to be recorded in advance and then shared or copied to head-mounted display devices for offline playback, and online playback of spatial videos is not supported.

[0086] In this embodiment, the left-eye encoded video stream and the right-eye encoded video stream obtained after being captured and encoded by the video acquisition device 100 can be transmitted in real time to the server 200 for video processing to obtain a first video stream to be played. The first video stream to be played can be sent to the client 300, and after video processing, a second video stream to be played can be obtained for real-time online playback on the client 300. In this way, real-time online playback of spatial video is realized, and the latest spatial video content can be viewed without the need for pre-recording of spatial video.

[0087] In some implementations, the encoding formats of the left-eye encoded video stream and the right-eye encoded video stream are AVC or HEVC.

[0088] Specifically, the encoding formats of the left-eye encoded video stream and the right-eye encoded video stream can be Advanced Video Coding (AVC) format or High Efficiency Video Coding (HEVC) format.

[0089] In related technologies, specific types of video capture equipment are required to record spatial video, and the recorded spatial video must be encoded using a specific MultiView-HEVC (MV-HEVC) format. MV-HEVC format video streams are only supported for playback on specific client models. If playback is required on other client models, format conversion is necessary, which may negatively impact the quality of the spatial video.

[0090] In this embodiment, the video acquisition device 100 can be any binocular camera device, thereby improving the versatility of the spatial video acquisition device 100. The captured left-eye and right-eye video streams are encoded in AVC or HEVC format to obtain AVC or HEVC format encoded left-eye and right-eye video streams, which are suitable for playback on various client devices 300. Thus, the subsequent first video stream to be played can be sent to various different types of client devices 300 for real-time online playback, expanding the application scope of spatial video, promoting the popularization and development of spatial video technology, and providing a high-quality spatial video experience for a wider range of users.

[0091] Please see Figure 2 In some implementations, the left-eye encoded video stream and the right-eye encoded video stream are transmitted in real time via RTMP.

[0092] Specifically, the Real-Time Messaging Protocol (RTMP) is a real-time data transmission protocol that can be used for real-time transmission of video streams and streaming media services. Using RTMP, the left-eye encoded video stream and the right-eye encoded video stream are transmitted in real-time to the server 200, allowing the client 300 to subsequently play the spatial video online in real-time.

[0093] Please see Figure 2 and Figure 3 This application also provides a spatial video processing method, applied to a server 200, the spatial video processing method comprising:

[0094] S201: Receive the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device 100. The left-eye encoded video stream and the right-eye encoded video stream are obtained by the video acquisition device 100 simultaneously capturing the same scene from different perspectives through the left-eye camera 10 and the right-eye camera 20 to obtain the left-eye video stream and the right-eye video stream; encode the left-eye video stream and the right-eye video stream respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream.

[0095] S202: Perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played;

[0096] S203: The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain the second video stream to be played for real-time playback.

[0097] In this embodiment, the left-eye and right-eye video streams captured by the video acquisition device 100 are encoded to obtain left-eye encoded video streams and right-eye encoded video streams, respectively. These encoded video streams can then be transmitted in real-time to the server 200 for video processing to obtain a first video stream to be played. This first video stream can be sent to the client 300, where it undergoes further processing to obtain a second video stream to be played. This second video stream can then be played online in real-time on the client 300. Thus, real-time online playback of spatial video is achieved, allowing users to view the latest spatial video content without the need for pre-recording.

[0098] Specifically, server 200 simultaneously receives two video streams transmitted in real time from video acquisition device 100: one corresponding to the left-eye encoded video stream and the other to the right-eye encoded video stream. Server 200 performs video processing on the left-eye and right-eye encoded video streams to obtain a first video stream to be played. Then, the first video stream to be played is transmitted to client 300 using streaming media transmission. Client 300 performs video processing on the first video stream to obtain a second video stream to be played in real time.

[0099] Please see Figure 2 and Figure 4 In some implementations, video processing is performed on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played (i.e., S202), including:

[0100] S2021: Synchronize the left-eye encoded video stream and the right-eye encoded video stream and merge the video streams to obtain a merged video stream;

[0101] S2022: Decode the merged video stream separately to obtain the left-view video stream and the right-view video stream;

[0102] S2023: Render the corresponding video frames from the left-view video stream and the right-view video stream to the corresponding display areas of the output screen, and merge them into a merged screen;

[0103] S2024: Encode each frame of the merged video to obtain the first video stream to be played.

[0104] Specifically, to ensure that the left-eye encoded video stream and the right-eye encoded video stream are synchronized in time for subsequent processing, time synchronization can be performed on the two streams. The system timestamp of server 200 can be used to synchronize the left-eye encoded video stream and the right-eye encoded video stream.

[0105] After time synchronization, the left-eye encoded video stream and the right-eye encoded video stream are merged according to their timestamps to obtain a merged video stream. The distribution of single-frame video information in the merged video stream is as follows: Figure 5 As shown, it includes: packet header (0x00EE), timestamp information, packet length and information of the first video stream, packet length and information of the second video stream, and packet trailer information (0x00FF). It can be understood that the left-eye encoded video stream can be the first video stream, in which case the right-eye encoded video stream is the second video stream; conversely, the right-eye encoded video stream can be the first video stream, in which case the left-eye encoded video stream is the second video stream.

[0106] Next, each video frame in the merged video stream is decoded separately, that is, the merged video stream is divided into two streams, corresponding to the left and right perspectives when the video capture device 100 is shooting, to obtain the left perspective video stream and the right perspective video stream.

[0107] The video frames from the left-view video stream are rendered to the left half of the output screen, and the video frames from the right-view video stream are rendered to the right half of the output screen, thus simulating the left and right view layout when the video capture device 100 is shooting.

[0108] It should be noted that since the left-eye and right-eye video streams captured by video capture device 100 have the same resolution, the left-view and right-view video streams also have the same resolution. The size of the left and right halves of the video stream can be the same as or different from the resolution of the corresponding video stream. When the size of the left and right halves of the video stream is different from the resolution of the corresponding video stream, the size of the left and right halves of the video stream is proportional to the resolution of the video stream.

[0109] When the sizes of the left and right halves of the video stream are the same as their respective resolutions, the left-view and right-view video streams can be directly rendered to their corresponding areas. When the sizes of the left and right halves are different from their respective resolutions, the left-view and right-view video streams are first scaled up proportionally according to their sizes, and then rendered to their respective areas.

[0110] After the two video streams are rendered, the left and right halves of the output image are merged to obtain a complete merged image. This means integrating the pixel information of the left and right halves into the same frame, which facilitates subsequent encoding and transmission. After obtaining the merged image, each frame of the merged image is encoded to obtain the first video stream to be played.

[0111] Please see Figure 6 and Figure 7 In some implementations, the left-eye encoded video stream and the right-eye encoded video stream are time-synchronized and merged to obtain a merged video stream (i.e., S2021), including:

[0112] S20211: Obtain the video frame rate of a monocular encoded video stream, which can be a left-eye encoded video stream or a right-eye encoded video stream.

[0113] S20212: Get the system timestamp of server 200;

[0114] S20213: Obtain the timestamp of the video frame in the monocular encoded video stream that is closest to the system timestamp, and the corresponding monocular encoded video frame;

[0115] S20214: Determine whether a monocular encoded video stream drops frames at the system timestamp based on the system timestamp, video frame timestamp, and video frame rate;

[0116] S20215: When a monocular coded video stream drops a frame, a monocular predicted video frame is generated based on a convolutional long short-term memory network model and used as a synchronous video frame under the system timestamp for video stream merging.

[0117] S20216: When there are no dropped frames in the monocular encoded video stream, the monocular encoded video frames are used as synchronous video frames under the system timestamp for video stream merging.

[0118] It should be noted that during the transmission of the left-eye encoded video stream and the right-eye encoded video stream to server 200, network latency or packet loss may occur, resulting in dropped frames in both streams. This, in turn, affects the real-time performance and video quality of the first video stream to be played. Therefore, time synchronization and merging of the left-eye encoded video stream and the right-eye encoded video stream can be performed. The specific process is as follows.

[0119] Get the video frame rate of the monocular encoded video stream, expressed as Frames Per Second (FPS). Get the system timestamp of server 200, denoted as T0. Query the video frame timestamp in the monocular encoded video stream that is closest to the system timestamp, denoted as Tn. Simultaneously, get the monocular encoded video frame corresponding to the video frame timestamp, denoted as Fn.

[0120] Based on the system timestamp, video frame timestamp, and video frame rate, it can be determined whether a monocular encoded video stream has dropped frames at the system timestamp. The difference between the system timestamp and the video frame timestamp, T0-Tn, is calculated. Simultaneously, the reciprocal of the video frame rate, 1 / A, is calculated, and the relationship between the difference and the reciprocal of the video frame rate is compared.

[0121] When the difference is greater than or equal to the reciprocal of the video frame rate, the monocular coded video stream is determined to be in a dropped frame state under the current system timestamp. At this time, a predicted video frame is generated based on the Convolutional Long Short-Term Memory Network (ConvLSTM) model, denoted as Fn+1. The predicted video frame is used as the synchronous video frame under the current system timestamp for video stream merging.

[0122] When the difference is less than the reciprocal of the video frame rate, the monocular encoded video stream is determined to be in a non-dropped frame state under the current system timestamp. At this time, the monocular encoded video frame corresponding to the video frame timestamp is taken as the synchronization video frame under the current system timestamp for video stream merging.

[0123] The following is a detailed explanation of the time synchronization of the left-eye encoded video stream. The video frame rate of the left-eye encoded video stream is obtained, which is also the frame rate of the left-eye camera 10. When the video acquisition device 100 transmits the left-eye encoded video stream to the server 200 in real time, the video frame rate is transmitted to the server 200 along with the left-eye encoded video stream.

[0124] Obtain the system timestamp of server 200. For each system timestamp, query the video frame timestamp closest to the system timestamp in the left-eye encoded video stream, denoted as the first timestamp, and obtain the left-eye encoded video frame corresponding to the first timestamp. Calculate the difference between the system timestamp and the first timestamp, and compare the difference with the reciprocal of the video frame rate. When the difference is greater than or equal to the reciprocal of the video frame rate, it is determined that the left-eye encoded video stream is in a dropped frame state under the current system timestamp. At this time, a left-eye predicted video frame is generated based on the convolutional long short-term memory network model, which is used as the left-eye synchronized video frame under the current system timestamp, and merged with the corresponding right-eye synchronized video frame for video stream merging.

[0125] When the difference is less than the reciprocal of the video frame rate, the left-eye encoded video stream is determined to be in a non-dropped frame state under the current system timestamp. At this time, the left-eye encoded video frame corresponding to the first timestamp is taken as the left-eye synchronized video frame under the current system timestamp, and the video stream is merged with the corresponding right-eye synchronized video frame.

[0126] The following section provides a detailed explanation of the time synchronization of the right-eye encoded video stream. The video frame rate of the right-eye encoded video stream is obtained; this frame rate is also the frame rate of the right-eye camera 20. It is important to note that, for ease of time synchronization and video stream merging, the frame rate of the left-eye camera 10 is kept consistent with the frame rate of the right-eye camera 20. When the video acquisition device 100 transmits the right-eye encoded video stream to the server 200 in real time, the video frame rate is transmitted to the server 200 along with the right-eye encoded video stream.

[0127] Obtain the system timestamp of server 200. For each system timestamp, query the video frame timestamp closest to the system timestamp in the right-eye encoded video stream, denoted as the second timestamp, and obtain the right-eye encoded video frame corresponding to the second timestamp. Calculate the difference between the system timestamp and the second timestamp, and compare the difference with the reciprocal of the video frame rate. When the difference is greater than or equal to the reciprocal of the video frame rate, it is determined that the right-eye encoded video stream is in a dropped frame state under the current system timestamp. At this time, a right-eye predicted video frame is generated based on the convolutional long short-term memory network model, which is used as the right-eye synchronized video frame under the current system timestamp, and merged with the corresponding left-eye synchronized video frame for video stream merging.

[0128] When the difference is less than the reciprocal of the video frame rate, the right-eye encoded video stream is determined to be in a non-dropped frame state under the current system timestamp. At this time, the right-eye encoded video frame corresponding to the second timestamp is taken as the right-eye synchronized video frame under the current system timestamp, and the video stream is merged with the corresponding left-eye synchronized video frame.

[0129] It should be noted that the time synchronization of the left-eye encoded video stream and the right-eye encoded video stream is performed simultaneously. After obtaining multiple left-eye synchronized video frames corresponding to the left-eye encoded video stream and multiple right-eye synchronized video frames corresponding to the right-eye encoded video stream, the left-eye synchronized video frames and right-eye synchronized video frames under the same system timestamp are merged to obtain the merged video stream.

[0130] In this embodiment, the left-eye encoded video stream and the right-eye encoded video stream are time-synchronized. When frame drops occur, predicted video frames are generated based on a convolutional long short-term memory network model to fill in the missing frames. This avoids the delay of waiting for packet retransmission or re-requesting video frame transmission in the event of packet loss or network latency, mitigating the impact of network latency and packet loss on video playback, improving the real-time performance of the first video stream to be played, and thus improving the real-time performance of spatial video playback. Furthermore, it can reduce problems such as blurry images, stuttering, and frame drops in real-time video playback, improving video quality.

[0131] Please see Figure 7 and Figure 8 In some implementations, a monocular predicted video frame is generated based on a convolutional long short-term memory network model as a synchronized video frame under the system timestamp (i.e., S20215), including:

[0132] S202151: Obtain multiple monocular encoded video frames prior to the video frame timestamp;

[0133] S202152: Preprocess multiple monocular coded video frames to obtain the corresponding five-dimensional tensor. The preprocessing includes normalization and format conversion.

[0134] S202153: Spatiotemporal feature extraction of five-dimensional tensors based on convolutional long short-term memory network model;

[0135] S202154: Based on the extracted spatiotemporal features, generate monocular prediction video frames as synchronized video frames under the system timestamp.

[0136] Specifically, multiple monocularly coded video frames preceding the video frame timestamp are obtained. In one example, 10 monocularly coded video frames preceding the video frame timestamp are obtained, namely Fn-9, Fn-8, Fn-7, Fn-6, Fn-5, Fn-4, Fn-3, Fn-2, Fn-1, and Fn. Then, these multiple monocularly coded video frames are normalized and converted to their format to obtain the corresponding five-dimensional tensor.

[0137] Normalization refers to adjusting multiple monocular coded video frames to a uniform range or distribution. For example, it involves transforming the pixel values ​​of multiple monocular coded video frames from an initial range (such as 0 to 255) to a standard range (such as 0 to 1 or -1 to 1). Normalization can make multiple monocular coded video frames more suitable for inference and prediction in subsequent processing.

[0138] After normalization, multiple monocular coded video frames are converted into corresponding five-dimensional tensors. A five-dimensional tensor can be represented as (batch size, time steps, channels, height, width), where batch size is the number of samples, time steps are the time step size, channels are the number of video frame channels, height is the video frame height, and width is the video frame width. For example, for an RGB video consisting of 10 monocular coded video frames, each frame being 64x64 pixels, the corresponding five-dimensional tensor is (1, 10, 3, 64, 64).

[0139] After obtaining the five-dimensional tensor, spatiotemporal features are extracted from it based on a convolutional long short-term memory (LSTM) network model. The convolutional LSTM network model is a deep learning model that combines the characteristics of convolutional neural networks and LSTM networks. While including convolutional operations, it retains the temporal modeling capabilities of LSTM networks, thus enabling the extraction of temporal dependencies between multiple monocular coded video frames while preserving their spatial structural information. The extracted spatiotemporal features include both the spatial structural information of the multiple monocular coded video frames and the temporal dependencies between them.

[0140] The convolutional long short-term memory network model transforms the extracted spatiotemporal features into spatiotemporal feature maps as output. Using a 3D convolutional (Conv3D) decoder architecture, the spatiotemporal feature map is input into the decoder. The decoder decodes the spatiotemporal feature map to obtain the monocular predicted video frame Fn+1, which can then be used as a synchronized video frame under the system timestamp.

[0141] When a monocular encoded video frame is a left-eye encoded video frame, multiple left-eye encoded video frames prior to the first timestamp are obtained from the left-eye encoded video stream. These multiple left-eye encoded video frames are then normalized and format-converted to obtain the corresponding left-eye five-dimensional tensor. Spatiotemporal features are extracted from the left-eye five-dimensional tensor using a convolutional long short-term memory network model.

[0142] The extracted spatiotemporal features are output as a spatiotemporal feature map. Using Conv3D as the decoder architecture, the spatiotemporal feature map is input into the decoder. The decoder decodes the spatiotemporal feature map to obtain the left-eye predicted video frame, which is then used as the synchronized video frame under the current system timestamp. This solves the problem of dropped frames in the left-eye encoded video frame.

[0143] When a monocular coded video frame is a right-eye coded video frame, multiple right-eye coded video frames prior to the second timestamp are retrieved from the right-eye coded video stream. These multiple right-eye coded video frames are then normalized and format-converted to obtain the corresponding right-eye five-dimensional tensor. Spatiotemporal features are extracted from the right-eye five-dimensional tensor using a convolutional long short-term memory network model.

[0144] The extracted spatiotemporal features are output as a spatiotemporal feature map. Using Conv3D as the decoder architecture, the spatiotemporal feature map is input to the decoder. The decoder decodes the spatiotemporal feature map to obtain the right-eye predicted video frame, which is then used as the synchronized video frame under the current system timestamp. This solves the problem of dropped frames in the right-eye encoded video frame.

[0145] In some implementations, the first video stream to be played is encoded in HEVC format.

[0146] Specifically, the merged frames can be input into an HEVC encoder to encode each frame of the merged frame. The HEVC encoder can compress each frame of the merged frame according to a specific compression algorithm to reduce the overall size of the merged frame and improve subsequent transmission efficiency, resulting in the first video stream to be played.

[0147] In this embodiment, the left-eye encoded video stream and the right-eye encoded video stream are synchronized and then encoded a second time, converting them into the HEVC format suitable for various client models. This expands the application scope of spatial video and improves its quality and stability.

[0148] In some implementations, the merged screen is a side-by-side format screen, and the first video stream to be played is a side-by-side format video stream.

[0149] Specifically, side-by-side format is a storage format for spatial video. The merged screen is a side-by-side format screen, and the first video stream to be played obtained by encoding the merged screen is also a side-by-side format video stream.

[0150] Please see Figure 2 and Figure 9 In some implementations, client 300 includes multiple head-mounted display devices. Sending a first video stream to be played to client 300 (i.e., S203) includes:

[0151] S2031: The first video stream to be played is sent to multiple head-mounted display devices via parallel forwarding.

[0152] Specifically, the client 300 may include multiple head-mounted display devices, and the server 200 may use a parallel forwarding method to simultaneously send the first video stream to be played to multiple head-mounted display devices. The parallel forwarding method ensures that the first video stream to be played can be transmitted to multiple head-mounted display devices at the same time, thereby meeting the needs of multiple users to watch spatial video in real time and synchronously.

[0153] Please see Figure 2 and Figure 10 This application also provides a spatial video processing method, applied to a client 300, the spatial video processing method including:

[0154] S301: Receive the first video stream to be played sent by the server 200, wherein the first video stream to be played originates from: the video acquisition device 100 simultaneously capturing the same scene from different perspectives using the left eye camera 10 and the right eye camera 20, obtaining a left eye video stream and a right eye video stream; encoding the left eye video stream and the right eye video stream respectively, obtaining a left eye encoded video stream and a right eye encoded video stream; the server 200 receives the left eye encoded video stream and the right eye encoded video stream transmitted in real time by the video acquisition device 100; and performs video processing on the left eye encoded video stream and the right eye encoded video stream to obtain the first video stream to be played.

[0155] S302: Perform video processing on the first video stream to be played to obtain the second video stream to be played in real time.

[0156] In this embodiment, the left-eye and right-eye video streams captured by the video acquisition device 100 are encoded to obtain left-eye encoded video streams and right-eye encoded video streams, respectively. These encoded video streams can then be transmitted in real-time to the server 200 for video processing to obtain a first video stream to be played. This first video stream can be sent to the client 300, where it undergoes further processing to obtain a second video stream to be played. This second video stream can then be played online in real-time on the client 300. Thus, real-time online playback of spatial video is achieved, allowing users to view the latest spatial video content without the need for pre-recording.

[0157] Please see Figure 2 and Figure 11 In some implementations, client 300 includes multiple head-mounted display devices, receiving a first video stream to be played (i.e., S301) sent by server 200, including:

[0158] S3011: Receives the first video stream to be played in parallel through multiple head-mounted display devices.

[0159] Specifically, the client 300 may include multiple head-mounted display devices, and the server 200 may use a parallel forwarding method to simultaneously send the first video stream to be played to multiple head-mounted display devices. Multiple head-mounted display devices can receive the first video stream to be played in parallel. This ensures that the first video stream to be played can be received by multiple head-mounted display devices simultaneously, thereby meeting the needs of multiple users to watch spatial video in real time and synchronously.

[0160] Please see Figure 12 In some embodiments, video processing is performed on the first video stream to be played to obtain a second video stream to be played for real-time playback (i.e., S302), including:

[0161] S3021: Decode and parse the first video stream to be played to obtain a decoded video stream;

[0162] S3022: Perform spatial reconstruction and rendering on the decoded video stream to obtain a second video stream to be played in real time.

[0163] Specifically, after receiving the first video stream to be played, the client 300 can run a spatial video calculation algorithm. Based on this algorithm, the first video stream to be played can be decoded and parsed to obtain a decoded video stream. Then, based on the depth information and geometric relationships of each frame in the decoded video stream, spatial reconstruction and rendering are performed to obtain the second video stream to be played.

[0164] The specific process of spatial reconstruction and rendering is as follows: Each frame in the decoded video stream is a left-right format image. Stereo correction is performed on these images to ensure that the left and right regions are on the same horizontal line. Depth information is extracted from the left and right region images using a stereo matching algorithm, such as semi-global block matching (SGBM). Based on the depth image, a point cloud is generated, which is then used to construct a 3D model. After obtaining the 3D model, the corresponding frames from the decoded video stream are mapped onto the 3D model as textures.

[0165] The second video stream to be played can be played in real time on client 300. In this way, users can get an immersive spatial video playback experience on client 300, feel realistic three-dimensional effects, and enhance the immersion and realism of the viewing experience.

[0166] Please see Figure 13 This application also provides a spatial video processing device 400, applied to a video acquisition device 100. The video acquisition device 100 includes a left-eye camera 10 and a right-eye camera 20. The spatial video processing device 400 includes a shooting module 410, an encoding module 420, and a transmission module 430. The shooting module 410 is used to simultaneously capture different perspectives of the same scene using the left-eye camera 10 and the right-eye camera 20, obtaining a left-eye video stream and a right-eye video stream. The encoding module 420 is used to encode the left-eye video stream and the right-eye video stream respectively, obtaining a left-eye encoded video stream and a right-eye encoded video stream. The transmission module 430 is used to transmit the left-eye encoded video stream and the right-eye encoded video stream to a server 200 in real time. The server 200 is used to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played; the first video stream to be played is then sent to a client 300; the client 300 is used to perform video processing on the first video stream to be played to obtain a second video stream to be played, for real-time playback.

[0167] In some implementations, the encoding formats of the left-eye encoded video stream and the right-eye encoded video stream are AVC or HEVC.

[0168] In some implementations, the left-eye encoded video stream and the right-eye encoded video stream are transmitted in real time via RTMP.

[0169] It should be noted that the explanation of the spatial video processing method applied to the video acquisition device 100 in the foregoing embodiments also applies to the spatial video processing device 400 in the embodiments of this application, and will not be elaborated here.

[0170] Please see Figure 14This application also provides a spatial video processing device 500, applied to a server 200. The spatial video processing device 500 includes a first receiving module 510, a first processing module 520, and a sending module 530. The first receiving module 510 is used to receive left-eye encoded video streams and right-eye encoded video streams transmitted in real time by a video acquisition device 100. The left-eye encoded video streams and right-eye encoded video streams originate from: the video acquisition device 100 simultaneously capturing different perspectives of the same scene through a left-eye camera 10 and a right-eye camera 20, obtaining left-eye video streams and right-eye video streams; and encoding the left-eye video streams and right-eye video streams respectively. The first processing module 520 is used to perform video processing on the left-eye encoded video streams and right-eye encoded video streams to obtain a first video stream to be played. The sending module 530 is used to send the first video stream to be played to a client 300, and the client 300 is used to perform video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0171] In some implementations, the first processing module 520 is specifically used to time-synchronize and merge the left-view encoded video stream and the right-view encoded video stream to obtain a merged video stream; to perform split decoding on the merged video stream to obtain a left-view video stream and a right-view video stream; to render the corresponding video frames in the left-view video stream and the right-view video stream to the corresponding display areas of the output screen respectively, and merge them into a merged screen; and to encode each frame of the merged screen to obtain a first video stream to be played.

[0172] In some implementations, the first processing module 520 is specifically used to obtain the video frame rate of the monocular encoded video stream, which is either a left-eye encoded video stream or a right-eye encoded video stream; obtain the system timestamp of the server 200; obtain the timestamp of the video frame in the monocular encoded video stream that is closest to the system timestamp, and the corresponding monocular encoded video frame; determine whether the monocular encoded video stream has dropped frames at the system timestamp based on the system timestamp, the video frame timestamp, and the video frame rate; when the monocular encoded video stream has dropped frames, generate monocular predicted video frames based on a convolutional long short-term memory network model, and use them as synchronous video frames under the system timestamp for video stream merging; when the monocular encoded video stream has not dropped frames, use the monocular encoded video frames as synchronous video frames under the system timestamp for video stream merging.

[0173] In some implementations, the first processing module 520 is specifically used to acquire multiple monocular coded video frames prior to the video frame timestamp; preprocess the multiple monocular coded video frames to obtain a five-dimensional tensor, wherein the preprocessing includes normalization and format conversion; extract spatiotemporal features from the five-dimensional tensor based on a convolutional long short-term memory network model; and generate monocular predicted video frames based on the extracted spatiotemporal features, which serve as synchronized video frames under the system timestamp.

[0174] In some implementations, the merged screen is a side-by-side format screen, and the first video stream to be played is a side-by-side format video stream.

[0175] In some implementations, the first video stream to be played is encoded in HEVC format.

[0176] In some implementations, the client 300 includes multiple head-mounted display devices, and the sending module 530 is specifically used to send the first video stream to be played to the multiple head-mounted display devices in a parallel forwarding manner.

[0177] It should be noted that the explanation of the spatial video processing method applied to the server 200 in the foregoing embodiments also applies to the spatial video processing apparatus 500 of the embodiments of this application, and will not be elaborated here.

[0178] Please see Figure 15 This application also provides a spatial video processing device 600, applied to a client 300. The spatial video processing device 600 includes a second receiving module 610 and a second processing module 620. The second receiving module 610 is used to receive a first video stream to be played sent by a server 200. The first video stream to be played originates from: a video acquisition device 100 simultaneously capturing different perspectives of the same scene using a left-eye camera 10 and a right-eye camera 20, obtaining a left-eye video stream and a right-eye video stream; encoding the left-eye video stream and the right-eye video stream respectively, obtaining a left-eye encoded video stream and a right-eye encoded video stream; the server 200 receiving the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device 100; and performing video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played. The second processing module 620 is used to perform video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0179] In some implementations, client 300 includes multiple head-mounted display devices, and second receiving module 610 is specifically configured to receive a first video stream to be played in parallel through the multiple head-mounted display devices.

[0180] In some implementations, the second processing module 620 is specifically used to decode and parse the first video stream to be played to obtain a decoded video stream; and to perform spatial reconstruction and rendering on the decoded video stream to obtain a second video stream to be played for real-time playback.

[0181] It should be noted that the explanation of the spatial video processing method applied to the client 300 in the foregoing embodiments also applies to the spatial video processing apparatus 600 in the embodiments of this application, and will not be elaborated here.

[0182] Please see Figure 16 This application also provides a spatial video processing system 700. The spatial video processing system 700 includes one or more processors 710 and a memory 720. The memory 720 stores a computer program, which, when executed by the processor 710, implements the spatial video processing method of any of the above embodiments.

[0183] For example, when the computer program is executed by the processor 710, the following spatial video processing method is implemented:

[0184] S101: Simultaneously capture the same scene from different perspectives using the left eye camera 10 and the right eye camera 20 to obtain the left eye video stream and the right eye video stream;

[0185] S102: Encode the left-eye video stream and the right-eye video stream separately to obtain the left-eye encoded video stream and the right-eye encoded video stream;

[0186] S103: The left-eye encoded video stream and the right-eye encoded video stream are transmitted to the server 200 in real time. The server 200 performs video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played. The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0187] For example, when the computer program is executed by the processor 710, the following spatial video processing method is implemented:

[0188] S201: Receive the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device 100. The left-eye encoded video stream and the right-eye encoded video stream are obtained by the video acquisition device 100 simultaneously capturing the same scene from different perspectives through the left-eye camera 10 and the right-eye camera 20 to obtain the left-eye video stream and the right-eye video stream; encode the left-eye video stream and the right-eye video stream respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream.

[0189] S202: Perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played;

[0190] S203: The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain the second video stream to be played for real-time playback.

[0191] It should be noted that the explanations of the spatial video processing method and spatial video processing device 400 in the foregoing embodiments also apply to the spatial video processing system 700 of the embodiments of this application, and will not be elaborated here.

[0192] Please see Figure 17 This application also provides a computer-readable storage medium 800 storing a computer program 810, which, when executed by a processor 820, implements the spatial video processing method of any of the above embodiments.

[0193] For example, when the program is executed by processor 820, the following spatial video processing method is implemented:

[0194] S101: Simultaneously capture the same scene from different perspectives using the left eye camera 10 and the right eye camera 20 to obtain the left eye video stream and the right eye video stream;

[0195] S102: Encode the left-eye video stream and the right-eye video stream separately to obtain the left-eye encoded video stream and the right-eye encoded video stream;

[0196] S103: The left-eye encoded video stream and the right-eye encoded video stream are transmitted to the server 200 in real time. The server 200 performs video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first video stream to be played. The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain a second video stream to be played for real-time playback.

[0197] For example, when the program is executed by processor 820, the following spatial video processing method is implemented:

[0198] S201: Receive the left-eye encoded video stream and the right-eye encoded video stream transmitted in real time by the video acquisition device 100. The left-eye encoded video stream and the right-eye encoded video stream are obtained by the video acquisition device 100 simultaneously capturing the same scene from different perspectives through the left-eye camera 10 and the right-eye camera 20 to obtain the left-eye video stream and the right-eye video stream; encode the left-eye video stream and the right-eye video stream respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream.

[0199] S202: Perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain the first video stream to be played;

[0200] S203: The first video stream to be played is sent to the client 300. The client 300 performs video processing on the first video stream to be played to obtain the second video stream to be played for real-time playback.

[0201] It should be noted that the explanations and descriptions of the spatial video processing method and spatial video processing apparatus 400 in the foregoing embodiments also apply to the computer-readable storage medium 800 of the embodiments of this application, and will not be elaborated here.

[0202] In summary, in the spatial video processing method, processing apparatus 400, processing system 700, and computer-readable storage medium 800 of this application, the left-eye video stream and right-eye video stream captured by the video acquisition device 100 are encoded to obtain a left-eye encoded video stream and a right-eye encoded video stream, which can be transmitted in real time to the server 200 for video processing to obtain a first video stream to be played. The first video stream to be played can be sent to the client 300, and after video processing, a second video stream to be played can be obtained, which can be played online in real time on the client 300. In this way, real-time online playback of spatial video is realized, and the latest spatial video content can be viewed without the need for pre-recording of spatial video.

[0203] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0204] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0205] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, a computer-readable storage medium can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0206] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0207] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0208] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method of spatial video processing, the method comprising: The spatial video processing method is applied to a video collection device, and the video collection device comprises a left-eye camera and a right-eye camera. Different angles of the same scene are simultaneously photographed by the left-eye camera and the right-eye camera to obtain a left-eye video stream and a right-eye video stream. The left-eye video stream and the right-eye video stream are respectively encoded to obtain a left-eye encoded video stream and a right-eye encoded video stream. The left-eye encoded video stream and the right-eye encoded video stream are transmitted in real time to a server, the server is configured to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first to-be-played video stream, and the first to-be-played video stream is sent to a client; the client is configured to perform video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

2. The spatial video processing method of claim 1, wherein, The encoding format of the left-eye encoded video stream and the right-eye encoded video stream is AVC format or HEVC format.

3. The spatial video processing method of claim 1, wherein, The left-eye encoded video stream and the right-eye encoded video stream are transmitted in real time by RTMP.

4. A method of spatial video processing, the method comprising: The spatial video processing method is applied to a server, and the spatial video processing method comprises the following steps. A left-eye encoded video stream and a right-eye encoded video stream transmitted in real time by a video collection device are received, wherein the left-eye encoded video stream and the right-eye encoded video stream are derived from the following steps: the video collection device simultaneously photographs different angles of the same scene by a left-eye camera and a right-eye camera to obtain a left-eye video stream and a right-eye video stream; and the left-eye video stream and the right-eye video stream are respectively encoded to obtain the left-eye encoded video stream and the right-eye encoded video stream. The left-eye encoded video stream and the right-eye encoded video stream are processed to obtain a first to-be-played video stream. The first to-be-played video stream is sent to a client, and the client is configured to perform video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

5. The spatial video processing method of claim 4, wherein, The left-eye encoded video stream and the right-eye encoded video stream are processed to obtain a first to-be-played video stream, which comprises the following steps. The left-eye encoded video stream and the right-eye encoded video stream are time-synchronized and merged to obtain a merged video stream. The merged video stream is split decoded to obtain a left-view video stream and a right-view video stream. Corresponding video frames in the left-view video stream and the right-view video stream are respectively rendered to corresponding display regions of an output picture and are merged into a merged picture. Each frame of the merged picture is encoded to obtain the first to-be-played video stream.

6. The spatial video processing method of claim 5, wherein, The left-eye encoded video stream and the right-eye encoded video stream are time-synchronized and merged to obtain a merged video stream, which comprises the following steps. A video frame rate of a monocular encoded video stream is obtained, the monocular encoded video stream being the left-eye encoded video stream or the right-eye encoded video stream. A system timestamp of the server is obtained. A video frame timestamp of the monocular encoded video stream that is closest to the system timestamp and a corresponding monocular encoded video frame are obtained. Whether the monocular encoded video stream drops a frame at the system timestamp is determined according to the system timestamp, the video frame timestamp and the video frame rate. When the monocular coded video stream is dropped, a monocular predicted video frame is generated based on a convolutional long short-term memory network model as a synchronized video frame at the system timestamp for video stream merging; When the monocular coded video stream is not dropped, the monocular coded video frame is taken as a synchronized video frame at the system timestamp for video stream merging.

7. The spatial video processing method of claim 6, wherein, The generation of the monocular predicted video frame based on the convolutional long short-term memory network model as the synchronized video frame at the system timestamp comprises: acquiring a plurality of monocular coded video frames before the video frame timestamp; preprocessing the plurality of monocular coded video frames to obtain a five-dimensional tensor, wherein the preprocessing comprises normalization processing and format conversion; spatiotemporal feature extraction of the five-dimensional tensor based on the convolutional long short-term memory network model; generating the monocular predicted video frame as the synchronized video frame at the system timestamp according to the extracted spatiotemporal features.

8. The spatial video processing method of claim 5, wherein, The merged picture is a left-right format picture, and the first to-be-played video stream is a left-right format video stream.

9. The spatial video processing method of claim 5, wherein, The coding format of the first to-be-played video stream is HEVC format.

10. The spatial video processing method of claim 4, wherein, The client comprises a plurality of head-mounted display devices, and the sending of the first to-be-played video stream to the client comprises: sending the first to-be-played video stream to the plurality of head-mounted display devices in a parallel forwarding manner.

11. A method of spatial video processing, the method comprising: The spatial video processing method is applied to a client, and the method comprises: receiving a first to-be-played video stream sent by a server, wherein the first to-be-played video stream is derived from: a video acquisition device simultaneously capturing different perspectives of the same scene through a left-eye camera and a right-eye camera to obtain a left-eye video stream and a right-eye video stream; encoding the left-eye video stream and the right-eye video stream respectively to obtain a left-eye coded video stream and a right-eye coded video stream; the server receiving the left-eye coded video stream and the right-eye coded video stream transmitted by the video acquisition device in real time; and performing video processing on the left-eye coded video stream and the right-eye coded video stream to obtain the first to-be-played video stream; performing video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

12. The spatial video processing method of claim 11, wherein, The client comprises a plurality of head-mounted display devices, and the receiving of the first to-be-played video stream sent by the server comprises: parallel receiving of the first to-be-played video stream by the plurality of head-mounted display devices.

13. The spatial video processing method of claim 11, wherein, The video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing comprises: decoding and parsing the first to-be-played video stream to obtain a decoded video stream; spatial reconstruction and rendering of the decoded video stream to obtain the second to-be-played video stream for real-time playing.

14. A spatial video processing apparatus, characterized by comprising: The video acquisition device comprises a left-eye camera and a right-eye camera, and the spatial video processing apparatus comprises: a shooting module configured to simultaneously capture different perspectives of the same scene through the left-eye camera and the right-eye camera to obtain a left-eye video stream and a right-eye video stream; The encoding module is configured to encode the left-eye video stream and the right-eye video stream respectively to obtain a left-eye encoded video stream and a right-eye encoded video stream. The transmission module is configured to transmit the left-eye encoded video stream and the right-eye encoded video stream to a server in real time, where the server is configured to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first to-be-played video stream, and transmit the first to-be-played video stream to a client; and the client is configured to perform video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

15. A spatial video processing apparatus, characterized by: The spatial video processing apparatus applied to a server comprises: The first receiving module is configured to receive a left-eye encoded video stream and a right-eye encoded video stream transmitted by a video acquisition device in real time, where the left-eye encoded video stream and the right-eye encoded video stream are derived from: the video acquisition device simultaneously captures different perspectives of the same scene by using a left-eye camera and a right-eye camera to obtain a left-eye video stream and a right-eye video stream; and the left-eye video stream and the right-eye video stream are encoded respectively to obtain the left-eye encoded video stream and the right-eye encoded video stream; The first processing module is configured to perform video processing on the left-eye encoded video stream and the right-eye encoded video stream to obtain a first to-be-played video stream; The sending module is configured to transmit the first to-be-played video stream to a client, and the client is configured to perform video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

16. A spatial video processing apparatus, characterized by comprising: The spatial video processing apparatus applied to a client comprises: The second receiving module is configured to receive a first to-be-played video stream transmitted by a server, where the first to-be-played video stream is derived from: a video acquisition device simultaneously captures different perspectives of the same scene by using a left-eye camera and a right-eye camera to obtain a left-eye video stream and a right-eye video stream; the left-eye video stream and the right-eye video stream are encoded respectively to obtain a left-eye encoded video stream and a right-eye encoded video stream; the server receives the left-eye encoded video stream and the right-eye encoded video stream transmitted by the video acquisition device in real time; and the left-eye encoded video stream and the right-eye encoded video stream are processed to obtain the first to-be-played video stream; The second processing module is configured to perform video processing on the first to-be-played video stream to obtain a second to-be-played video stream for real-time playing.

17. A spatial video processing system, characterized by The spatial video processing system comprises one or more processors and a memory, and the memory stores a computer program; when the computer program is executed by the processor, the spatial video processing method in any one of claims 1-13 is implemented.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor, and the spatial video processing method in any one of claims 1-13 is implemented.

Images