Video processing, control, playback method, device, and medium

Abstract

The present disclosure provides a video processing method, including: receiving multiple video code streams sent by multiple cameras, the multiple cameras have different shooting angles, so that the multiple video code streams correspond to multiple display angles; encoding the multiple video code streams to generate multiple synthetic code streams, each synthetic code stream is generated by encoding at least one video code stream. The present disclosure also provides a video control method, a video playback method, an encoder, a streaming media server, a terminal, and a computer-readable medium.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a video processing method, a video control method, a video playing method, an encoder, a streaming media server, a terminal, and a computer-readable medium. Background Art

[0002] The raw data of free viewpoint video is generally collected by a collection of multiple cameras deployed around the venue, and multiple cameras simultaneously shoot different angles of the same scene. The data is transmitted to the server through the network for processing and distribution, and finally presented through the intelligent terminal. Users can choose any one of them to watch, or synthesize a virtual intermediate scene based on the video sequence shot by several adjacent cameras for viewing, and realize seamless free browsing and switching between multiple viewpoints. Free viewpoint is generally used in various sports competitions, exhibitions, stadium events and other activities.

[0003] However, free video often cannot achieve both stereoscopic and smooth free viewpoint switching effects and clear display effects. Summary of the invention

[0004] The embodiments of the present disclosure provide a video processing method, a video control method, a video playing method, an encoder, a streaming media server, a terminal, and a computer-readable medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Receiving multiple video streams sent by multiple cameras, where the multiple cameras have different shooting angles, so that the multiple video streams correspond to multiple display angles;

[0007] The multiple video code streams are encoded to generate multiple composite code streams, each of which is generated by encoding at least one video code stream.

[0008] In some embodiments, before the step of encoding the multiple video code streams to generate multiple composite code streams, the video processing method further includes:

[0009] Determine the number of the video code streams for synthesizing each of the synthesized code streams.

[0010] In some embodiments, before the step of encoding the multiple video code streams to generate multiple composite code streams, the video processing method further includes:

[0011] The video stream synthesized into each synthesized stream is determined according to the adjacent relationship of display viewing angles corresponding to the multiple video streams, wherein the display viewing angles corresponding to the multiple video streams synthesized into the same synthesized stream are sequentially adjacent.

[0012] In some embodiments, after encoding multiple video code streams to generate multiple composite code streams, the encoding method further includes:

[0013] Time synchronization processing is performed on the multiple channels of synthesized code streams.

[0014] In some embodiments, after the step of encoding the multiple video code streams to generate multiple composite code streams, the video processing method further includes:

[0015] The multiple channels of synthesized code streams are synchronized to at least one streaming media server.

[0016] In a second aspect, an embodiment of the present disclosure provides a video control method, including:

[0017] Generate an address for each of the multiple composite code streams, wherein the multiple composite code streams are generated by encoding multiple video code streams, and each composite code stream is generated by encoding at least one video code stream;

[0018] In response to the play request, the corresponding synthesized code stream is transmitted to the terminal that sends the play request according to the address carried in the play request.

[0019] In some embodiments, after the step of generating an address for each of the multiple synthesized code streams, the video control method further includes:

[0020] Determine the association relationship between the addresses of the multiple synthesized code streams.

[0021] In a third aspect, an embodiment of the present disclosure provides a video playback method, including:

[0022] Decoding at least one of the multiple synthesized code streams to obtain at least one synthesized image matrix to be played, wherein each synthesized code stream can be decoded to obtain a synthesized image matrix, each synthesized image matrix is ​​divided into at least one sub-grid, the multiple synthesized code streams are generated by encoding multiple video code streams, each synthesized code stream is generated by encoding at least one video code stream, the multiple video code streams correspond to multiple display viewing angles, and each sub-grid corresponds to a display viewing angle corresponding to one video code stream;

[0023] Rendering one of the at least one sub-grids to which the at least one matrix of synthesized images to be played belongs to generate a playing picture.

[0024] In some embodiments, the video playback method further includes:

[0025] In response to an instruction to switch to a target subgrid, the target subgrid is rendered and displayed, wherein the synthetic image matrix to which the target subgrid belongs is one of the multiple synthetic image matrices that can be obtained after decoding multiple synthetic code streams.

[0026] In some embodiments, in response to the instruction to switch to the target sub-grid, before the step of rendering and displaying the target sub-grid, the decoding method further includes:

[0027] Determine whether the synthetic image matrix to which the target subgrid belongs is one of the at least one synthetic image matrix to be played;

[0028] When the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played, performing a step of rendering and displaying the target sub-grid;

[0029] When the synthetic image matrix to which the target subgrid belongs is not one of the at least one synthetic image matrix to be played, before the step of rendering and displaying the target subgrid, the video playing method further includes:

[0030] The synthesized code stream that can be decoded to obtain the synthesized image matrix to which the target sub-grid belongs is decoded to obtain the synthesized image matrix to which the target sub-grid belongs.

[0031] In some embodiments, the instruction to switch to the target sub-grid is an instruction generated according to the moving trajectory of the touch point on the playback screen. In response to the instruction to switch to the target sub-grid, before the step of rendering and displaying the target sub-grid, the video playback method further includes:

[0032] At least one target sub-grid is determined according to the moving direction of the touch point in the moving track.

[0033] In some embodiments, the display viewing angles corresponding to the multiple video code streams synthesized into the same synthesized code stream are adjacent in sequence, and the step of determining at least one target sub-grid according to the moving direction of the touch point in the moving trajectory includes:

[0034] According to the moving direction of the touch point in the moving track, a sub-grid corresponding to a display viewing angle adjacent to the display viewing angle of the playback screen is determined as the target sub-grid.

[0035] In some embodiments, the video playback method further includes:

[0036] Pre-decode at least one target composite code stream, wherein the target composite code stream is a composite code stream other than a composite code stream obtained by decoding the composite image matrix to which the target sub-grid belongs, among the multiple composite code streams.

[0037] In some embodiments, the instruction to switch to the target sub-grid is an instruction generated according to the moving trajectory of the touch point on the playback screen, and before the step of pre-decoding at least one target synthesized bitstream, the video playback method further includes:

[0038] At least one target synthesized code stream is determined according to the moving direction of the touch point in the moving track.

[0039] In some embodiments, before the step of determining at least one target synthesized code stream according to the moving direction of the touch point in the moving trajectory, the video playback method further includes:

[0040] The number of the target synthesized code streams to be pre-decoded is determined.

[0041] In some embodiments, before the step of decoding at least one of the multiple synthesized code streams to obtain at least one synthesized image matrix to be played, the video playback method further includes:

[0042] Generate a play request according to the video play instruction, wherein the play request carries the address of the video pointed to by the video play instruction;

[0043] Sending the play request to a streaming media server;

[0044] Receive at least one composite code stream corresponding to the address and sent by the streaming media server.

[0045] In a fourth aspect, an embodiment of the present disclosure provides a decoder, including:

[0046] one or more processors;

[0047] A storage device stores one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method described in the first aspect of the embodiment of the present disclosure.

[0048] In a fifth aspect, an embodiment of the present disclosure provides a streaming media server, including:

[0049] one or more processors;

[0050] A storage device stores one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the video control method described in the second aspect of the embodiment of the present disclosure.

[0051] In a sixth aspect, an embodiment of the present disclosure provides a terminal, including:

[0052] one or more processors;

[0053] A storage device stores one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the video playback method described in the third aspect of the embodiment of the present disclosure.

[0054] In a seventh aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, at least one of the following methods is implemented:

[0055] The video processing method according to the first aspect of the embodiment of the present disclosure;

[0056] The video control method according to the second aspect of the embodiment of the present disclosure;

[0057] The video playback method described in the third aspect of the embodiment of the present disclosure.

[0058] In the embodiment of the present disclosure, the encoder encodes multiple video streams generated by multiple cameras to generate multiple composite streams, wherein each composite stream is generated by encoding a limited number of video streams. When the multiple composite streams are pushed to the terminal for playback through the streaming media server, the number of grids cut out by the terminal by decoding one composite stream is also limited, so the display clarity can be guaranteed; the terminal decodes the multiple composite streams generated by the video processing method, and can obtain more video streams corresponding to different viewing angles, thereby achieving a more three-dimensional, larger viewing angle, and smoother switching free viewpoint effect; at the same time, the number of composite streams decoded simultaneously can be flexibly adjusted according to the terminal performance, thereby ensuring the compatibility of the video playback method described in the embodiment of the present disclosure with different terminals. In addition, in the embodiment of the present disclosure, multiple composite streams are generated simultaneously by multiple encoding services or by deploying multiple encoders concurrently, and the encoder only needs to synthesize a limited number of video streams into one composite stream at a time, which reduces the requirements for the encoder performance and ensures the video encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a video processing method in an embodiment of the present disclosure;

[0060] Figure 2 is a flowchart of some steps in another video processing method in an embodiment of the present disclosure;

[0061] Figure 3 is a flowchart of some steps in another video processing method in an embodiment of the present disclosure;

[0062] Figure 4 is a flowchart of some steps in another video processing method in an embodiment of the present disclosure;

[0063] Figure 5 is a flow chart of a video control method in an embodiment of the present disclosure;

[0064] Figure 6 is a flowchart of some steps in another video control method in an embodiment of the present disclosure;

[0065] Figure 7 is a flow chart of a video playback method in an embodiment of the present disclosure;

[0066] Figure 8 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0067] Fig. 9 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0068] Fig.10 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0069] Fig.11 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0070] Fig.12 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0071] Fig.13 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0072] Fig.14 is a flowchart of some steps in another video playback method in an embodiment of the present disclosure;

[0073] Fig.15 is a block diagram of a composition of an encoder in an embodiment of the present disclosure;

[0074] Fig.16 is a block diagram of a streaming media server in an embodiment of the present disclosure;

[0075] Fig.17 is a block diagram of a terminal in an embodiment of the present disclosure;

[0076] Fig.18 is a block diagram of a computer-readable medium in an embodiment of the present disclosure;

[0077] Fig.19is a schematic diagram of an implementation method of free viewpoint playback in an embodiment of the present disclosure;

[0078] Fig. 20 It is a schematic diagram of encoding and decoding in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] In order to enable those skilled in the art to better understand the technical solution of the present invention, the video processing method, video control method, video playback method, encoder, streaming media server, terminal, and computer-readable medium provided by the present invention are described in detail below in conjunction with the accompanying drawings.

[0080] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0081] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0082] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0083] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0084] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0085] The inventors of the disclosed embodiments have found that in order to achieve free viewpoint playback, the current approach is to synthesize multiple video streams captured by multiple cameras into one synthesized stream, and the playback terminal only needs to decode the one synthesized stream and render one of the decoded multiple video streams for playback. In order to present a more three-dimensional, larger viewing angle, and smoother switching free viewpoint effect, it is usually necessary to deploy more densely populated cameras. This will bring about the following problems: denser cameras will capture and generate more video streams, but the number of video streams that the encoder can process is limited. When the number of video streams is too large, multiple video streams cannot be synthesized into one synthesized stream; when the number of video streams synthesized into one synthesized stream is too large, the terminal player will cut out more grids through decoding, where one grid corresponds to one video stream, and the clarity will be greatly reduced when any of the grids is played, affecting the user experience.

[0086] In view of this, firstly, refer to Figure 1 , the present disclosure provides a video processing method, including:

[0087] In step S110, multiple video streams sent by multiple cameras are received, and the multiple cameras have different shooting angles, so that the multiple video streams correspond to multiple display angles;

[0088] In step S120, the multiple video code streams are encoded to generate multiple composite code streams, each of which is generated by encoding at least one video code stream.

[0089] In the embodiment of the present disclosure, each video stream corresponds to a camera, and different video streams correspond to different viewing angles of different cameras. The embodiment of the present disclosure does not impose a special upper limit on the number of video streams generated by the camera in step S110.

[0090] It should be noted that, in the disclosed embodiment, each composite bitstream is generated by encoding a limited number of video bitstreams based on the same strategy, and the video bitstreams generated by encoding different composite bitstreams are different from each other. It should also be noted that multiple composite bitstreams will form an association relationship according to the corresponding video bitstreams, for example, the display viewing angles corresponding to the multiple video bitstreams synthesized into the same composite bitstream are adjacent in sequence.

[0091] The embodiments of the present disclosure do not impose any special restrictions on the number of video bitstreams that can be encoded to generate a composite bitstream. For example, the number of video bitstreams that can be synthesized into one composite bitstream can be determined based on the performance of the encoder; or the number of video bitstreams that can be synthesized into one composite bitstream can be determined based on the display clarity requirements of the terminal during playback.

[0092] In the embodiments of the present disclosure, multiple encoding services can be concurrently used in the same encoder for composite encoding, and each encoding service generates at least one composite bitstream; multiple encoders can also be deployed, and multiple encoders can concurrently perform composite encoding, and each encoder generates at least one composite bitstream, wherein each encoder can concurrently use multiple encoding services, and each encoding service generates at least one composite bitstream. The embodiments of the present disclosure do not specifically limit this.

[0093] The disclosed embodiment does not specifically limit the specific manner of encoding at least one video bitstream to generate a composite bitstream in step S120. As an optional implementation, the video screen is represented by an image matrix, and the encoded composite bitstream can be decoded to obtain a composite image matrix. The disclosed embodiment does not specifically limit the specific form of the composite image matrix. As an optional implementation, after the composite image matrix is ​​rendered and displayed, the video screen is spliced ​​by video screens shot by a limited number of cameras.

[0094] In the video processing method provided in the embodiment of the present disclosure, multiple video streams generated by multiple cameras are encoded to generate multiple composite streams, wherein each composite stream is generated by encoding a limited number of video streams. When the multiple composite streams are pushed to the terminal for playback through the streaming media server, the number of grids cut out by the terminal by decoding one composite stream is also limited, so the display clarity can be guaranteed; at the same time, the terminal decodes the multiple composite streams generated by the video processing method, and can obtain more video streams corresponding to different display viewing angles, thereby achieving a more three-dimensional, larger viewing angle, and smoother switching free viewpoint effect. In addition, in the embodiment of the present disclosure, multiple composite streams are generated simultaneously by multiple encoding services concurrently or multiple encoders are deployed concurrently, and the encoder only needs to synthesize a limited number of video streams into one composite stream at a time, which reduces the requirements for the encoder performance and ensures the video encoding efficiency.

[0095] In some embodiments, reference Figure 2 Before step S120, the video processing method further includes:

[0096] In step S141, the number of the video code streams synthesized into each synthesized code stream is determined.

[0097] The embodiments of the present disclosure do not impose any special restrictions on the number of video bitstreams that can be encoded to generate a composite bitstream. For example, the number of video bitstreams that can be synthesized into one composite bitstream can be determined based on the performance of the encoder; or the number of video bitstreams that can be synthesized into one composite bitstream can be determined based on the display clarity requirements of the terminal during playback.

[0098] In the disclosed embodiment, the camera sending the video code stream corresponds to different shooting angles, that is, different video code streams correspond to different display angles, and the multiple display angles have a sequential adjacent relationship. As an optional implementation, according to the sequential adjacent relationship of the multiple display angles, the multiple video code streams are synthesized into one synthesized code stream.

[0099] Accordingly, in some embodiments, reference Figure 2 Before step S120, the video processing method further includes:

[0100] In step S142, the video stream synthesized into each synthesized stream is determined according to the adjacent relationship of the display viewing angles corresponding to the multiple video streams, wherein the display viewing angles corresponding to the multiple video streams synthesized into the same synthesized stream are sequentially adjacent.

[0101] In the embodiment of the present disclosure, after multiple encoding services are concurrently synthesized in the same encoder, or multiple encoders are concurrently synthesized to generate multiple synthesized code streams, time synchronization processing needs to be performed on the multiple synthesized code streams.

[0102] Accordingly, refer to Figure 3 In some embodiments, after step S120, the encoding method further includes:

[0103] In step S130, time synchronization processing is performed on the multiple channels of synthesized code streams.

[0104] In the disclosed embodiment, after the encoder generates multiple composite streams through encoding in steps S110 to S120, the multiple composite streams are pushed to the streaming media servers of multiple streaming media nodes, and the composite streams are sent to the terminal through the streaming media servers.

[0105] Accordingly, refer to Figure 4 In some embodiments, after step S120, the video processing method further includes:

[0106] In step S150, the multiple channels of synthesized code streams are synchronized to at least one streaming media server.

[0107] Second, refer to Figure 5 , the present disclosure provides a video control method, including:

[0108] In step S210, an address is generated for each of the multiple composite code streams, wherein the multiple composite code streams are generated by encoding multiple video code streams, and each composite code stream is generated by encoding at least one video code stream;

[0109] In step S220, in response to the play request, the corresponding synthesized code stream is transmitted to the terminal that sends the play request according to the address carried in the play request.

[0110] In the disclosed embodiment, each video bitstream generated by encoding a composite bitstream corresponds to a camera, and different cameras correspond to different shooting angles. Each composite bitstream is generated by encoding a limited number of video bitstreams based on the same strategy, and the video bitstreams generated by encoding different composite bitstreams are different from each other. Multiple composite bitstreams will form an association relationship according to the corresponding video bitstreams, for example, according to the display angle corresponding to the video bitstream, the multiple composite bitstreams form a fixed order relationship.

[0111] In the embodiment of the present disclosure, after the streaming media server receives the multiple synthesized code streams sent by the encoder, it generates a uniform resource locator (URL) for each synthesized code stream according to the association relationship of the multiple synthesized code streams through step S210, and sends the synthesized code stream to the terminal through step S220, so that the terminal can obtain the multiple synthesized code streams from the streaming media server according to the URL, thereby realizing a free viewpoint effect.

[0112] In the video control method provided by the embodiment of the present disclosure, the streaming media server can generate addresses for multiple synthetic code streams with associated relationships that are synthesized by encoding multiple video code streams, and can send synthetic code streams with associated relationships to the terminal in response to the terminal's playback request. Since each synthetic code stream is generated by encoding a limited number of video code streams, display clarity can be guaranteed; the terminal decodes the multiple synthetic code streams obtained from the streaming media server, and can obtain more video code streams corresponding to different display viewing angles, thereby achieving a more three-dimensional, larger viewing angle, and smoother switching free viewpoint effect.

[0113] In some embodiments, reference Figure 6 After step S210, the video control method further includes:

[0114] In step S230, the association relationship between the addresses of the multiple synthesized code streams is determined.

[0115] It should be noted that, in the disclosed embodiment, the association relationship of the addresses of the multiple synthesized code streams corresponds to the association relationship of the multiple synthesized code streams. For example, the association relationship between the multiple synthesized code streams is formed according to the association relationship of the display viewing angles corresponding to the video code streams, the addresses of the multiple synthesized code streams can reflect the association relationship between the multiple synthesized code streams, and the video code stream corresponding to a certain display viewing angle can be determined according to the addresses of the multiple synthesized code streams.

[0116] In the embodiment of the present disclosure, a continuous URL may be used as the address of multiple synthesized code streams to reflect the association relationship of the multiple synthesized code streams; or a corresponding identifier may be set to mark the association relationship of the multiple synthesized code streams. The embodiment of the present disclosure does not specifically limit this.

[0117] Thirdly, refer to Figure 7 , the present disclosure provides a video playback method, including:

[0118] In step S310, at least one of the multiple synthesized code streams is decoded to obtain at least one synthesized image matrix to be played, wherein each synthesized code stream can be decoded to obtain a synthesized image matrix, each synthesized image matrix is ​​divided into at least one sub-grid, the multiple synthesized code streams are generated by encoding multiple video code streams, each synthesized code stream is generated by encoding at least one video code stream, the multiple video code streams correspond to multiple display viewing angles, and each sub-grid corresponds to a display viewing angle corresponding to one video code stream;

[0119] In step S320, one of at least one sub-grid to which the at least one matrix of synthesized images to be played belongs is rendered to generate a play screen.

[0120] In the disclosed embodiment, each video code stream corresponds to a camera, and different cameras correspond to different shooting angles.

[0121] In the embodiment of the present disclosure, the video screen is represented by an image matrix, and the encoded synthetic code stream can be decoded to obtain a synthetic image matrix. The embodiment of the present disclosure does not specifically limit the specific form of the synthetic image matrix. As an optional implementation, after the synthetic image matrix is ​​rendered and displayed, the video screen is spliced ​​by video screens shot by a limited number of cameras.

[0122] It should be noted that, in the disclosed embodiment, each composite code stream is generated by encoding a limited number of video code streams based on the same strategy, and the video code streams generated by encoding different composite code streams are different from each other. It should also be noted that multiple composite code streams will form an association relationship according to the corresponding video code streams. For example, the display viewing angles corresponding to the multiple video code streams synthesized into the same composite code stream are adjacent in sequence. Correspondingly, the display viewing angles corresponding to the sub-grids in the composite image matrix that can be obtained after decoding the multiple composite code streams are also adjacent in sequence.

[0123] In the disclosed embodiment, when the encoder encodes and generates multiple composite streams, the number of video streams synthesized into one composite stream is determined according to the display clarity requirement of the terminal during playback. In the disclosed embodiment, decoding the composite stream is the inverse process of the encoder encoding multiple video streams to generate the composite stream. Therefore, the matrix of the composite image to be played obtained by decoding in step S310 can be divided into finite-path sub-grids, and different sub-grids correspond to different display viewing angles; after enlarging any sub-grid, the display clarity requirement of the terminal can still be met.

[0124] In step S320, only one sub-grid in the matrix of the composite image to be played may be rendered, and the playback picture obtained after the sub-grid is rendered may be displayed in full screen on the terminal display screen; or multiple sub-grids in the matrix of the composite image to be played may be rendered simultaneously, and the playback picture obtained after the multiple sub-grids are rendered may be displayed in split screen on the terminal display screen. The embodiments of the present disclosure do not specifically limit this.

[0125] It should also be noted that in the embodiment of the present disclosure, the number of the synthesized code streams decoded simultaneously in step S310 is not specifically limited. As an optional implementation, the number of the synthesized code streams decoded simultaneously can be determined according to the decoding performance of the terminal, thereby improving the compatibility of the video playback method described in the embodiment of the present disclosure with different terminals. For example, if the terminal has a strong decoding performance, more synthesized code streams are decoded simultaneously, and the smoothness of the free viewpoint switching can be ensured; if the terminal has a weak decoding performance, fewer synthesized code streams are decoded simultaneously.

[0126] In the video playback method provided by the embodiment of the present disclosure, the terminal obtains multiple synthetic code streams from the streaming media server, and the multiple synthetic code streams are generated by encoding multiple video code streams generated by multiple cameras by the encoder, wherein each synthetic code stream is generated by encoding a limited number of video code streams. When the terminal plays, the number of sub-grids cut out by the terminal player through decoding one synthetic code stream is also limited, so the display clarity can be guaranteed; the terminal decodes the multiple synthetic code streams generated by the video processing method, and can obtain more sub-grids corresponding to different display viewing angles, thereby achieving a more three-dimensional, larger viewing angle, and smoother switching free viewpoint effect; at the same time, it can also flexibly adjust the number of synthetic code streams decoded simultaneously according to the terminal performance, thereby ensuring the compatibility of the video playback method described in the embodiment of the present disclosure with different terminals.

[0127] In the disclosed embodiment, in response to a user instruction to switch a free viewpoint, a new sub-mesh is selected for rendering and display according to a synthesis strategy.

[0128] Accordingly, refer to Figure 8 In some embodiments, the video playback method further includes:

[0129] In step S330, in response to an instruction to switch to a target sub-grid, the target sub-grid is rendered and displayed, wherein the synthetic image matrix to which the target sub-grid belongs is one of the multiple synthetic image matrices that can be obtained after decoding multiple synthetic code streams.

[0130] In an embodiment of the present disclosure, if the new sub-grid selected in response to the user's instruction to switch the free viewpoint is one of the composite image matrices to be played decoded in step S310, the selected new sub-grid is directly rendered and displayed; if the composite image matrix to be played decoded in step S310 does not include the selected new sub-grid, it is necessary to decode the composite code stream of the composite image matrix to which the new sub-grid belongs, and then render and display the new sub-grid.

[0131] Accordingly, refer to Fig. 9 In some embodiments, in response to the instruction to switch to the target sub-grid, before step S330, the decoding method further includes:

[0132] In step S341, it is determined whether the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played;

[0133] When the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played, performing a step of rendering and displaying the target sub-grid;

[0134] When the synthetic image matrix to which the target sub-grid belongs is not one of the at least one synthetic image matrix to be played, in step S342, the synthetic code stream that can be decoded to obtain the synthetic image matrix to which the target sub-grid belongs is decoded to obtain the synthetic image matrix to which the target sub-grid belongs.

[0135] In the disclosed embodiment, the user switches the viewing angle by sliding left and right on the playback screen. When the terminal receives the instruction of the user sliding the screen, it determines the target viewing angle that the user needs to switch to according to the sliding direction, and then determines the target subgrid corresponding to the target viewing angle. As an optional implementation, the terminal determines the sliding direction of the user on the playback screen by detecting the moving track of the touch point on the playback screen, and further determines the target subgrid.

[0136] Accordingly, refer to Fig.10 In some embodiments, the instruction to switch to the target sub-grid is an instruction generated according to the moving trajectory of the touch point on the playback screen. In response to the instruction to switch to the target sub-grid, before the step of rendering and displaying the target sub-grid, the video playback method further includes:

[0137] In step S350, at least one target sub-grid is determined according to the moving direction of the touch point in the moving trajectory.

[0138] It should be noted that, in the disclosed example, each sub-grid corresponds to a video stream, each video stream corresponds to a camera, and different cameras have different positions, that is, each sub-grid corresponds to a display viewing angle. In step S350, determining at least one target sub-grid according to the moving direction of the touch point in the moving trajectory means determining the target viewing angle according to the moving direction of the touch point in the moving trajectory, and determining the sub-grid corresponding to the target viewing angle as the target sub-grid.

[0139] In the disclosed embodiment, the corresponding relationship between the sliding direction of the user on the playback screen and the switching of the display perspective can be preset. For example, when the user slides to the left on the playback screen, the sub-grid corresponding to the perspective on the left side of the current playback screen is determined as the target sub-grid; when the user slides to the right on the playback screen, the sub-grid corresponding to the perspective on the right side of the current playback screen is determined as the target sub-grid; when the user slides down on the playback screen, the sub-grid corresponding to the perspective on the upper side of the current playback screen is determined as the target sub-grid; when the user slides up on the playback screen, the sub-grid corresponding to the perspective on the lower side of the current playback screen is determined as the target sub-grid.

[0140] As an optional implementation, the subgrid corresponding to the right angle of the current playback screen is used as the next subgrid of the subgrid corresponding to the current playback screen, and the subgrid corresponding to the left angle of the current playback screen is used as the previous subgrid of the subgrid corresponding to the current playback screen. In step S350, when the moving direction of the touch point is from left to right, the next subgrid is used as the target subgrid; when the moving direction of the touch point is from right to left, the previous subgrid is used as the target subgrid.

[0141] As an optional implementation, when the user slides to switch perspectives on the playback screen, the sub-grid corresponding to the perspective that is consistent with the user's sliding direction and adjacent to the perspective of the current playback screen is determined as the target sub-grid, and the sub-grid corresponding to the perspective adjacent to the perspective of the current playback screen is rendered and displayed in real time, thereby improving the smoothness of the perspective switching.

[0142] Accordingly, refer to Fig.11 , step S350 includes:

[0143] In step S351, according to the moving direction of the touch point in the moving track, a sub-grid corresponding to a display viewing angle adjacent to the display viewing angle of the playback screen is determined as the target sub-grid.

[0144] In the disclosed embodiment, the next possible composite code stream to be displayed can be predicted according to the strategy and decoded in advance, thereby shortening the time consumed in decoding and rendering the new composite code stream when switching the free viewpoint, avoiding the black screen phenomenon when switching, and making the free viewpoint switching smoother.

[0145] Accordingly, refer to Fig.12 In some embodiments, the video playback method further includes:

[0146] In step S360, at least one target composite code stream is pre-decoded, where the target composite code stream is a composite code stream other than the composite code stream obtained by decoding the composite image matrix to which the target sub-grid belongs, among the multiple composite code streams.

[0147] Accordingly, refer to Fig.13 In some embodiments, the instruction to switch to the target sub-grid is an instruction generated according to the moving trajectory of the touch point on the playback screen. Before step S350, the video playback method further includes:

[0148] In step S371, at least one target synthesized code stream is determined according to the moving direction of the touch point in the moving trajectory.

[0149] As an optional implementation, the synthesized code stream corresponding to the right angle of the current playback screen is used as the next synthesized code stream of the synthesized code stream corresponding to the current playback screen, and the synthesized code stream corresponding to the left angle of the current playback screen is used as the previous synthesized code stream of the synthesized code stream corresponding to the current playback screen. In step S360, when the moving direction of the touch point is from left to right, the next one or several synthesized code streams are used as the target synthesized code streams; when the moving direction of the touch point is from right to left, the previous one or several synthesized code streams are used as the target synthesized code streams.

[0150] As another optional implementation, in step S360, when the moving direction of the touch point is from left to right, and there is no synthesized code stream on the right side of the decoded synthesized code stream, or the number of synthesized code streams on the right side of the decoded synthesized code stream is less than a predetermined number, the previous one or several synthesized code streams are taken as the target synthesized code stream; when the moving direction of the touch point is from right to left, and there is no synthesized code stream on the left side of the decoded synthesized code stream, or the number of synthesized code streams on the left side of the decoded synthesized code stream is less than a predetermined number, the next one or several synthesized code streams are taken as the target synthesized code stream.

[0151] In the embodiment of the present disclosure, the number of target synthesized code streams pre-decoded in step S360 can be flexibly adjusted to ensure compatibility.

[0152] Accordingly, refer to Fig.13 In some embodiments, before step S360, the video playback method further includes:

[0153] In step S372, the number of the target synthesized code streams to be pre-decoded is determined.

[0154] The embodiment of the present disclosure does not specifically limit how to perform step S372 to determine the number of target synthesized code streams. As an optional implementation, the number of pre-decoded target synthesized code streams can be determined according to the decoding performance of the terminal, thereby improving the compatibility of the video playback method described in the embodiment of the present disclosure with different terminals. For example, if the terminal has strong decoding performance, more target synthesized code streams are pre-decoded, and the smoothness of free viewpoint switching can be ensured; if the terminal has weak decoding performance, fewer target synthesized code streams are pre-decoded.

[0155] In the disclosed embodiment, after receiving the multiple synthesized code streams sent by the encoder, the streaming media server generates a URL address for each synthesized code stream according to the association relationship of the multiple synthesized code streams. The terminal sends a play request to the streaming media server and obtains the multiple synthesized code streams from the streaming media server according to the URL, thereby realizing a free viewpoint effect.

[0156] Accordingly, refer to Fig.14 In some embodiments, before step S310, the video playback method further includes:

[0157] In step S381, a play request is generated according to the video play instruction, and the play request carries the address of the video pointed to by the video play instruction;

[0158] In step S382, the play request is sent to a streaming media server;

[0159] In step S383, at least one composite code stream corresponding to the address and sent by the streaming media server is received.

[0160] Fourthly, refer to Fig.15 , the present disclosure provides a decoder, including:

[0161] One or more processors 101;

[0162] A memory 102 storing one or more programs, which, when executed by one or more processors, enables the one or more processors to implement the video processing method described in the first aspect of the embodiment of the present disclosure;

[0163] One or more I / O interfaces 103 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0164] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU) and the like; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus) and the like.

[0165] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0166] Fifth, refer to Fig.16 The present disclosure provides a streaming media server, including:

[0167] One or more processors 201;

[0168] A memory 202, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement the video control method according to the second aspect of the embodiment of the present disclosure;

[0169] One or more I / O interfaces 203 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0170] Among them, the processor 201 is a device with data processing capabilities, including but not limited to a central processing unit (CPU) and the like; the memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 203 is connected between the processor 201 and the memory 202, and can realize information interaction between the processor 201 and the memory 202, including but not limited to a data bus (Bus) and the like.

[0171] In some embodiments, the processor 201 , the memory 202 , and the I / O interface 203 are connected to each other via a bus 204 , and further connected to other components of the computing device.

[0172] Sixth, refer to Fig.17 , an embodiment of the present disclosure provides a terminal, including:

[0173] One or more processors 301;

[0174] A memory 302, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement the video playback method described in the third aspect of the embodiment of the present disclosure;

[0175] One or more I / O interfaces 303 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0176] Among them, the processor 301 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 302 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 303 is connected between the processor 301 and the memory 302, and can realize information interaction between the processor 301 and the memory 302, including but not limited to a data bus (Bus), etc.

[0177] In some embodiments, the processor 301 , the memory 302 , and the I / O interface 303 are connected to each other via a bus 304 , and further connected to other components of the computing device.

[0178] Seventh, refer to Fig.18 The present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, at least one of the following methods is implemented:

[0179] The video processing method according to the first aspect of the embodiment of the present disclosure;

[0180] The video control method according to the second aspect of the embodiment of the present disclosure;

[0181] The video playback method described in the third aspect of the embodiment of the present disclosure.

[0182] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present disclosure, the technical solutions provided by the embodiments of the present disclosure are described in detail below through specific embodiments:

[0183] Embodiment 1

[0184] Fig.19 This is a schematic diagram of implementing free viewpoint playback in the first embodiment.

[0185] like Fig.19 As shown, in one embodiment of the present invention, multiple video streams are acquired by shooting with multiple cameras such as camera 1, ..., camera N.

[0186] The encoder encodes multiple video code streams to generate multiple composite code streams, and performs time synchronization processing on the multiple composite code streams; then the multiple composite code streams are synchronized to the streaming media server.

[0187] The streaming media server transmits the multi-channel synthesized code streams to the terminal in response to the terminal's playback request.

[0188] The terminal decodes multiple video streams to obtain multiple composite image matrices, each of which is divided into multiple sub-grids, each of which corresponds to a video stream; the terminal renders the sub-grid to be displayed and displays it; the terminal can also update and render a new sub-grid according to the switching instruction, and pre-decode the composite stream according to the viewing angle switching direction; for example, the composite stream corresponding to the right viewing angle of the current playback screen is used as the next composite stream of the composite stream corresponding to the current playback screen, and the composite stream corresponding to the left viewing angle of the current playback screen is used as the previous composite stream of the composite stream corresponding to the current playback screen. When sliding to the left, it is predicted that the decoded composite stream is about to reach the boundary, so the next composite stream is decoded in advance; when sliding to the right, it is predicted that the decoded composite stream is about to reach the boundary, so the previous composite stream is decoded in advance.

[0189] Finally, free viewpoint playback is achieved.

[0190] Embodiment 2

[0191] Fig. 20 This is a schematic diagram of decoding the composite bitstream and switching and rendering the video bitstream in the second embodiment.

[0192] In the second embodiment, in order to ensure a three-dimensional and smooth free viewpoint switching effect, 36 cameras are deployed around the object in a fixed step size.

[0193] The encoder encodes 36 video streams from 36 cameras to generate multiple composite streams. In the second embodiment, 4 video streams are composited into 1 composite stream to ensure the display clarity of the terminal.

[0194] like Fig. 20 As shown, after the composite code stream is decoded, a composite image matrix consisting of 2*2 sub-grids can be obtained, each sub-grid corresponds to one video code stream, and finally 9 composite code streams are obtained.

[0195] In the second embodiment, the free viewpoint playback includes the following process:

[0196] The encoder concurrently uses 9 encoding services to encode and synthesize 36 video streams;

[0197] The encoder performs time synchronization processing on the 9-channel composite streams to ensure that the 9-channel composite streams are played synchronously;

[0198] The encoder synchronizes the composite stream to the streaming server node;

[0199] The streaming media server generates URL addresses for the 9-channel composite streams, which are URL1, URL2, ..., URL9;

[0200] The terminal starts playing. By default, the first sub-grid in the first composite stream is loaded. Fig. 20 Subgrid 1 in (a). It should be noted that the terminal decodes URL1 to obtain subgrid 1, subgrid 2, subgrid 3, and subgrid 4, but only enlarges subgrid 1 and tiles it to the playback window, and finally renders and displays it.

[0201] The terminal responds to the switching instruction and performs free viewpoint switching (in the second embodiment, the user switches to the right for illustration):

[0202] Scenario 1: Fig. 20 As shown in the dotted box (a), URL1 and URL2 are currently decoded, and subgrid 1 is displayed. The free viewpoint switches from left to right, from subgrid 1 to subgrid 2, and even to subgrid 5. The terminal detects that URL1 and URL2 have been decoded, and directly renders subgrid 2 or subgrid 5 for display;

[0203] Scenario 2: Fig. 20 As shown in the dotted box in (b), URL2 and URL3 have been decoded and sub-grid 10 is displayed.

[0204] The free viewpoint switches from left to right, switching to subgrid 11. The terminal detects that only subgrid 12 is left on the right of the composite stream URL3, and preloads the composite stream URL4, such as Fig. 20 As shown in (c), URL3 and URL4 have been decoded and sub-grid 11 is displayed.

[0205] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0206] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A video playback method, comprising: Decoding at least one of the multiple synthesized code streams to obtain at least one synthesized image matrix to be played; wherein each synthesized code stream can be decoded to obtain a synthesized image matrix, each synthesized image matrix is ​​divided into at least one sub-grid, the multiple synthesized code streams are generated by encoding multiple video code streams, each synthesized code stream is generated by encoding at least one video code stream, the multiple video code streams correspond to multiple display viewing angles, and each sub-grid corresponds to a display viewing angle corresponding to one video code stream; Rendering one of the at least one sub-grids to which the at least one matrix of the composite image to be played belongs to generate a playing picture; In response to an instruction to switch to a target sub-grid, determining whether the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played; When the synthetic image matrix to which the target sub-grid belongs is not one of the at least one synthetic image matrix to be played, a synthetic code stream that can be decoded to obtain the synthetic image matrix to which the target sub-grid belongs is decoded to obtain the synthetic image matrix to which the target sub-grid belongs, and the target sub-grid is rendered and displayed; When the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played, the target sub-grid is rendered and displayed.

2. The video playback method according to claim 1, wherein: The instruction to switch to the target sub-grid is an instruction generated according to the moving trajectory of the touch point on the playing screen; before the step of determining whether the synthetic image matrix to which the target sub-grid belongs is one of the at least one synthetic image matrix to be played, the video playing method further includes: At least one target sub-grid is determined according to the moving direction of the touch point in the moving track.

3. The video playback method according to claim 2, wherein: The display viewing angles corresponding to the multiple video code streams synthesized into the same synthesized code stream are adjacent in sequence; The step of determining at least one target sub-grid according to the moving direction of the touch point in the moving track comprises: According to the moving direction of the touch point in the moving track, a sub-grid corresponding to a display viewing angle adjacent to the display viewing angle of the playback screen is determined as the target sub-grid.

4. The video playback method according to claim 1, wherein: The video playback method further includes: Pre-decode at least one target composite code stream; wherein the target composite code stream is a composite code stream other than the composite code stream of the composite image matrix to which the target sub-grid belongs obtained by decoding among the multiple composite code streams.

5. The video playback method according to claim 4, wherein: The instruction to switch to the target sub-grid is an instruction generated according to the moving track of the touch point on the playing screen; Before the step of pre-decoding at least one target synthesized bitstream, the video playback method further includes: At least one target synthesized code stream is determined according to the moving direction of the touch point in the moving track.

6. The video playback method according to claim 5, wherein: Before the step of determining at least one target synthesized code stream according to the moving direction of the touch point in the moving trajectory, the video playback method further includes: The number of the target synthesized code streams to be pre-decoded is determined.

7. The video playback method according to claim 1, wherein: Before the step of decoding at least one of the multiple synthesized code streams to obtain at least one synthesized image matrix to be played, the video playback method further includes: Generate a play request according to the video play instruction, wherein the play request carries the address of the video pointed to by the video play instruction; Sending the play request to a streaming media server; Receive at least one composite code stream corresponding to the address and sent by the streaming media server.

8. A terminal, comprising: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the video playback method according to any one of claims 1 to 7.

9. A computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the video playback method according to any one of claims 1 to 7.

Images