Video confluence coding method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a video confluence coding method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the coding video streams provided by the plurality of first user sides are acquired, the plurality of channels of coding video streams are subjected to confluence processing to obtain the plurality of confluence video frames, the video feature information of each confluence video frame is acquired, and the target constant quality factor is determined according to the video feature information; the target confluence video stream can be sent to the second user side, and the second user side only needs to receive one channel of confluence video stream and does not need to start a plurality of decoders to decode different channels of video streams. The decoding overhead of the second user side is reduced, the constant quality factor of coding can be adaptively adjusted according to the confluence video content, and the video confluence coding effect in a multi-user video live broadcast scene is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a video concatenation encoding method, apparatus, device, storage medium and product. Background Art

[0002] In the live video business, multi-person rooms are a common application scenario. When users watch multi-person live video on the user end, they will pull multiple video streams at the same time and start multiple decoders to decode the multiple streams, resulting in huge decoding overhead on the user end.

[0003] In order to reduce decoding overhead, the server usually decodes multiple video streams and then splices them to form a new video source, which is then encoded to obtain a multi-person scene confluence. The user end only receives one confluence video stream for decoding and viewing. The encoding of the spliced ​​video source is generally based on the average bitrate (ABR). Video encoding based on the average bitrate aims to keep the average bitrate constant. In the multi-person video live broadcast scenario, the picture quality is often greatly sacrificed, and the video confluence encoding effect is poor. Summary of the invention

[0004] The embodiments of the present application provide a video confluence encoding method, apparatus, device, storage medium and product to solve the technical problem that the video confluence encoding scheme in the related art often greatly sacrifices the picture quality in the multi-person video live broadcast scenario, resulting in poor video confluence encoding effect. The constant quality factor of the encoding is adaptively adjusted according to the confluence video content, which can reduce the encoding bit rate without causing subjective perception loss, thereby improving the video confluence encoding effect in the multi-person video live broadcast scenario.

[0005] In a first aspect, an embodiment of the present application provides a video confluence encoding method, including:

[0006] Acquire multiple coded video streams provided by the first user terminal, and perform concatenation processing on the multiple coded video streams to obtain multiple concatenated video frames;

[0007] Acquire video feature information of each of the merged video frames, and determine a target constant quality factor according to the video feature information;

[0008] Each of the merged video frames is encoded according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to a second user terminal so that the second user terminal can play the target merged video stream.

[0009] In a second aspect, an embodiment of the present application provides a video confluence encoding device, including a confluence processing module, a parameter determination module and an encoding processing module, wherein:

[0010] The concatenation processing module is configured to obtain a plurality of encoded video streams provided by the first user terminal, and perform concatenation processing on the plurality of encoded video streams to obtain a plurality of concatenated video frames;

[0011] The parameter determination module is configured to obtain video feature information of each of the merged video frames, and determine a target constant quality factor according to the video feature information;

[0012] The encoding processing module is configured to encode each of the merged video frames according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to the second user terminal so that the second user terminal can play the target merged video stream.

[0013] In a third aspect, an embodiment of the present application provides a video concurrency encoding device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the video confluence encoding method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the video confluence encoding method as described in the first aspect when executed by a computer processor.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the video confluence encoding method as described in the first aspect.

[0018] The embodiment of the present application obtains encoded video streams provided by multiple first user terminals, and merges the multiple encoded video streams to obtain multiple merged video frames, obtains video feature information of each merged video frame, and determines a target constant quality factor based on the video feature information, and encodes each merged video frame based on the target constant quality factor to obtain a target merged video stream, and can send the target merged video stream to the second user terminal for the second user terminal to play the target merged video stream. After merging and encoding the multiple encoded video streams, the merged video stream is sent to the second user terminal. The second user terminal only needs to receive one merged video stream, and does not need to start multiple decoders for decoding different video streams, thereby reducing the decoding overhead of the second user terminal, and can adaptively adjust the encoding constant quality factor according to the merged video content, can reduce the encoding bit rate without causing subjective perception loss, and improve the video merging encoding effect in a multi-person video live broadcast scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a video confluence encoding method provided by an embodiment of the present application;

[0020] Figure 2 is a flowchart of another video confluence encoding method provided by an embodiment of the present application;

[0021] Figure 3 It is a structural schematic diagram of a video concatenation encoding device provided in an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a video confluence encoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0024] The video confluence encoding method provided in the present application can be applied to multi-person room live broadcast scenarios in video live broadcast services. It aims to first merge and encode multiple encoded video streams before sending the merged video stream to the second user terminal. It can adaptively adjust the constant quality factor of the encoding according to the merged video content, reduce the encoding bit rate without causing subjective perception loss, and improve the video confluence encoding effect in multi-person video live broadcast scenarios.

[0025] In the existing multi-person room video live broadcast, when users watch multi-person scenes on mobile terminals, they will pull multiple streams at the same time. The decoding of multiple streams will start multiple decoders, resulting in huge decoding overhead. In order to reduce the decoding overhead, the multiple streams can be merged into a single video stream. On the server side, it is usually necessary to decode the multiple source streams first and then splice them to form a new video source, and then encode the new video source to obtain the confluence of the multi-person scene. Finally, the user side only receives one confluence for decoding and viewing. Commonly used bit rate control methods in video encoding include: constant quantization parameter CQP (Constant Quantization Parameter), fixed quality factor CRF (Constant Rate Factor) and average bit rate ABR (Average Bitrate). Constant quantization parameter CQP is usually used for performance testing of coding tools and coding standard iterations. Video encoding is performed in a fixed quantization parameter QP manner, and the encoding bit rate is not controlled. The fixed quality factor CRF is generally used in scenarios where quality is the goal, such as consumer video storage, Blu-ray, DVD, etc. The fixed quality factor CRF can usually be seen as a CQP rate control scheme that conforms to the psychological visual model. The quality is close to the CQP rate control scheme but it saves more bitrate. For transmission videos such as live broadcasts and video conferencing, the encoding bitrate usually needs to be more stable to ensure the low latency and smoothness of the end-to-end video encoding link. Therefore, the ABR rate control scheme with a constant average bitrate is more commonly used.

[0026] Different from the single-person live broadcast scene, the content of multi-person live broadcast is usually more complex and changeable. The fixed code table ABR (Average Bitrate) encoding method commonly used in single-person live broadcast scenes cannot meet the application of multi-person scene confluence. The encoder in the ABR bit rate control scheme does not perceive the content of the coding unit, which usually leads to a significant decrease in video quality. The ABR bit rate control excessively pursues the average distribution of the bit rate in each frame, which will cause a sudden drop in picture quality when the video content changes / complex. In extreme cases, serious block effects and ringing effects will occur. The existing video confluence encoding scheme often greatly sacrifices picture quality in multi-person video live broadcast scenes. Based on this, a video confluence encoding method of an embodiment of the present application is provided to solve the technical problem that the existing video confluence encoding scheme often greatly sacrifices picture quality in multi-person video live broadcast scenes, and the video confluence encoding effect is poor.

[0027] Figure 1 A flowchart of a video fusion encoding method provided in an embodiment of the present application is given. The video fusion encoding method provided in an embodiment of the present application can be executed by a video fusion encoding device, which can be implemented by hardware and / or software and integrated in a video fusion encoding device (such as a server).

[0028] The following description is made by taking a video merging encoding device executing a video merging encoding method as an example. Figure 1 , the video confluence encoding method comprises:

[0029] S110: Acquire multiple coded video streams provided by the first user terminal, and perform concatenation processing on the multiple coded video streams to obtain multiple concatenated video frames.

[0030] The first user end provided by this solution is a terminal device that provides an encoded video stream, which can be a host end or a viewer end in a multi-person room live broadcast scene. For example, when multiple hosts are chatting or fighting with each other via video, or when the host and the viewer are chatting via video, the host end and / or viewer end participating in the video connection will collect the encoded video stream as the first user end and send it to the video confluence encoding device. The second user end provided by this solution is a terminal device for receiving and playing the confluence video stream to display the corresponding picture after the confluence of multiple encoded video streams, such as a viewer end that enters the corresponding live broadcast room to watch the live video.

[0031] Exemplarily, the coded video streams uploaded by multiple first user terminals in a multi-person video live broadcast room are obtained, and the multiple coded video streams are merged to obtain a plurality of continuous merged video frames, wherein each merged video frame records the picture information corresponding to the coded video streams of the multiple first user terminals, that is, a merged video frame and the pictures corresponding to the multiple coded video streams are displayed simultaneously. For example, the coded video streams provided by the multiple first user terminals are continuously spliced, and the video frames in the coded video streams are continuously spliced ​​to obtain a plurality of temporally continuous merged video frames.

[0032] S120: Obtain video feature information of each merged video frame, and determine a target constant quality factor according to the video feature information.

[0033] Exemplarily, after obtaining the merged video frame, the video feature information of each merged video frame is extracted, and the target constant quality factor (CRF, Constant Rate Factor) for encoding the merged video frame is determined based on the video feature information. Optionally, when determining the target constant quality factor for encoding the merged video frame based on the video feature information, the target constant quality factor can be obtained by analyzing and processing the video feature information based on a preset target video quality evaluation index. After encoding the merged video frame based on the constant quality factor, the video quality evaluation index corresponding to the encoded target merged video stream is consistent with the preset target video quality evaluation index or the difference is within the set difference range.

[0034] Optionally, the target constant quality factor is determined based on the video feature information, which may be determined by a trained neural network to correspond to the video feature information, or may be determined based on a mapping relationship between preset video feature information and preset constant quality factors. For example, a mapping relationship between a plurality of different preset video feature information and preset constant quality factors is pre-set (under this mapping relationship, the video quality evaluation index corresponding to the video frame obtained by encoding the video frame based on the constant quality factor is consistent with the preset target video quality evaluation index or the difference is within the set difference range), after determining the video feature information of the merged video frame, the similarity between the video feature information and each preset video feature information is determined, and based on the mapping relationship, the preset constant quality factor corresponding to the preset video feature information corresponding to the highest similarity is used as the target constant quality factor. Among them, the similarity between the video feature information can be represented according to a distance metric and / or a similarity metric, such as Hamming distance, Euclidean distance, cosine similarity, etc.

[0035] In one embodiment, the video feature information provided by the present solution may be a combination of one or more of spatial-temporal information (SITI, Spatial Info Time Info, spatial information and temporal information of the video, where the spatial information may be the correlation within the image (e.g., the content similarity between blocks in the image), and the temporal information may be the information between consecutive video frames (e.g., the motion vector of a continuously moving object)), block division depth information, motion vector information, and average quantization parameter, to accurately reflect the video features of the merged video frame, improve the accuracy of determining the target constant quality factor, and improve the video merge coding effect.

[0036] S130: encoding each merged video frame according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to the second user terminal so that the second user terminal can play the target merged video stream.

[0037] Exemplarily, after determining the target constant quality factor of the merged video frame, the merged video frame may be encoded according to the target constant quality factor to obtain a merged coded frame corresponding to the merged video frame, and the target merged video stream may be obtained based on the merged coded frame obtained by continuous encoding. For example, the encoder may adjust the encoding parameters (such as the encoding bit rate) according to the target constant quality factor based on CRF encoding so that the video encoding quality reaches the quality level corresponding to the target constant quality factor.

[0038] After obtaining the target merged video stream, the target merged video stream can be continuously sent to the second user terminal. After receiving the target merged video frame, the second user terminal can play the target merged video stream to display the screen corresponding to the multiple encoded video streams.

[0039] Optionally, when sending the target merged video stream to the second user terminal, the target merged video stream can also be transcoded based on the video level required by the second user terminal (different video levels correspond to different resolutions and / or bit rates), and the transcoded target merged video stream is sent to the second user terminal.

[0040] In the above, by obtaining the encoded video streams provided by multiple first user terminals, and converging the multiple encoded video streams to obtain multiple merged video frames, the video feature information of each merged video frame is obtained, and the target constant quality factor is determined according to the video feature information, and each merged video frame is encoded according to the target constant quality factor to obtain the target merged video stream, and the target merged video stream can be sent to the second user terminal for the second user terminal to play the target merged video stream. After the multiple encoded video streams are merged and encoded, the merged video stream is sent to the second user terminal. The second user terminal only needs to receive one merged video stream, and there is no need for the second user terminal to start multiple decoders to decode different video streams, thereby reducing the decoding overhead of the second user terminal, and the constant quality factor of the encoding can be adaptively adjusted according to the merged video content, and the encoding bit rate can be reduced without causing subjective perception loss, thereby improving the video confluence encoding effect in the multi-person video live broadcast scenario.

[0041] Based on the above embodiments, Figure 2 A flowchart of another video concatenation encoding method provided in an embodiment of the present application is given. The video concatenation encoding method is a specific embodiment of the above-mentioned video concatenation encoding method. Figure 2 , the video confluence encoding method comprises:

[0042] S210: Acquire multiple coded video streams provided by the first user terminals, and decode the multiple coded video streams to obtain multiple decoded video frames.

[0043] S220: splicing multiple decoded video frames to obtain a merged video frame.

[0044] After acquiring the encoded video streams provided by the multiple first user terminals, the encoded video streams are decoded to obtain decoded video frames corresponding to the encoded video streams, and the decoded video frames are spliced ​​to obtain a merged video frame.

[0045] In one embodiment, before the decoded video frames of various channels are spliced, the decoded video frames of various channels may be aligned according to the timestamp information of the decoded video frames, and the aligned decoded video frames of various encoded video streams may be spliced ​​into the same merged video frame.

[0046] This solution decodes the encoded video streams provided by multiple first user terminals into multiple decoded video frames, and splices the multiple decoded video frames in a video confluence encoding device to obtain a confluence video frame. There is no need for the second user terminal to start multiple decoders to decode different video streams, thereby reducing the decoding overhead of the second user terminal and improving the user's live viewing experience in multi-person video live broadcasts.

[0047] S230: Obtain video feature information of each merged video frame, and input the video feature information into a trained quality factor prediction model, and analyze and process the video feature information based on a preset target video quality evaluation index through the quality factor prediction model to obtain a target constant quality factor.

[0048] The video quality evaluation index provided by this scheme can be a combination of one or more of video multimethod assessment fusion (VMAF), peak signal-to-noise ratio (PSNR), mean opinion score (MOS), and structural similarity (SSIM). This scheme takes video multimethod assessment fusion as a video quality evaluation index as an example for description.

[0049] This solution extracts video feature information of the confluent video frame through the trained quality factor prediction model. After the quality factor prediction model is trained, the quality factor prediction model can be configured in the video confluent encoding device. The quality factor prediction model provided by this solution can analyze and process the input video frame based on the preset target video quality evaluation index, and output a constant quality factor. After the input video frame is encoded based on the constant quality factor, the video quality evaluation index corresponding to the encoded video frame obtained by encoding is consistent with the preset target video quality evaluation index or the difference is within the set difference range.

[0050] Exemplarily, after obtaining the video feature information of the merged video frame, the video feature information can be input into a quality factor prediction model, and the video feature information is analyzed and processed based on a preset target video quality evaluation index by the quality factor prediction model to obtain a target constant quality factor corresponding to the video feature information. This solution uses a quality factor prediction model to analyze and process the video feature information based on a preset target video quality evaluation index to accurately obtain a target constant quality factor, and can adaptively adjust the constant quality factor of the encoding according to the video feature information reflecting the merged video content, and can reduce the encoding bit rate without causing subjective perception loss, thereby improving the video merge encoding effect in a multi-person video live broadcast scenario.

[0051] In a possible embodiment, the training process of the quality factor prediction model provided by the present solution includes: taking feature information samples and real video quality evaluation indicators as model inputs and sample constant quality factors as model outputs, training the quality factor prediction model, and adjusting the network parameters of the quality factor prediction model according to the predicted constant quality factor output by the quality factor prediction model and the first loss corresponding to the sample constant quality factor, as well as the second loss corresponding to the predicted video quality evaluation indicator and the real video quality evaluation indicator. The predicted video quality evaluation indicator is obtained according to the encoding result of encoding the training data based on the predicted constant quality factor.

[0052] Exemplarily, multiple training data for training the quality factor prediction model are collected, wherein the training data may be sample images, and feature information samples corresponding to each training data are extracted, and the real video quality evaluation index (training label GT, Ground Truth) and sample constant quality factor corresponding to each training data are determined. After the collection of training data and the construction of the quality factor prediction model (for example, the quality factor prediction model is constructed based on a neural network), the quality factor prediction model is trained with the feature information sample and the real video quality evaluation index as the model input and the sample constant quality factor as the model output, so that the quality factor prediction model fits the nonlinear mapping relationship between the feature information sample and the real video quality evaluation index and the sample constant quality factor through iterative learning. And in the process of training the quality factor prediction model, according to the first loss corresponding to the predicted constant quality factor and the sample constant quality factor output by the quality factor prediction model, and the second loss corresponding to the predicted video quality evaluation index and the real video quality evaluation index, the network parameters of the quality factor prediction model are optimized and adjusted so that the target loss function converges or is less than the preset loss value. The target loss function may be determined based on the first loss and the second loss, for example, the sum of the first loss and the second loss, or the weighted sum of the first loss and the second loss is used as the value corresponding to the target loss function. Alternatively, the network parameters of the quality factor prediction model may be optimized and adjusted so that the first loss and / or the second loss converge or are less than a preset loss threshold.

[0053] The first loss provided by the present solution can be determined based on the distance between the predicted constant quality factor and the corresponding sample constant quality factor (e.g., Hamming distance, Euclidean distance, etc.), and the second loss can be determined based on the distance between the predicted video quality evaluation index and the corresponding real video quality evaluation index (e.g., Hamming distance, Euclidean distance, etc.). The predicted video quality evaluation index can be obtained based on the encoding result obtained by encoding the training data based on the predicted constant quality factor, for example, the characteristic information sample corresponding to the training data is input into the quality factor prediction model, the quality factor prediction model outputs the predicted constant quality factor, and the encoding result is obtained after encoding the training data based on the predicted constant quality factor. The video quality evaluation index corresponding to the encoding result is the predicted video quality evaluation index.

[0054] This scheme trains the quality factor prediction model through feature information samples, real video quality evaluation indicators and sample constant quality factors, and adjusts the network parameters of the quality factor prediction model according to the first loss corresponding to the predicted constant quality factor and the sample constant quality factor, and the second loss corresponding to the predicted video quality evaluation indicator and the real video quality evaluation indicator, so as to accurately train the quality factor prediction model, improve the accuracy of determining the target constant quality factor, and improve the video confluence encoding effect.

[0055] In a possible embodiment, the video confluence encoding method provided by the present solution may also include, before training the quality factor prediction model, using the feature information sample and the real video quality evaluation index as the model input and the sample constant quality factor as the model output: encoding the training data at a fixed bit rate to obtain a first sample encoding result, and extracting the feature information sample based on the first sample encoding result; determining multiple sample constant quality factors, encoding the training data based on each sample constant quality factor, respectively, to obtain a second sample encoding result corresponding to each sample constant quality factor, and determining the real video quality evaluation index based on the second sample encoding result.

[0056] Exemplarily, before training the quality factor prediction model, a set amount of training data (sample images) can be collected, and based on multiple different bit rates, each training data can be encoded at a fixed bit rate to obtain a first sample encoding result, and features can be extracted from each first sample encoding result to obtain feature information samples corresponding to each first sample encoding result.

[0057] In one embodiment, multiple sample constant quality factors can be determined, and the training data can be encoded (CRF encoding) based on each sample constant quality factor to obtain the second sample encoding results corresponding to each training data encoded under different sample constant quality factors, and the video feature information corresponding to each of the second sample encoding results can be calculated, and the video feature information can be used as the feature information sample corresponding to the second sample encoding result. Subsequently, the feature information sample and the real video quality evaluation index can be used as the model input, and the sample constant quality factor can be used as the model output to train the quality factor prediction model. This solution obtains feature information samples by performing fixed bit rate encoding and feature extraction on the training data, and obtains the real video quality evaluation index by encoding the training data based on different sample constant quality factors and calculating the video quality evaluation index, so as to quickly and accurately collect data for training the quality factor prediction model and improve the training efficiency and accuracy of the quality factor prediction model.

[0058] In one embodiment, the video confluence encoding method provided by the present solution determines a plurality of sample constant quality factors, and may determine the plurality of sample constant quality factors within a preset constant quality factor range according to a preset step size.

[0059] Exemplarily, according to a preset step length, with the endpoint of the preset constant quality factor range as the starting point, multiple sample constant quality factors are determined within the preset constant quality factor range. For example, assuming that the numerical range corresponding to the preset constant quality factor range is [10, 31], and the preset step length is 0.5, then starting from 10 as a sample constant quality factor, every 0.5 as a sample constant quality factor, multiple sample constant quality factors such as 10, 10.5, 11, ..., 30.5, 31 are obtained. By determining multiple sample constant quality factors within the preset constant quality factor range according to the preset step length, the present solution can encode and extract features from the training data according to these sample constant quality factors, obtain the video feature information corresponding to each training data under different sample constant quality factors, effectively enrich the diversity of feature information samples, improve the training accuracy of the quality factor prediction model, and improve the video confluence encoding quality.

[0060] In one embodiment, multiple sample videos can be collected, and sample video frames in the sample videos are used as training data. When encoding and feature extraction are performed on the training data to obtain the video feature information corresponding to each training data under different sample constant quality factors, a set number of sample video frames in multiple image groups (such as GOP, miniGOP) in the sample video can be encoded and feature extracted. For example, 5 sample video frames in multiple miniGOPs in the sample video are encoded and feature extracted, thereby reducing the amount of data processing and improving the efficiency of training sample collection. Subsequently, the sample video sequences can be CRF encoded in sequence within the preset constant quality factor range of [10,31] with a preset step size of 0.5 to obtain the preset target video quality evaluation index VMAF of each sample video sequence under different sample constant quality factors CRF as the training label GT. For example, in the training process of the quality factor prediction model, the corresponding feature information samples and the preset target video quality evaluation index VMAF can be used as input in units of image groups GOP. The quality factor prediction model fits the nonlinear mapping relationship between the feature information samples and the preset target video quality evaluation index VMAF and the sample constant quality factor CRF through iterative learning. The difference between the predicted constant quality factor CRF predicted by the quality factor prediction model and the sample constant quality factor CRF (i.e., the first loss) and the difference between the predicted video quality evaluation index VMAF corresponding to the predicted constant quality factor CRF and the preset target video quality evaluation index VMAF (i.e., the second loss) are used as losses to help the quality factor prediction model update network parameters. In the actual application link, the video feature information is combined with the preset target video quality evaluation index VMAF and input into the trained quality factor prediction model. The quality factor prediction model will give the predicted value (i.e., the target constant quality factor CRF) for the corresponding video and the preset target video quality evaluation index VMAF, and then use the target constant quality factor CRF to encode the merged video frame after merging.

[0061] S240: Encode each merged video frame according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to the second user terminal so that the second user terminal can play the target merged video stream.

[0062] In a possible embodiment, the video confluence encoding method provided by the present scheme encodes each confluence video frame according to a target constant quality factor, which can be: inputting the target constant quality factor into the encoder, using the target constant quality factor as the encoding quantization target through the encoder to determine the encoding parameters for encoding each confluence video frame, and encoding each confluence video frame based on the encoding parameters.

[0063] Exemplarily, after determining the target constant quality factor, the target constant quality factor is transmitted to the encoder. After receiving the merged video frame that needs to be encoded, the encoder will use the target constant quality factor as the encoding quantization target to determine the encoding parameters (such as encoding bit rate) for encoding the merged video frame, and encode the merged video frame according to the encoding parameters. The closer the average video quality (such as VMAF value) of multiple target merged video frames obtained by the encoding parameters determined based on the target constant quality factor is to the video quality corresponding to the target constant quality factor.

[0064] Among them, the target constant quality factor is sent to the encoder as a coding quantization target. The encoder will adjust the coding parameters (such as coding bit rate) according to the target constant quality factor to achieve the coding quality corresponding to the target constant quality factor. The smaller the target constant quality factor, the better the expected coding quality, tending to higher coding quality. The larger the target constant quality factor, the worse the coding quality, tending to lower coding quality. By setting different target constant quality factors, the coding bit rate can be reduced without causing subjective perception loss to meet the needs of different resolutions and different video contents, thereby improving the video confluence coding effect in multi-person video live broadcast scenarios.

[0065] In the above, by obtaining the encoded video streams provided by multiple first user terminals, and converging the multiple encoded video streams to obtain multiple merged video frames, the video feature information of each merged video frame is obtained, and the target constant quality factor is determined according to the video feature information, and each merged video frame is encoded according to the target constant quality factor to obtain the target merged video stream, and the target merged video stream can be sent to the second user terminal for the second user terminal to play the target merged video stream. After the multiple encoded video streams are merged and encoded, the merged video stream is sent to the second user terminal. The second user terminal only needs to receive one merged video stream, and there is no need for the second user terminal to start multiple decoders to decode different video streams, thereby reducing the decoding overhead of the second user terminal, and the constant quality factor of the encoding can be adaptively adjusted according to the merged video content, and the encoding bit rate can be reduced without causing subjective perception loss, thereby improving the video confluence encoding effect in the multi-person video live broadcast scenario. At the same time, by introducing a quality factor prediction model based on the video feature information of the merged video frame and the preset target video quality evaluation index on the server side (video merge encoding device), the transcoding efficiency of the merged video can be improved. While ensuring that the target merged video stream has the same subjective picture quality as the traditional ABR encoding, less bit rate consumption can be achieved, and the video compression efficiency on the transcoding side can be improved. The constant quality factor can be dynamically adjusted through the perception of the video content by the quality factor prediction model, further improving the video compression efficiency in the multi-person merge scenario.

[0066] Figure 3is a schematic diagram of the structure of a video concatenation encoding device provided in an embodiment of the present application. Figure 3 The video confluence encoding device includes a confluence processing module 31, a parameter determination module 32 and an encoding processing module 33.

[0067] Among them, the confluence processing module 31 is configured to obtain multiple encoded video streams provided by the first user terminal, and confluence the multiple encoded video streams to obtain multiple confluence video frames; the parameter determination module 32 is configured to obtain video feature information of each confluence video frame, and determine the target constant quality factor according to the video feature information; the encoding processing module 33 is configured to encode each confluence video frame according to the target constant quality factor to obtain a target confluence video stream, which is used to send the target confluence video stream to the second user terminal so that the second user terminal can play the target confluence video stream.

[0068] In the above, by obtaining the encoded video streams provided by multiple first user terminals, and converging the multiple encoded video streams to obtain multiple merged video frames, the video feature information of each merged video frame is obtained, and the target constant quality factor is determined according to the video feature information, and each merged video frame is encoded according to the target constant quality factor to obtain the target merged video stream, and the target merged video stream can be sent to the second user terminal for the second user terminal to play the target merged video stream. After the multiple encoded video streams are merged and encoded, the merged video stream is sent to the second user terminal. The second user terminal only needs to receive one merged video stream, and there is no need for the second user terminal to start multiple decoders to decode different video streams, thereby reducing the decoding overhead of the second user terminal, and the constant quality factor of the encoding can be adaptively adjusted according to the merged video content, and the encoding bit rate can be reduced without causing subjective perception loss, thereby improving the video confluence encoding effect in the multi-person video live broadcast scenario.

[0069] In a possible embodiment, the confluence processing module 31 performs confluence processing on multiple encoded video streams to obtain multiple confluence video frames, and is configured to: decode multiple encoded video streams to obtain multiple decoded video frames; and splice multiple decoded video frames to obtain a confluence video frame.

[0070] In a possible embodiment, the video feature information includes a combination of one or more of spatial and temporal domain information, block partition depth information, motion vector information, and an average quantization parameter.

[0071] In a possible embodiment, the parameter determination module 32 determines the target constant quality factor based on the video feature information, and is configured as follows: the video feature information is input into a trained quality factor prediction model, and the video feature information is analyzed and processed by the quality factor prediction model based on a preset target video quality evaluation index to obtain the target constant quality factor.

[0072] In a possible embodiment, the video confluence encoding device also includes a model training module, which is configured to train a quality factor prediction model. The training process of the quality factor prediction model is configured as follows: using feature information samples and real video quality evaluation indicators as model inputs and sample constant quality factors as model outputs, the quality factor prediction model is trained, and network parameters of the quality factor prediction model are adjusted according to the predicted constant quality factor output by the quality factor prediction model and the first loss corresponding to the sample constant quality factor, as well as the second loss corresponding to the predicted video quality evaluation indicator and the real video quality evaluation indicator. The predicted video quality evaluation indicator is obtained according to the encoding result of encoding the training data based on the predicted constant quality factor.

[0073] In a possible embodiment, the video confluence encoding device also includes a data collection module, which is configured to: perform fixed bit rate encoding on the training data to obtain a first sample encoding result, and extract a feature information sample based on the first sample encoding result; determine multiple sample constant quality factors, encode the training data based on each sample constant quality factor, obtain a second sample encoding result corresponding to each sample constant quality factor, and determine a real video quality evaluation index based on the second sample encoding result.

[0074] In a possible embodiment, the data collection module determines a plurality of sample constant quality factors and is configured to: determine the plurality of sample constant quality factors within a preset constant quality factor range according to a preset step size.

[0075] It is worth noting that in the embodiment of the above-mentioned video confluence encoding device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0076] The embodiment of the present application further provides a video confluence encoding device, which can integrate the video confluence encoding apparatus provided in the embodiment of the present application. Figure 4 Schematic diagram of the structure of a video concatenation encoding device provided in an embodiment of the present application. Figure 4 The video confluence encoding device is configured as follows: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the video confluence encoding method provided in the above embodiment. The video confluence encoding device, equipment, and computer provided above can be used to execute the video confluence encoding method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0077] The embodiment of the present application also provides a non-volatile storage medium storing computer executable instructions, which are used to execute the video confluence encoding method provided in the above embodiment when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the video confluence encoding method provided above, can also execute the related operations in the video confluence encoding method provided in any embodiment of the present application. The video confluence encoding apparatus, device and storage medium provided in the above embodiments can execute the video confluence encoding method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the video confluence encoding method provided in any embodiment of the present application.

[0078] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a user terminal or a processor therein to execute all or part of the steps of the video confluence encoding method provided in each embodiment of the present application.

Claims

1. A video concatenation encoding method, characterized in that: include: Acquire multiple coded video streams provided by the first user terminal, and perform concatenation processing on the multiple coded video streams to obtain multiple concatenated video frames; Acquire video feature information of each of the merged video frames, and determine a target constant quality factor according to the video feature information; Each of the merged video frames is encoded according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to a second user terminal so that the second user terminal can play the target merged video stream.

2. The video concatenation encoding method according to claim 1, characterized in that: The step of combining multiple coded video streams to obtain multiple combined video frames includes: Decoding multiple coded video streams to obtain multiple decoded video frames; The multiple decoded video frames are spliced ​​together to obtain a merged video frame.

3. The video concatenation encoding method according to claim 1, characterized in that: The video feature information includes a combination of one or more of spatial and temporal domain information, block partition depth information, motion vector information and an average quantization parameter.

4. The video concatenation encoding method according to claim 1, characterized in that: The determining a target constant quality factor according to the video feature information includes: The video feature information is input into a trained quality factor prediction model, and the video feature information is analyzed and processed by the quality factor prediction model based on a preset target video quality evaluation index to obtain a target constant quality factor.

5. The video concatenation encoding method according to claim 4, characterized in that: The training process of the quality factor prediction model includes: The quality factor prediction model is trained with feature information samples and real video quality evaluation indicators as model inputs and sample constant quality factors as model outputs. The network parameters of the quality factor prediction model are adjusted according to the predicted constant quality factor output by the quality factor prediction model and the first loss corresponding to the sample constant quality factor, as well as the predicted video quality evaluation indicator and the second loss corresponding to the real video quality evaluation indicator. The predicted video quality evaluation indicator is obtained according to the encoding result of encoding the training data based on the predicted constant quality factor.

6. The video concatenation encoding method according to claim 5, characterized in that: Before training the quality factor prediction model using the feature information sample and the real video quality evaluation index as the model input and the sample constant quality factor as the model output, the method further includes: Performing fixed-rate encoding on the training data to obtain a first sample encoding result, and extracting a feature information sample according to the first sample encoding result; Determine multiple sample constant quality factors, encode the training data based on each of the sample constant quality factors, obtain second sample encoding results corresponding to each of the sample constant quality factors, and determine a real video quality evaluation index based on the second sample encoding results.

7. The video concatenation encoding method according to claim 6, characterized in that: The step of determining a plurality of sample constant quality factors comprises: A plurality of sample constant quality factors are determined in a preset constant quality factor range according to a preset step size.

8. The video concatenation encoding method according to claim 1, characterized in that: The encoding process of each of the merged video frames according to the target constant quality factor includes: The target constant quality factor is input into the encoder, and the encoder uses the target constant quality factor as an encoding quantization target to determine encoding parameters for encoding each of the merged video frames, and encodes each of the merged video frames based on the encoding parameters.

9. A video concatenation encoding device, characterized in that: It includes a confluence processing module, a parameter determination module and an encoding processing module, wherein: The concatenation processing module is configured to obtain a plurality of encoded video streams provided by the first user terminal, and perform concatenation processing on the plurality of encoded video streams to obtain a plurality of concatenated video frames; The parameter determination module is configured to obtain video feature information of each of the merged video frames, and determine a target constant quality factor according to the video feature information; The encoding processing module is configured to encode each of the merged video frames according to the target constant quality factor to obtain a target merged video stream, which is used to send the target merged video stream to the second user terminal so that the second user terminal can play the target merged video stream.

10. A video concatenation encoding device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video confluence encoding method as described in any one of claims 1-8.

11. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the video concurrency encoding method as described in any one of claims 1 to 8 when executed by a computer processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video concurrency encoding method according to any one of claims 1 to 8 is implemented.

Images