Method and system for encoding multiple video streams based on precoding information multiplexing

Abstract

The embodiment of the application discloses a kind of based on pre-coding information multiplexing multi-channel video stream encoding method and system.The technical scheme provided in the embodiment of the application, by decoding and scaling processing based on input source video stream, obtain the video sequence of multiple different resolutions, select the video sequence of specified resolution as analysis road video, the video sequence of the rest resolution is multiplexing road video;Analysis road video is pre-encoded, generates first pre-coding information, for formal encoding of analysis road video;According to the resolution of multiplexing road video and the resolution of analysis road video, first pre-coding information is converted into second pre-coding information, for the formal encoding of corresponding multiplexing road video.Using the above technical means, the pre-coding process of multiplexing road video can be omitted, while ensuring the encoding quality of multiplexing road video, improving the encoding efficiency of multiplexing road video, realizing the encoding acceleration effect of multi-channel video stream.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of video coding, in particular to a multi-path video stream coding method and system based on pre-coding information multiplexing. BACKGROUND

[0002] At present, in a video live broadcast and on-demand system, in order to adapt to different bandwidth conditions and different terminal devices, a server usually provides video streams of different resolutions and different code rates. The server uses a multi-resolution transcoding architecture to transcode an input source video stream into video streams of different resolutions. In addition, in order to reduce the coding operation complexity, the multi-resolution transcoding architecture uses the correlation between the decoding information of the input source video stream and the coding information in the coding of the multi-path video stream to quickly predict the coding unit depth selection and the mode selection in the transcoding process of the multi-path video stream, thereby improving the coding speed.

[0003] However, most of the related multi-path video stream coding acceleration schemes are for coding acceleration of the formal coding process of the video stream, and there is a lack of corresponding coding acceleration optimization setting for the pre-coding process of the video stream, and the calculation complexity is relatively high. SUMMARY

[0004] The embodiment of the present application provides a multi-path video stream coding method and system based on pre-coding information multiplexing, which can reduce the calculation complexity of the video stream pre-coding in the multi-resolution transcoding architecture, improve the multi-path video stream coding speed, and solve the technical problem of the high calculation complexity of the video stream pre-coding process in the multi-resolution transcoding architecture.

[0005] In a first aspect, the embodiment of the present application provides a multi-path video stream coding method based on pre-coding information multiplexing, comprising:

[0006] Decoding and scaling processing are performed based on an input source video stream to obtain video sequences of different resolutions, a video sequence of a specified resolution is selected as an analysis path video, and the video sequences of the remaining resolutions are selected as multiplexing path videos;

[0007] The analysis path video is pre-coded to generate first pre-coding information, which is used for formal coding of the analysis path video;

[0008] The first pre-coding information is converted into second pre-coding information according to the resolutions of the multiplexing path videos and the resolution of the analysis path video, which is used for formal coding of the corresponding multiplexing path video.

[0009] In a second aspect, the embodiment of the present application provides a multi-path video stream coding system based on pre-coding information multiplexing, comprising:

[0010] a scaling module configured to decode and scale the input source video stream to obtain a plurality of video sequences with different resolutions, select a video sequence with a specified resolution as an analysis path video, and select the remaining video sequences with different resolutions as multiplexing path videos;

[0011] a pre-encoding module configured to pre-encode the analysis path video to generate first pre-encoding information for formal encoding of the analysis path video;

[0012] a multiplexing module configured to convert the first pre-encoding information into second pre-encoding information according to resolutions of the multiplexing path videos and a resolution of the analysis path video, and use the second pre-encoding information for formal encoding of the corresponding multiplexing path videos.

[0013] In a third aspect, an embodiment of the present application provides a multi-path video stream encoding device based on pre-encoding information multiplexing, comprising:

[0014] a memory and one or more processors;

[0015] the memory is configured to store one or more programs;

[0016] when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the multi-path video stream encoding method based on pre-encoding information multiplexing as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are configured to perform the multi-path video stream encoding method based on pre-encoding information multiplexing when executed by a computer processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which contains instructions, and the instructions make a computer or a processor execute the multi-path video stream encoding method based on pre-encoding information multiplexing when the instructions are executed on the computer or the processor.

[0019] The embodiment of the present application obtains multiple video sequences with different resolutions by decoding and scaling processing based on the input source video stream, selects a video sequence with a specified resolution as an analysis path video, and selects the video sequences with the rest resolutions as multiplexing path videos; pre-encodes the analysis path video to generate first pre-encoding information for formal encoding of the analysis path video; and converts the first pre-encoding information into second pre-encoding information according to the resolutions of the multiplexing path videos and the resolution of the analysis path video, for formal encoding of the corresponding multiplexing path videos. By using the above technical means, the pre-encoding information of the analysis path video is converted according to the resolution conversion, and the converted pre-encoding information is used for formal encoding of the multiplexing path videos. In this way, the pre-encoding process of the multiplexing path videos can be omitted, the encoding efficiency of the multiplexing path videos is improved while the encoding quality of the multiplexing path videos is ensured, and the encoding acceleration effect of the multiple video streams is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a multiple video stream encoding method based on pre-encoding information multiplexing provided by the embodiment of the present application;

[0021] Figure 2 is an encoding flowchart in the embodiment of the present application;

[0022] Figure 3 is a pre-encoding information conversion flowchart in the embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of a multiple video stream encoding system based on pre-encoding information multiplexing provided by the embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of a multiple video stream encoding device based on pre-encoding information multiplexing provided by the embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further describes the specific embodiments of the present application with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, but not all contents. Before discussing the example embodiments in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted by flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes can be terminated when the operations are completed, but can also have additional steps not included in the drawings. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0026] The application provides a multi-path video stream encoding method based on pre-encoding information multiplexing, aiming to utilize the correlation of pre-encoding information among multi-path videos with different resolutions, convert the pre-encoding information of the analysis path video according to the resolution conversion, and use the converted pre-encoding information to perform formal encoding of the multiplexing path video. In this way, the pre-encoding process of the multiplexing path video is simplified, the encoding efficiency of the multiplexing path video is improved, and the encoding acceleration effect of the multi-path video stream is realized.

[0027] In the video live broadcast and on-demand scenarios, in order to adapt to different bandwidth conditions and different terminal devices, the server usually provides video streams with different resolutions and different code rates, and the multi-resolution transcoding architecture emerges as the times require. In the related multi-resolution transcoding architecture, the input video stream is first decoded into the YUV domain, and then scaled to generate multiple resolution source videos, and finally an encoder is started for each resolution source video for encoding. Since each resolution source video is independently pre-encoded and encoded, the multi-resolution transcoding architecture brings huge computing pressure to the server, especially when multi-resolution HEVC transcoding is required, the computing complexity of HEVC directly affects the number of parallel transcoding paths under the real-time transcoding condition of the server. Therefore, there is still much speed space for optimization of the encoder under the multi-resolution transcoding architecture.

[0028] Although the video to be encoded under the multi-resolution transcoding architecture has different resolutions, the basic content is consistent, and there is strong correlation between different resolutions in terms of encoding characteristics. For example, the occurrence position of the scene switching frame is usually consistent in the encoding decision of multiple resolutions, and the CU partition depth and CU mode of the same frame type have relatively consistent statistical characteristics in the encoding results of multiple resolutions. Therefore, utilizing the encoding result of a specific resolution to accelerate the encoding process of other resolutions can greatly improve the encoding speed. In addition, for the input video stream encoding format of the multi-resolution transcoding framework, whether it is AVC or HEVC, since the compression standards follow the traditional block partition structure and the encoding algorithm has certain continuity, it is also a feasible way to utilize the decoding information of the source video to accelerate the encoding process of multiple resolutions.

[0029] The existing multi-resolution transcoding architecture acceleration scheme, whether with the help of the decoding information of the input source video or with the help of the encoding information of a certain resolution, most of the selected acceleration is the rate-distortion optimization of other path encoding block division rate and the rate-distortion optimization part of the encoding block mode selection. For the encoder with a pre-encoding module, the pre-encoding module will be used to pre-encode the video sequence to be encoded before formal encoding, make encoding frame type decision according to the pre-encoding information, and generate a spatial adaptive qp offset table (qpAqOffset). At the same time, the encoding tree analysis will be carried out to obtain the temporal adaptive qp offset table (qpCutreeOffset). Therefore, the calculation complexity of the pre-encoding module in this type of encoder is also considerable. Based on this, the multi-path video stream encoding method based on pre-encoding information multiplexing provided by the embodiments of the present application is provided to solve the technical problem of high calculation complexity of the video stream pre-encoding process in the multi-resolution transcoding architecture.

[0030] Embodiments:

[0031] Figure 1 A flowchart of a multi-path video stream encoding method based on pre-encoding information multiplexing provided by the embodiments of the present application is given. The multi-path video stream encoding method based on pre-encoding information multiplexing provided in the embodiments can be executed by a multi-path video stream encoding device based on pre-encoding information multiplexing. The multi-path video stream encoding device based on pre-encoding information multiplexing can be realized by software and / or hardware. The multi-path video stream encoding device based on pre-encoding information multiplexing can be composed of two or more physical entities, or can be composed of one physical entity. Generally, the multi-path video stream encoding device based on pre-encoding information multiplexing can be an encoder device, a computer, a mobile phone, a tablet computer, and the like.

[0032] The following describes the multi-path video stream encoding method based on pre-encoding information multiplexing executed by the multi-path video stream encoding device based on pre-encoding information multiplexing as an example. Referring to Figure 1 , the multi-path video stream encoding method based on pre-encoding information multiplexing specifically includes:

[0033] S110, decoding and scaling processing based on the input source video stream to obtain a plurality of video sequences with different resolutions, and selecting a video sequence with a specified resolution as an analysis path video and the remaining video sequences with different resolutions as multiplexing path videos;

[0034] S120, pre-encoding the analysis path video to generate first pre-encoding information for formal encoding of the analysis path video;

[0035] S130, converting the first pre-encoding information into second pre-encoding information according to the resolution of the multiplexing path video and the resolution of the analysis path video for formal encoding of the corresponding multiplexing path video.

[0036] The pre-encoding module of the multi-resolution video stream encoding process is used to convert the pre-encoding information of a specific resolution video stream into pre-encoding information of other resolutions, so as to accelerate the pre-encoding process of other resolution video streams. Thus, the operation complexity of the entire multi-resolution transcoding architecture is reduced, and the server can support more channels and higher resolution real-time transcoding.

[0037] Exemplarily, referring to Figure 2 , a multi-channel video stream encoding flowchart of the embodiment of the application is provided. For the source video stream input into the multi-resolution transcoding architecture, it is transcoded into different resolution outputs according to actual transcoding requirements. Assuming that the server input is a 1080p (resolution is 1080x1920) source video stream (which can be in H.264 / HEVC encoding format), it needs to be transcoded into 720p (720x1280), 540p (540x960), and 1080p three-channel HEVC stream outputs. First, the source video stream needs to be decoded into YUV space, then pushed into three scaling modules in a certain order through the frame pushing control module, scaled according to the requirements of each target resolution, and the scaled YUV video sequences of each resolution are input into the HEVC encoder for encoding. The formal encoding modules of the three-channel HEVC encoder are consistent, while the pre-encoding modules are operated in normal mode and multiplex mode. In the normal mode, the pre-encoding module needs to complete 1 / 2 downsampling, spatial adaptive qp offset analysis to obtain the spatial adaptive qp offset table (qpAqOffset), calculation of 8x8 block intra and inter optimal encoding cost (intraCost and interCost) and inter block motion vector, and coding tree (CU-tree) analysis, as well as storage of the above pre-encoding information generated in the pre-encoding process. In the multiplex mode, only the corresponding pre-stored pre-encoding information needs to be read to fit the spatial adaptive qp offset table, intra and inter optimal encoding cost information suitable for the current resolution, and at the same time, the coding tree analysis of the current multiplex channel video can be completed according to the pre-encoding information.

[0038] Among them, the video using normal mode for pre-encoding is called analysis path video, and the other video using multiplex mode for pre-encoding is called multiplex path video. According to the actual test, after weighing the coding quality and acceleration effect, the video stream with 540p resolution is selected as the analysis path video, the pre-encoding of the video stream is carried out using the normal mode, and the pre-encoding information generated in the pre-encoding process is stored in the public buffer which can be accessed by the encoders of other video streams. The video streams with 720p and 1080p resolution are used as multiplex path video and pre-encoded using multiplex mode. By reading the pre-encoding information of the 540p video stream, pre-encoding information conforming to the resolution of each video stream is generated for subsequent formal encoding. The analysis path video encoding process is no different from normal encoding, so there is no change in quality and speed. The multiplex path video omits part of the pre-encoding calculation, so as to realize the improvement of the encoding speed. It should be noted that in actual application, according to different resolutions of the multiple video streams to be transcoded, the actual coding quality and acceleration effect can be combined to select the YUV video sequence with a specified resolution as the analysis path video, and the rest as the multiplex path video. The specified resolution of the analysis path video in the present application is not fixedly limited, and the actual coding quality and acceleration effect are optimized, which will not be described here.

[0039] Since in the pre-encoding stage, the main time-consuming processes are 1 / 2 low resolution downsampling, spatial adaptive qp offset analysis, intra and inter optimal cost calculation and coding tree analysis, the present application simplifies the pre-encoding process of the multiplex path by multiplexing the corresponding spatial adaptive qp offset table and 8x8 block intra and inter optimal encoding cost, and performing coding tree analysis based on the multiplexed pre-encoding information.

[0040] Among them, 1 / 2 low resolution downsampling is mainly used to generate Y component pixel planes for pre-encoding, in order to be able to search 8x8 blocks in pixel units. When downsampling, 4 1 / 2 low resolution Y component pixel planes need to be generated according to different sampling positions. The pre-encoding process mainly calculates the intra and inter encoding cost of 8x8 pixel blocks on the 1 / 2 low resolution Y component, the inter block motion vector information, and then analyzes the coding tree based on these information to calculate the temporal adaptive qp offset. Therefore, by multiplexing the intra and inter optimal encoding cost and inter block motion vector in the pre-encoding information of the analysis path video, the current pre-encoding stage can omit the construction of 1 / 2 low resolution pixel planes, which can save memory and part of the calculation amount.

[0041] The spatial domain adaptive QP offset analysis is calculated based on the source resolution image, and the source resolution image is usually divided into 16x16 pixel blocks for statistical analysis, and the variance and other information of each 16x16 block are calculated to generate a spatial domain adaptive QP offset table. The larger the resolution is, the more 16x16 pixel blocks there are, and the greater the time consumption of the spatial domain adaptive QP offset module. However, the spatial domain adaptive QP offset is usually used in combination with the temporal domain adaptive QP offset table, and the temporal domain adaptive QP offset (qpCutreeOffset) obtained by the coding tree analysis has a greater impact on coding than the spatial domain adaptive QP offset. Therefore, the spatial domain adaptive QP offset of the analysis road video can be directly reused.

[0042] The coding tree analysis is ultimately to obtain the temporal domain adaptive QP offset table. In the implementation process, the intra optimal coding cost intraCost and the inter optimal coding cost interCost of each 8x8 block and the motion vector Mvs of the block need to be calculated according to the frame structure. Then, the coding cost reference chain of the block is constructed according to the information, the propagation cost PropagateCost of each block is calculated from back to front for each reference chain. Finally, the QP offset of each block is calculated according to PropagateCost and intraCost. The calculation complexity of the coding tree analysis is concentrated in the calculation of intraCost and interCost, which is also the module with the highest calculation complexity in the pre-coding process. Therefore, the intraCost, interCost and Mvs of a specific resolution are considered to be reused. However, considering that the 8x8 blocks of different resolutions cover different texture regions, the reference chain constructed by the coding tree analysis will have large differences due to inconsistent motion boundaries. Therefore, the generation of the temporal domain adaptive QP offset is still calculated by each multiplexed road video, and appropriate fitting correction is needed when multiplexing intraCost, interCost and Mvs to reduce the impact of different resolutions of 8x8 blocks covering different texture regions.

[0043] Based on this, the multi-resolution transcoding acceleration scheme of the present application can mainly include two parts, one part is the push frame control and common buffer maintenance independent of the external encoder, the push frame control module is used for coordinating the encoding sequence of the analysis path video and the multiplex path video; the common buffer maintains a buffer area buffer which can be accessed by the multiple encoders, wherein the pre-encoding information of the analysis path video is written into the buffer area during the pre-encoding process, and the multiplex path video reads the information from the buffer area. The other part is two working modes of the pre-encoding module in the encoder, in the normal mode, the encoder normally performs the pre-encoding process calculation, and in the multiplex mode, the encoder only needs to generate the pre-encoding information suitable for the resolution of the current multiplex path video according to the pre-encoding information of the analysis path video, and at the same time, the multiplex pre-encoding information completes the coding tree analysis of the current video. Through the multiplexing of the pre-encoding information, the encoding acceleration of the multi-path video stream in the embodiment of the present application is realized.

[0044] The common buffer is a memory space which can be accessed by the analysis path video and the multiplex path video, and the size is related to the number of single frames of the pre-encoding information and the encoding speed of the analysis path video and the multiplex path video. It needs to ensure that the multiplex path video can find the pre-encoding information with the same sequence number in the buffer, so as to avoid the situation that the analysis path video rewrites and covers the unused pre-encoding information in the buffer due to too small buffer. Alternatively, the present application opens a 64-frame pre-encoding information size memory for data exchange between multiple encoders, the buffer is filled by the analysis path video, when the analysis path video fills the buffer, the first frame data in the buffer needs to be deleted, and the subsequent data is sequentially moved to the front end of the buffer, and the new data is written to the last end of the buffer.

[0045] Specifically, before converting the first pre-encoding information into the second pre-encoding information according to the resolution of the multiplex path video and the resolution of the analysis path video, comprising:

[0046] Determine the sequence number of the current to-be-encoded video frame of the multiplex path video, extract the first pre-encoding information of the corresponding sequence number of the analysis path video, to convert and generate the second pre-encoding information of the current to-be-encoded video frame.

[0047] In the multi-resolution transcoding architecture, the decoded YUV video sequence of the input video stream is sent into multiple independent encoders for encoding. For different video frames, the picture sequence number (POC, Picture Order Count) in the display order is increasing. Therefore, when encoding a video frame in the multiplex path, the pre-encoding information of a video frame with the same sequence number as the analysis path needs to be found, and the encoding information corresponding to the sequence number is used for the encoding of the current video frame.

[0048] Further, the pre-encoding of the analysis path video comprises: frame pushing control of the analysis path video, pre-encoding frame by frame according to sequence number order; and converting first pre-encoding information into second pre-encoding information according to the resolution of the multiplex path video and the resolution of the analysis path video, which comprises: according to the first sequence number information of the video frame of the analysis path video which has completed pre-encoding, triggering the conversion of the second pre-encoding information of the video frame of the multiplex path video corresponding to the second sequence number information.

[0049] Since the multiplex path video needs to use the pre-encoding information of the analysis path video, it must wait for the analysis path video to complete the pre-encoding of the video frame corresponding to the sequence number, and then start the encoding of the video frame corresponding to the sequence number of the multiplex path video after the pre-encoding information is generated.

[0050] Generally, the pre-encoding module needs to cache "lookaheadDepth (pre-encoding lookahead list length) + bframes (b frames) + 1" frame sequences to start the pre-encoding analysis process, and one analysis can complete the analysis of all frames of a small group (minigop), and the key frame is often output separately. Therefore, the multiplex path video needs to wait for "lookaheadDepth + bframes" frames to start encoding, so as to ensure that the multiplex path video finds the same pre-encoding information as itself POC in the common cache.

[0051] Taking 540p as the analysis path video as an example, the external encoding parameter is set to lookaheadDepth = 4 and bframes = 2. Then when the 540p encoder inputs 6 frames, the pre-encoding analysis starts to calculate and complete the pre-encoding information calculation and storage of the first frame with poc = 0. When 7 frames are input, the pre-encoding information calculation and storage of three frames with poc = 3, 1 and 2 are completed, and when 10 frames are input, the pre-encoding information calculation and storage of three frames with poc = 6, 4 and 5 are completed. If 1080p is the multiplex path video, when the 540p input frame with poc = 7 enters the encoder, the 1080p can input the first frame data with its own poc = 0, and query the pre-encoding information with poc = 0 in the common cache. The frame pushing control process of the analysis path video and the multiplex path video and the state change of the common cache every time the frame is pushed are shown in Table 1:

[0052] Table 1

[0053]

[0054]

[0055] Further, the pre-encoding information generated by the analysis path is stored frame by frame. Let cuCount be the total number of coding blocks obtained after the 1 / 2 down-sampling pixel plane of the analysis path video is divided into 8x8 blocks. Then a frame of pre-encoding information contains the following contents:

[0056] qpAqOffset[cuCount]: spatial adaptive QP offset table, contains QP offset for each 16x16 block of the original frame, the number of 16x16 blocks in the original frame is consistent with the number of 8x8 blocks in the 1 / 2 down-sampled pixel plane, so cuCount QP offsets for AQ algorithm are needed for each frame. The spatial adaptive QP offset table is a double type array.

[0057] lowresCosts[BFRAME_MAX+2][BFRAME_MAX+2][cuCount]: intra and inter optimal encoding cost, which are all stored in the lowresCosts data structure. lowresCosts is a three-dimensional array, the first two dimensions represent the distance of the current frame and the forward reference frame and the distance of the current frame and the backward reference frame, if the forward and backward distances are 0, then the lowresCosts actually stores the intra cost, and the rest are inter cost. The third dimension represents the position of the 8x8 block, for example, lowresCosts[1][1][0] of the current frame refers to the bi-prediction cost of the first 8x8 block of the current frame and the previous and next frames, and lowresCosts[0][0][0] represents the intra encoding cost of the first 8x8 block of the current frame. The specific data type of lowresCosts is uint16_t, which takes the highest two bits to store the reference frame list information of the inter cost. Therefore, using a lowresCosts data structure can store the intra and inter optimal encoding costs and the list information required by the coding tree analysis. The first two dimensions of lowresCosts are related to the reference distance, and the maximum reference distance is the maximum number of b-frames allowed + 2, where the maximum number of b-frames BFRAME_MAX can be adjusted more reasonably according to actual encoding requirements. When extracting the corresponding information from the data structure, the intra optimal encoding cost intraCost[i], the inter optimal encoding cost interCost[i], and the list information need to be calculated from lowresCost, and the calculation formulas are as follows:

[0058] intraCost[i] = lowresCost[0][0][i] & 0x3fff

[0059] interCost[i] = lowresCost[dist0][dist1][i] & 0x3fff

[0060] list[i] = lowresCost[dist0][dist1][i] >> 14

[0061] Wherein, dist0 represents the distance between the current frame and the forward reference frame, dist1 is the distance between the current frame and the backward reference frame, dist0 and dist1 are given in the analysis process of the coding tree analysis, the frame structure of the analysis path video and the multiplex path video is consistent, and the determination of dist0 and dist1 in the coding tree analysis is also consistent, and i represents the position of each 8x8 block in the loop.

[0062] Mvs[2][BFRAME_MAX+2][cuCount]: the inter-frame coding cost corresponding motion vector information is stored in Mvs, the first dimension represents forward / backward prediction, the second dimension represents the distance between the current frame and the reference frame, and the third dimension is the position of the 8x8 block. For example, Mvs[1][2][0] represents the backward prediction of the first 8x8 block of the current frame, and the distance to the reference frame is 2, that is, the prediction mv of the second backward frame. Mvs is a three-dimensional array of MV type, and MV type is a structure body, occupying four bytes, the first two bytes store the x component of mv, and the last two bytes store the y component of mv.

[0063] In addition, the sequence number information of each frame video frame needs to be stored, which is used to indicate which frame these pre-coding information belongs to, so that the multiplex path video can adapt to its own frame sequence; the resolution information (i.e. width and height information) of the analysis path video is used to generate pre-coding information suitable for its own resolution when the multiplex path video is encoded;

[0064] After that, based on the pre-coding information generated in the above analysis path pre-coding process, on the one hand, it is used for the formal encoding of the subsequent analysis path video, and assists the mode decision process of the formal encoding. On the other hand, it is stored in the shared buffer for the multiplex path video to extract and convert the pre-coding information. The first pre-coding information of the analysis path video is converted into the second pre-coding information suitable for the multiplex path video.

[0065] Wherein, as shown in Figure 3 The conversion process of the second pre-coding information includes:

[0066] S1301, calculating scaling ratio information based on the resolution of the multiplex path video and the resolution of the analysis path video;

[0067] S1302, calculating the second pre-coding information according to the scaling ratio information and the first pre-coding information, the first pre-coding information including the first spatial adaptive qp offset, the first intra-frame optimal coding cost, the first inter-frame optimal coding cost and the first motion vector information, and the second pre-coding information including the second spatial adaptive qp offset, the second intra-frame optimal coding cost, the second inter-frame optimal coding cost and the second motion vector information.

[0068] Based on the above analysis of the spatial domain adaptive QP offset table, the intra and inter optimal coding cost and the inter block motion vector stored in the pre-encoding process of the analysis video, they are defined as the first spatial domain adaptive QP offset, the first intra optimal coding cost, the first inter optimal coding cost and the first motion vector information. The corresponding converted second coding information includes the second spatial domain adaptive QP offset, the second intra optimal coding cost, the second inter optimal coding cost and the second motion vector information. When the pre-encoding information conversion is performed, the resolution difference between the analysis video and the multiplexing video is analyzed to determine the corresponding scaling ratio, so as to perform the pre-encoding information conversion according to the scaling ratio.

[0069] After the multiplexing video starts encoding, it needs to push frames to the encoder. The frame sequence structure entering the encoder has a pointer of external public cache. After entering the pre-encoding process, the pre-encoding information with the same sequence number as the current frame sequence number is first found in the public cache, and the data is assigned to the corresponding data structure of the current frame. Due to the different resolutions of the analysis video and the multiplexing video, the number of 8x8 blocks is different, resulting in the difference in the dimension of the multiplexing data, so the pre-encoding information needs to be processed according to the scaling ratio between the two.

[0070] By reading the resolution of the analysis video in the pre-encoding information, i.e. the width (widAnalysis) and the height (heiAnalysis), and combining the resolution of the current multiplexing video, i.e. the width (widReuse) and the height (heiReuse), the scaling ratios in the X and Y directions are calculated as follows:

[0071]

[0072]

[0073] Among them, scaledRatioX represents the scaling ratio in the X direction, and scaledRatioY represents the scaling ratio in the Y direction. Based on the above scaling ratio and the corresponding first pre-encoding information, the corresponding second pre-encoding information can be calculated.

[0074] Based on the scaling ratio, the correspondence between the coding block coordinates of the multiplexing video and the coding block coordinates of the analysis video is first determined according to the scaling ratio information; and the second spatial domain adaptive QP offset is calculated based on the correspondence and the first spatial domain adaptive QP offset.

[0075] According to the 8x8 block position of the current multiplexing video and the scaling ratio calculated above, the 8x8 block position of the corresponding analysis video is calculated. Assuming that the 8x8 block position coordinates of the current multiplexing video are (cuX, cuY), then the 8x8 block coordinates of the corresponding analysis video are (cux, cuy):

[0076]

[0077]

[0078] After obtaining the block coordinates of the analysis path video corresponding to the current block of the multiplex path video, the relevant pre-encoding information of the analysis path video is assigned to the corresponding data structure of the current frame, and the first spatial adaptive qp offset qpAqOffset is obtained through the encoding block coordinates analysis[cuxy] The second spatial adaptive qp offset qpAqOffSet is calculated analysis[cuxy] As follows:

[0079] qpAqOffset reuse[cuXY] = qpAqOffset analysis[cuxy]

[0080] In addition, there are:

[0081] cuXY = cuX + cuY * widCU, cuxy = cux + cuy * widcu

[0082] Where widCU represents the number of 8x8 blocks in the 1 / 2 down-sampling column direction of the multiplex path video, widCU represents the number of 8x8 blocks in the 1 / 2 down-sampling column direction of the analysis path video, and other data structure assignments are consistent with the spatial adaptive qp offset.

[0083] After determining the correspondence between the encoding block coordinates of the multiplex path video and the encoding block coordinates of the analysis path video according to the scaling ratio information, it further includes: based on the data storage structure of the first intra optimal encoding cost and the first inter optimal encoding cost, obtaining the corresponding second intra optimal encoding cost and the second inter optimal encoding cost, and correcting the second intra optimal encoding cost according to the resolution of the multiplex path video and the resolution of the analysis path video.

[0084] For the multiplex path video, the intra optimal encoding cost intraCost[i], the inter optimal encoding cost interCost[i] and the list information need to be calculated from the selected multiplex IowresCost according to the calculation formula of the above encoding cost information. And the calculated intraCost needs to be corrected as follows:

[0085]

[0086] It can be understood that when the analysis road video resolution is less than the multiplex road video, the 8x8 block pixel block thereof will cover more texture areas, at this time, the intra coding is more inclined to generate larger residual error, so the multiplex intra optimal coding cost intraCost is often larger than the intra optimal coding cost intraCost calculated by the multiplex road video itself. When the intraCost of an 8x8 block of the analysis road video is small, it is highly probable that the pixel block is simple in texture, and the corresponding intra optimal coding cost intraCost of the multiplex road video is also small. In this way, different intra optimal coding costs intraCost are selected in different cases to realize accurate correction of the pre-coding information.

[0087] In addition, the first motion vector information is scaled and corrected according to the scaling ratio information and the set correction coefficient to obtain second motion vector information. The multiplex of the motion vector information Mvs also needs to be scaled and corrected according to the scaling ratio:

[0088] Mvs reuse ·x=Mvs analysis ·x*scaledRatioX+0.5

[0089] Mvs reuse ·y=Mvs analysis ·y*scaledRatioY+0.5

[0090] Wherein, 0.5 represents a correction coefficient, Mvs analysis ·x represents the first motion vector information in the X direction, Mvs reuse ·x represents the second motion vector information in the X direction, Mvs analysis ·y represents the first motion vector information in the Y direction, Mvs reuse ·y represents the second motion vector information in the Y direction.

[0091] Based on the second pre-coding information obtained through the above conversion, the encoding tree analysis is performed using the same, and then the information is used as the pre-coding information of the corresponding video frame of the multiplex road video to assist the formal encoding of the video frame.

[0092] Up to now, by multiplexing the analysis road pre-coding information, the multiplex road pre-coding process can be simplified, and the encoding speed of the multi-channel video stream can be accelerated. Taking the transcoding of a 1080p video stream as an example, according to actual tests, the performance of the final algorithm on the server is that when the input is a 1080p video stream, and the 360p, 540p, 720p and 1080p four-channel HEVC streams are transcoded, compared with the original transcoding framework, the encoding fps can be improved by about 19% under the condition of bdrate loss of 3.6%, so that the encoding efficiency is greatly improved while the encoding quality is guaranteed, and the real-time requirement of video encoding in the video live scene is met.

[0093] The above, by decoding and scaling processing based on the input source video stream, a plurality of different resolution video sequences are obtained, a video sequence with a specified resolution is selected as an analysis path video, and the rest of the video sequences with different resolutions are selected as multiplex path videos; the analysis path video is pre-encoded to generate first pre-encoding information for formal encoding of the analysis path video; and the first pre-encoding information is converted into second pre-encoding information according to the resolution of the multiplex path video and the resolution of the analysis path video for formal encoding of the corresponding multiplex path video. By using the above technical means, the pre-encoding information of the analysis path video is converted according to the resolution conversion, and the converted pre-encoding information is used for formal encoding of the multiplex path video. In this way, the pre-encoding process of the multiplex path video can be omitted, the coding quality of the multiplex path video is guaranteed, the coding efficiency of the multiplex path video is improved, and the coding acceleration effect of the multiple video streams is realized.

[0094] On the basis of the above embodiment, Figure 4 The structure diagram of a multiple video stream coding system based on pre-encoding information multiplexing provided by the present application is shown in the figure. Figure 4 The multiple video stream coding system based on pre-encoding information multiplexing provided by the present embodiment specifically includes a scaling module 21, a pre-encoding module 22, and a multiplexing module 23.

[0095] The scaling module 21 is configured to decode and scale the input source video stream to obtain a plurality of video sequences with different resolutions, select a video sequence with a specified resolution as an analysis path video, and select the rest of the video sequences with different resolutions as multiplex path videos.

[0096] The pre-encoding module 22 is configured to pre-encode the analysis path video to generate first pre-encoding information for formal encoding of the analysis path video.

[0097] The multiplexing module 23 is configured to convert the first pre-encoding information into second pre-encoding information according to the resolution of the multiplex path video and the resolution of the analysis path video for formal encoding of the corresponding multiplex path video.

[0098] Specifically, before converting the first pre-encoding information into the second pre-encoding information according to the resolution of the multiplex path video and the resolution of the analysis path video, the following steps are included:

[0099] The sequence number of the current to-be-encoded video frame of the multiplex path video is determined, and the first pre-encoding information corresponding to the sequence number of the analysis path video is extracted to generate the second pre-encoding information of the current to-be-encoded video frame.

[0100] The pre-encoding of the analysis path video includes:

[0101] The analysis path video is frame-controlled for pre-encoding in sequence number order.

[0102] The first pre-encoding information is converted into second pre-encoding information according to the resolution of the multiplexed video and the resolution of the analysis video, including:

[0103] The second pre-encoding information conversion of the video frame corresponding to the second sequence number information of the multiplexed video is triggered according to the first sequence number information of the video frame of the analysis video which has completed pre-encoding.

[0104] Specifically, the first pre-encoding information is converted into second pre-encoding information according to the resolution of the multiplexed video and the resolution of the analysis video, including:

[0105] The scaling ratio information is calculated based on the resolution of the multiplexed video and the resolution of the analysis video;

[0106] The second pre-encoding information is calculated according to the scaling ratio information and the first pre-encoding information, the first pre-encoding information including first spatial adaptive qp offset, first intra optimal coding cost, first inter optimal coding cost and first motion vector information, and the second pre-encoding information including second spatial adaptive qp offset, second intra optimal coding cost, second inter optimal coding cost and second motion vector information.

[0107] The second pre-encoding information is calculated according to the scaling ratio information and the first pre-encoding information, including:

[0108] The corresponding relationship between the coding block coordinates of the multiplexed video and the coding block coordinates of the analysis video is determined according to the scaling ratio information;

[0109] The second spatial adaptive qp offset is calculated based on the corresponding relationship and the first spatial adaptive qp offset.

[0110] After the corresponding relationship between the coding block coordinates of the multiplexed video and the coding block coordinates of the analysis video is determined according to the scaling ratio information, further including:

[0111] The data storage structure of the first intra optimal coding cost and the first inter optimal coding cost is parsed based on the corresponding relationship to obtain the corresponding second intra optimal coding cost and second inter optimal coding cost, and the second intra optimal coding cost is corrected according to the resolution of the multiplexed video and the resolution of the analysis video.

[0112] The second pre-encoding information is calculated according to the scaling ratio information and the first pre-encoding information, including:

[0113] The first motion vector information is scaled and corrected to obtain the second motion vector information according to the scaling ratio information and a set correction coefficient.

[0114] The above, by decoding and scaling processing based on the input source video stream, a plurality of different resolution video sequences are obtained, a video sequence with a specified resolution is selected as an analysis path video, and the rest of the resolution video sequences are multiplexed path videos; the analysis path video is pre-encoded to generate first pre-encoding information for formal encoding of the analysis path video; and the first pre-encoding information is converted into second pre-encoding information according to the resolution of the multiplexed path video and the resolution of the analysis path video, which is used for formal encoding of the corresponding multiplexed path video. By using the above technical means, the pre-encoding information of the analysis path video is converted according to the resolution conversion, and the converted pre-encoding information is used for formal encoding of the multiplexed path video. In this way, the pre-encoding process of the multiplexed path video can be omitted, the coding quality of the multiplexed path video is guaranteed, the coding efficiency of the multiplexed path video is improved, and the coding acceleration effect of the multi-path video stream is realized.

[0115] The multi-path video stream coding system based on pre-encoding information multiplexing provided by the embodiments of the application can be configured to execute the multi-path video stream coding method based on pre-encoding information multiplexing provided by the above embodiments, and has corresponding functions and advantages.

[0116] Based on the above actual examples, the embodiments of the application further provide a multi-path video stream coding device based on pre-encoding information multiplexing, which is described with reference to Figure 5 The multi-path video stream coding device based on pre-encoding information multiplexing includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory, as a computer readable storage medium, can be configured to store software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the multi-path video stream coding method based on pre-encoding information multiplexing (for example, a scaling module in the multi-path video stream coding system based on pre-encoding information multiplexing) described in any embodiment of the application. The communication module is configured to perform data transmission. The processor executes the software programs, instructions, and modules stored in the memory, thereby performing various functional applications and data processing of the device, that is, implementing the multi-path video stream coding method based on pre-encoding information multiplexing. The input device can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device can include a display device such as a display screen. The multi-path video stream coding device based on pre-encoding information multiplexing provided above can be configured to execute the multi-path video stream coding method based on pre-encoding information multiplexing provided by the above embodiments, and has corresponding functions and advantages.

[0117] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores computer executable instructions, the computer executable instructions are configured to execute a multi-channel video stream encoding method based on pre-coding information multiplexing when executed by a computer processor. The storage medium can be any various types of memory device or storage device. Of course, the computer executable instructions of the computer readable storage medium provided by the embodiments of the present application are not limited to the multi-channel video stream encoding method based on pre-coding information multiplexing as described above, but can also execute the related operations in the multi-channel video stream encoding method based on pre-coding information multiplexing provided by any embodiments of the present application.

[0118] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a computer program product, the technical solutions of the present application essentially or say the part of the contribution to the prior art or the whole or part of the technical solutions can be embodied in the form of software product, the computer program product is stored in a storage medium, including a plurality of instructions to make a computer device, mobile terminal or processor execute the whole or part of the steps of the multi-channel video stream encoding method based on pre-coding information multiplexing described in various embodiments of the present application.

Claims

1. A multi-channel video stream coding method based on precoding information multiplexing, characterized in that, include: Based on the input source video stream, decoding and scaling are performed to obtain multiple video sequences with different resolutions. The video sequence with a specified resolution is selected as the analysis path video, and the video sequences with other resolutions are multiplexed path video. The analysis path video is pre-encoded to generate first pre-encoded information, which is used for the formal encoding of the analysis path video; Calculate scaling information based on the resolution of the multiplexed video and the resolution of the analyzed video; The second precoding information is calculated based on the scaling information and the first precoding information for the formal encoding of the corresponding multiplexed video. The first precoding information includes a first spatial adaptive QP offset, a first intra-frame optimal coding cost, a first inter-frame optimal coding cost, and first motion vector information. The second precoding information includes a second spatial adaptive QP offset, a second intra-frame optimal coding cost, a second inter-frame optimal coding cost, and second motion vector information. The calculation includes scaling and correcting the first motion vector information according to the scaling information and a set correction coefficient to obtain the second motion vector information.

2. The multi-stream video coding method based on precoding information multiplexing according to claim 1, characterized in that, The step of calculating the second precoding information based on the scaling information and the first precoding information includes: The correspondence between the coded block coordinates of the multiplexed video and the coded block coordinates of the analyzed video is determined based on the scaling information. The second spatial adaptive qp offset is calculated based on the correspondence and the first spatial adaptive qp offset.

3. The multi-stream video coding method based on precoding information multiplexing according to claim 2, characterized in that, After determining the correspondence between the coded block coordinates of the multiplexed video and the coded block coordinates of the analyzed video based on the scaling information, the method further includes: Based on the correspondence, the data storage structure of the first intra-frame optimal coding cost and the first inter-frame optimal coding cost is analyzed to obtain the corresponding second intra-frame optimal coding cost and the second inter-frame optimal coding cost. The second intra-frame optimal coding cost is then corrected according to the resolution of the multiplexed video and the resolution of the analyzed video.

4. The multi-stream video coding method based on precoding information multiplexing according to claim 1, characterized in that, Before calculating the second precoding information based on the scaling information and the first precoding information, the process includes: The sequence number of the current video frame to be encoded in the multiplexed video is determined, and the first precoding information of the sequence number corresponding to the analysis video is extracted to generate the second precoding information of the current video frame to be encoded.

5. The multi-stream video coding method based on precoding information multiplexing according to claim 1, characterized in that, The precoding of the analysis path video includes: The video of the analysis path is subjected to frame-by-frame control, and pre-encoded frame by frame according to the sequence number order; The step of converting the first precoding information into second precoding information based on the resolution of the multiplexed path video and the resolution of the analyzed path video includes: Based on the first sequence number information of the video frame that has completed precoding in the analysis path video, the conversion of the second precoding information of the video frame corresponding to the second sequence number information in the multiplexed path video is triggered.

6. A multi-channel video stream coding system based on precoding information multiplexing, characterized in that, include: The scaling module is configured to perform decoding and scaling processing based on the input source video stream to obtain multiple video sequences with different resolutions. The video sequence with a specified resolution is selected as the analysis path video, and the video sequences with other resolutions are multiplexed path video. The precoding module is configured to precode the analysis path video to generate first precoding information for the formal encoding of the analysis path video; The multiplexing module is configured to calculate scaling information based on the resolution of the multiplexed video and the resolution of the analyzed video. The second precoding information is calculated based on the scaling information and the first precoding information for the formal encoding of the corresponding multiplexed video. The first precoding information includes a first spatial adaptive QP offset, a first intra-frame optimal coding cost, a first inter-frame optimal coding cost, and first motion vector information. The second precoding information includes a second spatial adaptive QP offset, a second intra-frame optimal coding cost, a second inter-frame optimal coding cost, and second motion vector information. The calculation includes scaling and correcting the first motion vector information according to the scaling information and a set correction coefficient to obtain the second motion vector information.

7. A multi-channel video stream coding device based on precoding information multiplexing, characterized in that, include: Memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-stream video coding method based on precoding information multiplexing as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a computer processor, are configured to perform the multi-stream video coding method based on precoding information multiplexing as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the multi-channel video stream coding method based on pre-coding information multiplexing as described in any one of claims 1-5.

Images