Live video image encoding method and apparatus, device, and medium

Abstract

The application relates to the field of network live broadcast, and discloses a live video image coding method and device, equipment and a medium. The method comprises the following steps: determining whether to enable a fast decision mode according to the coding cost difference between a current image block in a current video frame of a live stream and the periphery image block which has been coded; when the fast decision mode is enabled, the luminance component and the chroma component of the current image block are used to judge whether the current image block belongs to a simplified coding skip block in sequence; when the current image block belongs to the skip block, the current image block is coded as corresponding skip information in the live stream; when the current image block does not belong to the skip block, a prediction coding mode of the current image block is determined, the current image block is coded into the live stream according to the best prediction coding mode determined by the determination; and the coded live stream is transmitted to a live broadcast room for display. The application can accelerate the coding speed by reducing unnecessary coding calculation, meanwhile, the video quality is maintained, and the coding efficiency and quality of the live video image are improved.

Description

Technical Field

[0001] This application relates to live streaming technology, and more particularly to a live streaming video image encoding method, apparatus, device, and medium thereof. Background Technology

[0002] Live streaming platforms need to process massive amounts of video data in real time, and the encoding technology of this video data affects the efficient utilization of devices and network resources at each node of the live stream transmission path. Specifically in the field of live video image encoding, traditional encoding methods often face a trade-off between efficiency and quality. These methods typically require detailed encoding processing of each image block in the video frames within the live stream, including complex inter-frame prediction, transform, quantization, and entropy coding steps to ensure video quality. However, this comprehensive processing approach leads to high computational overhead and latency in scenarios with high real-time requirements, such as live streaming, affecting the smoothness of the live stream and the user experience.

[0003] The efficiency issues of traditional coding methods are particularly prominent when dealing with high dynamic range or complex scenes. Because each image block requires meticulous analysis and processing, the encoder must minimize the bitrate while maintaining video quality, which is a challenge in resource-constrained live streaming environments. Furthermore, traditional methods still perform the entire encoding process when processing image blocks with high similarity to reference frames, resulting in unnecessary computational waste and reduced coding efficiency.

[0004] To address these issues, one approach is to improve coding efficiency by introducing fast decision-making patterns. These patterns simplify the coding process and reduce the number of processing steps for image blocks, aiming to increase coding speed and reduce latency while maintaining acceptable video quality. However, these fast decision-making patterns often lack detailed analysis of image block content, leading to situations where it's difficult to accurately determine whether an image block truly needs simplified coding, or where excessive video quality is lost during simplification. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a live video image encoding method and corresponding apparatus, devices, and non-volatile readable storage medium.

[0006] According to one aspect of this application, a live video image encoding method is provided, comprising the following steps:

[0007] Based on the difference in encoding cost between the current image block in the current video frame of the live stream and its already encoded surrounding image blocks, determine whether to enable the fast decision mode;

[0008] When the fast decision mode is enabled, the current image block is progressively determined to be a simplified simplification block based on the luminance and chrominance components of the current image block. If it is a simplification block, the current image block is encoded as the corresponding simplification information in the live stream.

[0009] When the image block does not belong to a sectional block, the predictive coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the best predictive coding mode determined by the decision.

[0010] The encoded live stream is then transmitted to the live streaming room for display.

[0011] According to another aspect of this application, a live video image encoding apparatus is provided, comprising:

[0012] The mode enable module is set to determine whether to enable the fast decision mode based on the difference in encoding cost between the current image block in the current video frame of the live stream and its already encoded surrounding image blocks;

[0013] The simplification encoding module is configured to, when the fast decision mode is enabled, progressively determine whether the current image block belongs to a simplification block based on the luminance component and chrominance component of the current image block. If it belongs to a simplification block, the current image block is encoded into the corresponding simplification information in the live stream.

[0014] The default encoding module is configured to determine the predictive encoding mode of the current image block when it does not belong to a sectional block, and encode the current image block into the live stream according to the optimal predictive encoding mode determined by the decision;

[0015] The transmission and display module is configured to transmit the encoded live stream to the live broadcast room for display.

[0016] According to another aspect of this application, a live video image encoding device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the live video image encoding method described in this application.

[0017] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the live video image encoding method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0018] This application makes a series of improvements over traditional technologies, achieving a wealth of technical effects, specifically including but not limited to:

[0019] First, this application intelligently determines whether to enable the fast decision mode by comparing the coding cost difference between the current image block and its coded surrounding image blocks. This allows for the rapid identification of image blocks that are highly similar to their surrounding coded blocks, thereby avoiding unnecessary detailed coding processing of these blocks, saving computational resources, and reducing coding latency.

[0020] Secondly, when the fast decision-making mode is enabled, this application further determines whether an image block belongs to a simplifiable block based on its luminance and chrominance components. This allows for precise evaluation of the image block's content using quantization parameters and statistical characteristics, ensuring that only blocks truly requiring simplified coding are marked as simplifiable blocks. For image blocks that are not simplifiable blocks, the optimal predictive coding mode is still determined, and the image block is encoded into the live stream according to this mode, guaranteeing video quality. This ensures the operational efficiency of the encoding node devices and the real-time nature of the live stream transmission.

[0021] Furthermore, the live stream encoded using the technical solution of this application is transmitted to the live streaming room for display, ensuring real-time transmission and high-quality playback of the live content. In this way, this application not only improves encoding efficiency but also enhances the viewing experience of live video, demonstrating significant advantages, especially in environments with limited network bandwidth or computing resources. Attached Figure Description

[0022] Figure 1 This is an exemplary network architecture suitable for applying the live video image encoding method of this application;

[0023] Figure 2 This is a flowchart illustrating one embodiment of the live video image encoding method of this application;

[0024] Figure 3 This is a flowchart illustrating the process of determining whether to enter the rapid decision-making mode in an embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating the process of determining the coding cost index in an embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating the process of progressively determining whether the current image block is a sectional block in an embodiment of this application.

[0027] Figure 6 This is a schematic diagram of the process for determining whether the current image block is a sectional block based on the luminance component blocks in an embodiment of this application;

[0028] Figure 7 This is a schematic block diagram of the live video image encoding device of this application;

[0029] Figure 8This is a schematic diagram of the structure of a live video image encoding device used in this application. Detailed Implementation

[0030] Before detailing the specific embodiments of the technical solution of this application, we will first disclose the network architecture and application scenarios suitable for supporting the implementation of the technical solution of this application.

[0031] The technical solution of this application is applicable to the field of live video image encoding, and is particularly suitable for scenarios requiring high-efficiency encoding and real-time transmission. In this context, the technical solution of this application can be applied in a typical online live streaming platform, such as... Figure 1 As shown, the platform consists of a live streaming server 83, a media server 85, terminal devices 90 for broadcasters, and terminal devices 92 for viewers.

[0032] The live streaming server 83 is responsible for managing the creation of live streaming rooms and the reception and distribution of live streams. As the central node of the live stream, it receives the live stream from the broadcaster's user terminal device 90 and distributes it to the media server 85 and the viewer's user terminal device 92. The live streaming server 83 is also responsible for handling live streaming room management tasks, such as user authentication and live streaming room settings.

[0033] Media server 85 is responsible for encoding, transcoding, and storing the live stream. It performs necessary transcoding based on the viewer's network conditions and device capabilities to ensure the live stream is transmitted to the viewer in a suitable format and quality. Media server 85 is also responsible for storing the live stream for playback or other subsequent processing.

[0034] The broadcaster's terminal device 90 is the source of the live broadcast content. The broadcaster captures video through the terminal device 90 and sends the raw video stream to the live broadcast server 83. The broadcaster's terminal device 90 can be a smartphone, tablet, laptop, or professional camera equipment, which is connected to the live broadcast server 83 via the Internet.

[0035] The viewer's terminal device 92 is the receiving end of the live broadcast content. Viewers watch the live broadcast through terminal devices 92, which can be smartphones, tablets, personal computers, or smart TVs, etc. The viewer's terminal device 92 connects to the live broadcast server 83 via the Internet to receive and play the live broadcast stream.

[0036] The live stream transmission process is as follows: The broadcaster's terminal device 90 captures video and notifies the live streaming server 83. The live streaming server 83 distributes the live stream to the media server 85. The media server 85 encodes and transcodes the live stream, then sends the processed live stream's network address back to the live streaming server 83. The live streaming server 83 then transmits the live stream's network address to the viewer's terminal device 92, so that the terminal device 92 can load and play the live stream based on the network address.

[0037] The live video image encoding method of this application can be programmed into a computer program product, which can be selectively deployed on the broadcaster's terminal device 90 or media server 85. This program product includes the algorithms and logic required to implement the technical solution of this application, and can intelligently determine whether to enable a fast decision mode based on the encoding cost difference between the current image block in the current video frame of the live stream and its already encoded surrounding image blocks, and accordingly perform the judgment and encoding processing of the truncated block.

[0038] The technical solution of this application is applicable to protocols that encode based on image blocks, and is particularly suitable for scenarios that encode based on the AV1 protocol, because AV1 involves significant system overhead and time consumption in decision prediction mode. Through the technical solution of this application, encoding efficiency can be improved while maintaining video quality, reducing the transmission latency of live streams, and providing users with a smoother live streaming viewing experience.

[0039] Please see Figure 2 According to the live video image encoding method provided in this application, it can be implemented as a computer program product, installed and run on various node devices of a network live streaming platform, such as the terminal devices of broadcasters and media servers, to be responsible for the efficient encoding processing of video frames within the live stream. In some embodiments of the method, the following steps are included:

[0040] Step S3100: Based on the difference in encoding cost between the current image block in the current video frame of the live stream and its encoded surrounding image blocks, determine whether to enable the fast decision mode;

[0041] In the process of encoding live video images, the first step is to acquire the video frame data required for encoding the live stream. Video frame data is typically stored in YUV format, where Y represents the luminance component and UV represents the chrominance component. The luminance component contains most of the image's energy and has the greatest impact on human visual perception, while the chrominance component contains the image's color information.

[0042] During inter-frame prediction in the encoding process, the encoder uses previously encoded reference frames to predict the content of the current video frame. Reference frames are previously encoded video frames that can be compared with the current video frame to identify changes in the image. This prediction is based on the fact that only a small portion of the content typically changes between consecutive frames in a video, while most of the content remains unchanged. By comparing the current video frame and the reference frame, the encoder can determine motion vectors that describe the movement of objects or scenes in the image.

[0043] Before this step, the current video frame has been divided into multiple sub-blocks, a process called image block partitioning. Each image block is treated as an independent unit for processing. This is done to improve coding efficiency, as different regions of the video content may require different coding strategies and corresponding prediction modes to achieve optimal coding efficiency. For example, fast-moving regions may require smaller blocks to capture details, while stationary regions can use larger blocks to reduce the number of encoded bits.

[0044] Because live streaming demands strict real-time performance, encoders must process massive amounts of video data within a limited timeframe. Traditional encoding methods require detailed encoding of each image block, including complex inter-frame prediction, transform, quantization, and entropy coding steps. This leads to high computational overhead and latency in real-time demanding scenarios like live streaming, impacting the smoothness of the broadcast and user experience. Therefore, by introducing a fast decision mode, the encoder can quickly identify image blocks with high similarity to surrounding encoded blocks, avoiding unnecessary detailed encoding of these blocks. This saves computational resources, reduces encoding time, increases encoding speed, and ensures real-time transmission of live content, meeting the demands of efficient encoding and smooth playback in live streaming scenarios. To this end, the encoder acquires the current video frame and divides it into multiple image blocks. Before encoding each image block, it can use that block as the current image block and determine whether the fast decision mode is suitable by identifying the encoding cost difference between it and its surrounding image blocks.

[0045] To determine whether to enable the fast decision mode, the encoder compares the encoding cost of the current image patch with that of its coded neighboring image patches. This comparison can be achieved by calculating the statistical difference between the two, for example, by comparing their variance, mean, or other statistical characteristics. If the difference in encoding cost between the current image patch and its coded neighboring image patches is sufficiently small, it indicates that the current image patch may not require extensive encoding processing, and therefore the fast decision mode can be enabled to improve encoding efficiency. See the following embodiments for details:

[0046] In one embodiment, the encoder compares the coding cost differences between the current image block and its coded neighboring image blocks by calculating the Rate-Distortion Optimization (RDO) cost. In this process, the encoder first performs inter-frame prediction on the current image block to obtain a predicted block, and then calculates the prediction error. Based on the prediction error, the encoder estimates the number of encoded bits and reconstruction error under different coding modes, selecting the mode with the lowest RDO cost as a reference. If the RDO cost of the current image block is small compared to the RDO cost of its neighboring coded image blocks, such as below a preset threshold, it indicates that the current image block and its neighboring image blocks are similar in coding cost. Therefore, the encoder can enable a fast decision mode, reducing detailed coding processing steps.

[0047] In another embodiment, the encoder uses the Sartre Transform (SATD) as a metric to measure the difference in coding costs. For the current image block and its predicted block, the encoder calculates the SATD value of the residual block, a measure that considers the sum of the absolute differences of the transformed coefficients. By comparing the SATD values ​​of the current image block with those of its neighboring encoded image blocks, the encoder can assess the difference in coding costs between them. If the SATD value is low, such as below a preset threshold, it indicates that the difference between the current image block and its neighboring image blocks in the transform domain is small, and the encoder can consider the current image block suitable for using a fast decision mode.

[0048] In another embodiment, the encoder combines mean squared error (SSD) and transform coefficient energy to evaluate coding cost differences. The encoder first calculates the SSD value between the current image block and the predicted block, then performs a transform, such as a discrete cosine transform (DCT), on the residual block determined by comparing the current image block with the predicted block, and calculates the energy of the transform coefficients. By comparing the SSD and transform coefficient energy of the current image block with those of surrounding encoded image blocks, the encoder can more comprehensively evaluate coding cost differences. If these values ​​are all below a preset threshold, the encoder can determine that the current image block is suitable for a fast decision mode, thereby improving coding efficiency.

[0049] The positional relationship between the current image patch and its surrounding image patches is determined based on the video frame partitioning strategy. Typically, a video frame is divided into multiple coding blocks during encoding, each with its own position and index. The surrounding image patches refer to the image patches adjacent to the current image patch in the video frame, including image patches in each direction within the 3x3 grid centered on the current image patch. These surrounding image patches share edges with the current image patch; therefore, the difference in encoding cost between them can serve as an important basis for determining whether to enable the fast decision mode. During encoding, the encoding of each image patch can be processed in parallel using multiple threads. Accordingly, the encoder can use these positional relationships to obtain the encoding cost metrics of the already encoded surrounding image patches and compare them with the cost metrics of the current image patch to decide whether to use the fast decision mode for encoding the current image patch, thereby optimizing encoding efficiency and video quality.

[0050] Step S3200: When the fast decision mode is enabled, the current image block is progressively determined to be a simplified simplification block based on the luminance component and chrominance component of the current image block. If it is a simplification block, the current image block is encoded into the corresponding simplification information in the live stream.

[0051] After determining that the current image block is suitable for the fast decision mode, the encoder can progressively determine whether the current image block belongs to a simplifiable block based on its luminance and chrominance components. As the primary energy carrier of the image, the luminance component's coding decision has a significant impact on the overall coding efficiency and quality. Therefore, the encoder first evaluates the luminance component of the current image block.

[0052] There are two possible implementation methods for determining the luminance component: whole-block determination and segmented-block determination. In the whole-block determination embodiment, the encoder treats the current image block as a whole, directly performs statistical feature analysis on its luminance component, such as calculating the variance or mean, and compares these statistical features with a threshold set based on quantization parameters. If the statistical features of the luminance component are lower than the threshold, it indicates that the luminance variation of the current image block is not significant, and it may be similar to surrounding encoded image blocks, thus the current image block can be considered to belong to a segment block.

[0053] In contrast, in the segmented block determination embodiment, the encoder further subdivides the luminance component into multiple segments and performs statistical feature analysis on each segment similar to that in the previous embodiment. This method is suitable for regions with rich image details or complex motion, and can more finely identify changes in image blocks, thereby more accurately determining whether a segment is a segmented block. For each segment, if its statistical features are all below the quantization threshold, the entire luminance component is considered to belong to a segmented block.

[0054] If the current image block is determined not to be a segment block based on the luminance component, the process can proceed to step S3300. If it is determined to be a segment block, further determination of the chrominance component is required to confirm whether the current image block belongs to a segment block.

[0055] In determining the chroma components, the encoder treats the chroma components as a whole because chroma information typically contains less energy than the luminance component, and the human eye is less sensitive to chroma than to luminance. The encoder calculates the statistical characteristics of the chroma components. These statistical characteristics can be any statistical feature suitable for measuring the overall characteristics of the image, such as variance or mean, and compares them with a threshold mapped to the quantization parameters applied during encoding. If the statistical characteristics of the chroma components are below the threshold, it indicates that the chroma variation is small, and the current image block can be identified as a sectional block. Otherwise, the current layer block is a non-sectional block, and the process proceeds to step S3300.

[0056] Based on the combined luminance and chrominance component analysis, if the current image block is identified as a montage block, the encoder will simplify its encoding, recording only necessary montage information, such as montage markers and motion vectors, and encode it into the live stream, instead of performing full encoding. This simplified encoding method significantly reduces the number of encoding bits while maintaining video quality. Especially in scenarios with high real-time requirements, such as live streaming, it can significantly improve encoding efficiency and live stream transmission speed.

[0057] Step S3300: When the image block does not belong to a sectional block, determine the predictive coding mode of the current image block, and encode the current image block into the live stream according to the optimal predictive coding mode determined by the decision.

[0058] When step S3200 determines that the current image block does not belong to a sectional block, the predictive coding mode for the current image block is determined according to the preset standard procedure of the coding protocol. The selection of the predictive coding mode is based on the analysis of the content of the current image block, with the aim of finding the best predictive mode to minimize coding residuals, thereby optimizing coding efficiency and video quality.

[0059] When deciding on a predictive coding mode, the encoder considers multiple prediction modes, including but not limited to intra-frame prediction, inter-frame prediction, and combinations thereof. Intra-frame prediction modes utilize texture information within image patches for prediction, while inter-frame prediction modes are based on motion information of similar regions in a reference frame. The encoder evaluates the effectiveness of each prediction mode, typically by comparing the prediction residuals with the corresponding rate-distortion costs.

[0060] In a specific implementation, the encoder first performs inter-frame predictions for the current image block based on a reference frame, corresponding to various prediction modes. During this process, the encoder calculates residual blocks for each prediction mode and evaluates the coding cost of these residual blocks. For example, the encoder might use motion estimation techniques to determine the best matching region and calculate the corresponding motion vectors. Then, the encoder selects the optimal prediction mode based on the prediction results, the mode that produces the minimum prediction residuals and the lowest coding cost.

[0061] After determining the optimal prediction mode, the encoder identifies the corresponding residual information. This residual information reflects the difference between the current image block and the predicted block and is a key data point to focus on during subsequent encoding. Next, the encoder performs transform encoding on the residual information, including Discrete Cosine Transform (DCT) and quantization steps, to further compress the data and prepare it for entropy encoding.

[0062] Finally, the encoder encodes the quantized transform coefficients and, combined with the prediction mode information corresponding to the best prediction mode, writes the encoded result into the live stream. This step ensures that the encoded data can be correctly transmitted and decoded to reconstruct high-quality video content at the receiving end.

[0063] Step S3400: Transmit the encoded live stream to the live streaming room for display.

[0064] Depending on the device used for encoding—such as the broadcaster's terminal device or a media server—the encoder needs to transmit the encoded video stream to the live broadcast room for display. Specifically, the encoder needs to package and transmit the encoded data to ensure that the encoded video stream is effectively transmitted over the network to the viewer's terminal device and played and displayed in the live broadcast room's graphical user interface. The encoded video stream contains all the necessary information for reconstructing video frames at the decoding end, including transform coefficients, prediction modes, motion vectors, and montage information.

[0065] During transmission, the encoder organizes the data of the encoded blocks according to a specific format to form a video stream suitable for network transmission. This data includes transform coefficients, prediction modes, motion vectors, and delimited information. For image blocks marked as delimited blocks, the encoder only transmits simplified delimited information, such as delimited tags and motion vectors, without including the complete encoded data. This reduces the size of the video stream, thereby reducing network bandwidth usage and transmission latency.

[0066] At the decoding end, i.e., at the viewer's terminal device, the decoder receives the video stream and reconstructs the video frames based on the data transmitted by the encoder. For montage blocks, the decoder uses montage information, combined with motion vectors and reference frames, to reconstruct the content of the image block. For example, if an image block is marked as a montage block, the decoder will directly copy the corresponding block from the reference frame, or perform appropriate motion compensation by combining motion vectors to restore the original content of the image block.

[0067] This processing method allows the decoder to reduce computation and processing time without sacrificing video quality, especially when processing montage blocks, as these blocks do not require complex transformations and quantization. This approach is particularly important in live streaming scenarios because it can significantly improve the transmission efficiency and real-time playback of the video stream, providing users with a smooth viewing experience.

[0068] By implementing the above embodiments, this application effectively solves the technical problems existing in the prior art and can achieve significant technical advantages, including but not limited to:

[0069] First, this application intelligently determines whether to enable a fast decision mode by comparing the encoding cost difference between the current image patch and its coded neighboring image patches. This strategy enables the encoder to quickly identify image patches that are highly similar to their neighboring coded patches, thus avoiding unnecessary detailed encoding processing of these patches. This intelligent decision-making mechanism not only saves computing resources but also reduces encoding time and increases encoding speed, ensuring that live content can be transmitted in real time and meeting the needs of live streaming scenarios for efficient encoding and smooth playback.

[0070] Secondly, in the field of live video image coding, this application achieves a dual improvement in efficiency and quality through precise montage block determination and selection of the optimal predictive coding mode. This method enables efficient and reliable video coding in resource-constrained environments, such as mobile devices or scenarios with limited network bandwidth, providing users with a smoother live streaming viewing experience.

[0071] Furthermore, this application achieves a dual improvement in the efficiency and quality of live video image encoding by reducing unnecessary encoding calculations and increasing encoding speed while maintaining video quality. This translates into significant economies of scale for online live streaming platforms. In this way, while maintaining the efficiency and stability of live stream transmission, network resources are utilized to the maximum extent, providing users with a smoother, higher-quality live streaming experience and generating platform-level revenue.

[0072] Based on any embodiment of the method in this application, please refer to Figure 3Based on the difference in encoding cost between the current image block in the current video frame of the live stream and its already encoded surrounding image blocks, determine whether to enable the fast decision mode, including:

[0073] Step S3110: Determine the surrounding image blocks that are adjacent to the current image block position, and obtain the determined encoding cost indicators of multiple surrounding image blocks that have been encoded.

[0074] The encoder first determines the position of the current image patch within the video frame and identifies its directly adjacent neighboring image patches. These neighboring image patches can be horizontally or vertically adjacent to the current image patch, or they can be diagonally adjacent, depending on the video frame partitioning strategy and the encoder design. The encoder collects coding cost metrics for these neighboring image patches. These metrics can include, but are not limited to, any one or more of the following: the number of transform coefficients, the size of the quantization parameters, and the number of encoded bits, in order to measure the coding cost.

[0075] The coding cost metric of surrounding image patches can be obtained through pre-stored coding information, which has been calculated and recorded during the coding of surrounding image patches. For example, if a surrounding image patch generates fewer transform coefficients during coding, its coding cost metric will be relatively low; conversely, if it generates more transform coefficients, its coding cost metric will be relatively high.

[0076] Step S3120: Based on the position information of the surrounding image blocks in the video frame, match corresponding weights to the determined coding cost indicators of the multiple surrounding image blocks and fuse them into a cost measurement threshold.

[0077] Considering the impact of image patch position within a video frame on its encoding cost, this embodiment weights the encoding cost metrics of surrounding image patches to fuse them into a single cost measurement threshold. Specifically, the encoder assigns different weights to the encoding cost metrics of surrounding image patches based on their specific positions within the video frame. For example, image patches located at the edge of a video frame may differ from those in the center in terms of motion and texture, thus affecting the encoding cost differently. Image patches in the edge region may have a higher encoding cost due to the frame boundary, while image patches in the center region may have a lower encoding cost due to more repetitive textures.

[0078] To accurately reflect this positional influence, the encoder assigns a weight to the coding cost metric of each surrounding image patch. This weight can be preset based on the relative position of the image patch to the current image patch and their relative importance in the video frame. For example, surrounding image patches that share more edges with the current image patch may be given a higher weight because they have a greater impact on the coding decisions of the current image patch. The weights can be determined based on pre-defined rules or automatically adjusted using a learning algorithm based on historical coding data.

[0079] The encoder assigns weights to the coding cost metrics of all surrounding image patches, and then merges these weighted metrics to calculate a comprehensive cost measurement threshold. This threshold takes into account both the coding cost of surrounding image patches and their positional influence, providing a more accurate reference for whether to enable the fast decision mode for the current image patch.

[0080] In one embodiment, corresponding permissions can be determined for each surrounding image patch, and a cost measurement threshold can be determined accordingly. An exemplary formula for calculating the cost measurement threshold is as follows:

[0081]

[0082] Where: Threshold is a cost metric used to determine whether to enable the fast decision mode. N is the number of surrounding image patches. w i Cost is the weight of the i-th surrounding image patch, reflecting the degree of influence of this image patch on the coding cost of the current image patch. i It is the coding cost metric for the i-th surrounding image block, which can be the number of transform coefficients, the size of the quantization parameters, the number of coding bits, etc.

[0083] This formula comprehensively considers the encoding cost of surrounding image patches and their positional impact, providing a more accurate reference for whether to enable the fast decision mode for the current image patch. In this way, the encoder can more reasonably balance encoding efficiency and video quality, especially in real-time demanding scenarios such as live streaming. This accurate cost assessment is crucial for maintaining the smoothness and stability of the live stream.

[0084] In another embodiment, the average encoding cost index of all encoded surrounding image blocks can be calculated, and a preset weight can be assigned to this average value according to the position of the current image block in the video frame to obtain the corresponding cost measurement threshold. This simplifies the computational complexity while still reflecting the decision-making influence of surrounding image blocks on the current image block.

[0085] Step S3130: Determine the coding cost index of the current image block, compare the coding cost index of the current image block with the cost measurement threshold, and exit the fast decision mode if the cost cost index is greater than the cost measurement threshold; otherwise, enter the fast decision mode.

[0086] The decision to enable the fast decision mode for the current image patch is based on a comparison of the current image patch's coding cost metric with a cost measurement threshold. Specifically, the encoder first calculates the coding cost metric for the current image patch, which reflects the coding complexity and resource consumption of the current image patch.

[0087] After calculating the coding cost metric for the current image patch, the encoder compares it with a previously calculated cost metric threshold. The cost metric threshold is a reference value that comprehensively considers the coding costs of surrounding image patches and their location impact, providing a benchmark for coding decisions. If the coding cost metric for the current image patch is greater than the cost metric threshold, it means that the coding cost of the current image patch is relatively high, requiring more complex coding processing. Therefore, the encoder decides not to activate the fast decision mode and instead continues with the standard detailed coding process.

[0088] Conversely, if the coding cost metric of the current image patch is less than or equal to the cost metric threshold, the encoder considers the coding cost of the current image patch to be low and can process it using the fast decision mode. This mode allows the encoder to skip some detailed coding steps, thereby speeding up the coding process and reducing the consumption of computational resources. The fast decision mode is particularly suitable for image patches that are highly similar to surrounding already coded image patches, because they do not require additional detailed coding to achieve satisfactory coding results.

[0089] The unique advantage of this embodiment lies in its ability to provide an accurate reference for determining whether to enable the fast decision mode for the current image block by assigning different weights to different surrounding image blocks and comprehensively considering the encoding cost and positional impact of these blocks. This comprehensive evaluation allows the encoder to more reasonably balance encoding efficiency and video quality, optimizing the transmission efficiency of the live stream. Furthermore, by comparing the encoding cost index of the current image block with a cost measurement threshold, it further accurately identifies image blocks that require simplified encoding, thereby reducing the number of encoded bits and improving encoding speed while ensuring video quality.

[0090] Based on any embodiment of the method in this application, please refer to Figure 4 Determine the coding cost metrics for the current image patch, including:

[0091] Step S3131: Perform motion estimation on the current image block based on the reference frame to obtain the predicted block;

[0092] Motion estimation determines the best-matching region of the current image patch within a reference frame by analyzing the similarity between the current image patch and previously encoded reference frames. Specifically, the encoder searches for regions similar to the current image patch in the reference frame and calculates the corresponding motion vectors, which describe the displacement of the current image patch relative to the reference frame.

[0093] When performing motion estimation, the encoder can choose from various methods, such as full search, fast search, or block-based motion estimation. Full search searches all possible locations in the reference frame to find the best matching block, but it is computationally intensive. Fast search reduces computation and improves motion estimation speed by limiting the search range or using heuristic algorithms. Block-based motion estimation methods divide the image into multiple small blocks and perform motion estimation on each block separately, thereby improving coding efficiency.

[0094] A prediction block is obtained through motion estimation, generated based on the best-matching region found in the reference frame. The generation of the prediction block leverages the temporal correlation between consecutive frames in the video. Typically, only a small portion of the content changes between consecutive frames, while most of the content remains unchanged. Therefore, through motion estimation, the encoder can effectively utilize information from the reference frame, reducing the encoding complexity of the current image block.

[0095] Step S3132: Calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to the quantization parameters to obtain the transform coefficient matrix;

[0096] The residual block is the difference information between the current image block and the predicted block obtained through motion estimation. It can be determined by subtracting the predicted block from the current image block. The residual block reflects the deviation between the current image block and the corresponding region in the predicted frame. This deviation information is the data that needs to be focused on during the encoding process.

[0097] After determining the residual block, a Discrete Cosine Transform (DCT) is performed on the residual block to transform it from the spatial domain to the frequency domain, revealing the frequency components of the image data. After the DCT transformation, the encoder quantizes the coefficients obtained by the transformation according to preset quantization parameters to obtain the transform coefficient matrix.

[0098] Quantization reduces the number of encoded bits by decreasing the precision of the transform coefficients, thus impacting video quality. Quantization parameters can be preset or dynamically generated. These parameters directly affect coding efficiency and video quality; higher quantization parameters lead to greater information loss, while lower parameters retain more detail but increase the number of encoded bits.

[0099] Step S3133: Calculate the sum of squares of the non-zero coefficients in the transform coefficient matrix as the transform coefficient energy, which serves as the coding cost index for the current image block.

[0100] After quantization, the encoder calculates the sum of squares of the non-zero coefficients in the transform coefficient matrix. This value is called the transform coefficient energy, which serves as a metric for the coding cost of the current image patch. The transform coefficient energy reflects the complexity of the residual block in the transform domain. The more non-zero coefficients and the larger the sum of squares, the more detailed information the residual block contains, requiring more bits for encoding. This metric provides the encoder with a reference for measuring the coding cost of the current image patch, helping it decide whether to enable the fast decision mode for that patch.

[0101] It's easy to understand that the encoding cost of each surrounding image block of the current image block is also determined based on the above process. In this way, the encoder can rationally allocate encoding resources according to the content complexity of the image block, optimizing encoding efficiency and video quality.

[0102] This embodiment achieves dual optimization of coding efficiency and video quality through accurate motion estimation and residual block processing, and its technical advantages are obvious, including but not limited to:

[0103] First, by utilizing motion estimation techniques, the encoder can accurately capture the temporal correlation between consecutive frames in a video. By finding the best matching region in a reference frame, it can effectively predict the content of the current image block. This process not only reduces coding complexity but also improves prediction accuracy, especially when there is rapid motion or complex scene changes in the video content.

[0104] Secondly, by performing discrete cosine transform and quantization on the residual blocks, the encoder can convert data in the spatial domain to the frequency domain and precisely control the number of encoded bits according to the quantization parameters. This allows the encoder to dynamically adjust the quantization parameters based on the actual complexity of the video content, achieving a reasonable allocation of encoding resources. For example, in areas where the video content changes little, a higher quantization parameter can be used to reduce the number of encoded bits; while in areas rich in detail, a lower quantization parameter is used to retain more detailed information.

[0105] Finally, by calculating the transform coefficient energy as a coding cost metric, the encoder can quantify the coding cost of each image block and decide whether to enable the fast decision mode. This decision mechanism allows the encoder to quickly identify image blocks that are highly similar to surrounding already coded blocks, avoiding unnecessary detailed coding processing, thereby saving computational resources and reducing coding latency.

[0106] Based on any embodiment of the method in this application, please refer to Figure 5Based on the luminance and chrominance components of the current image block, it is progressively determined whether the current image block belongs to a simplified simplification block. If it belongs to a simplification block, the current image block is encoded into the corresponding simplification information in the live stream, including:

[0107] Step S3210: Divide the current image block into multiple small blocks based on the brightness component of the current image block, and determine whether the current image block belongs to a simplified encoding block based on the multiple small blocks;

[0108] The luminance component, as the main energy carrier of an image, contains most of the visual information; therefore, its encoding decision has a significant impact on the overall encoding efficiency and quality. In specific implementation, the encoder first divides the luminance component of the current image block into multiple smaller blocks. For example, the luminance component of a large image block, such as a 64x64 block, can be evenly divided into several 4x4 or 8x8 smaller blocks, and each smaller block will be independently processed for subsequent encoding.

[0109] In one embodiment, the encoder performs statistical feature analysis on small blocks of each luminance component. Common statistical features include variance and mean, which reflect the degree of variation in pixel values ​​within a small block. By comparing these statistical features with a quantization threshold mapped to preset quantization parameters, the encoder can assess the encoding cost and complexity of each small block. If the statistical features of a small block are below the threshold, it indicates that the small block has little variation and may be similar to surrounding encoded small blocks; therefore, the small block can be considered a truncated block. Otherwise, if the statistical features of any small block exceed the threshold, the entire current image block is considered a non-truncated block.

[0110] Step S3220: When it is determined that the current image block belongs to a simplification block based on the luminance component, continue to determine whether the current image block belongs to a simplification block based on the overall chrominance component of the current image block.

[0111] When the current image block is determined to be a sectional block based on its luminance component, the next step is to comprehensively evaluate the chrominance component of the current image block to further confirm whether the image block can be simplified for encoding. The chrominance component typically contains the color information of the image; its impact on visual perception is relatively smaller than that of the luminance component, but it still plays an important role in image quality. Therefore, the encoder evaluates the chrominance component of the current image block to ensure the accuracy of the encoding decision.

[0112] In this process, the encoder first extracts the chroma component data of the current image block, typically represented in UV format, where U and V represent the blue and red components of the chroma, respectively. Since the energy of chroma information is usually lower than that of the luminance component, the encoder treats the chroma component as a whole, rather than further dividing it into smaller blocks. This approach simplifies computation and reduces encoding complexity.

[0113] Similar to step S3210, the encoder then calculates the statistical characteristics of the chroma components, such as variance and mean. These statistical characteristics reflect the degree of variation of the chroma components. If the variance or mean of the chroma components is lower than a preset quantization threshold, it indicates that the chroma variation is not significant, and the current image block may be similar in chroma to surrounding encoded image blocks. Therefore, the image block can be considered a sectional block; otherwise, it is considered a non-sectional block.

[0114] Step S3230: When it is determined that the current image block belongs to a montage block based on the chroma components, the current image block is encoded into the corresponding montage information in the live stream;

[0115] If the chroma component analysis confirms that the current image block belongs to a montage block, the encoder will perform simplified encoding. Simplified encoding typically only records necessary information, such as montage markers and motion vectors, without performing full encoding. Simplified image blocks can be quickly reconstructed at the decoder because they require less processing, thus reducing decoding latency and improving video playback responsiveness. In live streaming environments, this fast encoding and decoding capability is particularly important for providing a seamless viewing experience, especially under conditions of fluctuating network conditions or limited computing resources. This method can significantly reduce the number of encoded bits while maintaining video quality, particularly in real-time demanding scenarios such as live streaming, improving encoding efficiency and smoothness.

[0116] Step S3240: If the current image block does not belong to a sectional block, execute the step of deciding the predictive coding mode of the current image block and encoding the current image block into the live stream according to the best predictive coding mode determined by the decision.

[0117] If the analysis results of either the luminance component or the chrominance component of the current image block indicate that the image block does not belong to a sectional block, the encoder can jump to the decision stage, starting from step S3300, to determine the optimal predictive coding mode for the image block and then encode the current image block. Please refer to the description of step S3300 for details, which will not be repeated here.

[0118] This embodiment achieves unique and significant technical advantages by introducing a dual-layer mechanism of luminance and chrominance components to determine sectional blocks, including but not limited to:

[0119] First, by subdividing the luminance component and independently analyzing the statistical characteristics of each small block, fine-grained capture of image block changes is achieved. This subdivision mechanism allows the encoder to evaluate image content at a more granular level, thereby more accurately identifying sectional blocks that do not actually require detailed encoding. For example, even if the entire image block looks similar to the reference frame macroscopically, if the statistical characteristics of one small block exceed a preset threshold, indicating a significant change in that small block, the encoder will treat the entire image block as a non-sectional block, ensuring that no important details are missed.

[0120] Secondly, as long as any small block of the luminance component is determined to be a non-node block, the entire current image block is considered a non-node block. This approach effectively simplifies the decision-making process, reduces the computational burden caused by excessive subdivision, and avoids the need to perform chrominance component analysis on each small block separately, thereby improving coding efficiency. Especially when dealing with image blocks where luminance changes are significant but chrominance changes are small, it can make decisions quickly, saving valuable coding resources and time.

[0121] Overall, the advantage of implementing this montage block identification mechanism lies in its integration of analysis results from both luminance and chrominance components, providing a highly efficient and accurate method for montage block recognition. This method not only improves coding efficiency and reduces computational complexity but also ensures video quality, especially in real-time demanding scenarios such as live streaming, significantly enhancing video stream transmission speed and playback smoothness. Furthermore, by reducing unnecessary coding steps and bit counts, this method also helps save network bandwidth, optimize storage resources, and provide users with a smoother and higher-quality live streaming viewing experience.

[0122] Based on any embodiment of the method in this application, please refer to Figure 6 The current image block is divided into multiple smaller blocks based on its luminance components. Based on these smaller blocks, it is determined whether the current image block belongs to a simplified encoding block, including:

[0123] Step S3211: Determine the corresponding quantization threshold and threshold matrix based on the quantization parameters used to calculate the coding cost index, and divide the brightness component of the current image block into multiple small blocks;

[0124] Determining whether an image block belongs to a simplifiable block is crucial for improving coding efficiency. First, the encoder needs to determine the quantization threshold and threshold matrix based on the quantization parameters. These parameters and matrices are important bases for evaluating the coding cost of image blocks. The quantization parameter (QP) is a key factor controlling coding quality in video coding; it directly affects the quantization process of the transform coefficients, thus influencing the quality of the encoded image and the number of bits required.

[0125] Specifically, the encoder sets a quantization threshold based on preset quantization parameters. This threshold is used to evaluate the statistical characteristics of the luminance component of an image patch, such as variance or mean. These statistical characteristics reflect the degree of variation in pixel values ​​within the image patch and are important indicators for determining whether an image patch has high coding cost. The threshold matrix contains multiple thresholds corresponding to the quantization parameters and is used in the subsequent transform coefficient scoring process.

[0126] Regarding quantization thresholds, the encoder predefines a series of quantization thresholds based on the QP value. These thresholds reflect the level of statistical characteristics that image patches should achieve at different quantization levels. For example, a lower QP value corresponds to a higher quantization threshold, allowing more details to be preserved; while a higher QP value corresponds to a lower quantization threshold, resulting in more detail loss to reduce the number of encoded bits. The quantization threshold can be determined experimentally or dynamically adjusted based on historical encoded data and the characteristics of the video content.

[0127] Setting the threshold matrix is ​​more complex. It contains multiple thresholds corresponding to quantization parameters, used in the subsequent transform coefficient scoring process. The threshold matrix can be customized based on different features of the video content and coding objectives. For example, for fast-moving video content, the threshold matrix can be set with lower thresholds to capture more motion details; while for static or less detailed video content, the threshold matrix can be set with higher thresholds to reduce unnecessary encoding bits. Each element of the threshold matrix corresponds to a possible value of the transform coefficient. The encoder uses this matrix to determine whether the transform coefficient is large enough to warrant detailed encoding. The threshold matrix can also be optimized using machine learning methods based on extensive coding experiment data to achieve the best balance between coding efficiency and video quality.

[0128] Next, the encoder divides the luminance component of the current image patch into multiple smaller blocks, typically 4x4 or 8x8 pixels in size. This division allows the encoder to perform more granular analysis of the image patch, as different regions of the video content may require different encoding strategies. For example, a large image patch may contain multiple regions with different motion or texture variations; by dividing it into smaller blocks, the encoder can analyze each region independently, thus more accurately assessing the encoding cost of each region.

[0129] Step S3212: Determine the statistical characteristics of each small block of the brightness component of the current image block, and compare the statistical characteristics with the quantization threshold. If the statistical characteristics of each small block are all less than the quantization threshold, continue to the next step; otherwise, the current image block is a non-section block.

[0130] After subdividing the luminance components of the current image patch, the next step is to determine the statistical characteristics of these patches and compare them with a quantization threshold. Specifically, the encoder calculates the statistical characteristics of each luminance component patch. Commonly used statistical characteristics include variance and mean. Variance measures the distribution range of pixel values ​​within a patch, while the mean reflects the average level of pixel values. These statistical characteristics reflect the degree of variation in pixel values ​​within a patch and are important criteria for evaluating whether an image patch needs detailed encoding.

[0131] After calculating the statistical features of each block, the encoder compares these features with a quantization threshold. If the statistical features of all blocks are below the quantization threshold, it means that these blocks are not significantly different and are likely similar to surrounding already encoded blocks. Therefore, the entire current image block can be considered a simplification block. This comparison process essentially evaluates the encoding cost and complexity of the image block to determine whether simplification encoding is necessary. For example, if an image block is divided into four 4x4 blocks, the encoder calculates the variance and mean of each of the four blocks and compares them with the quantization threshold. Only when the variance and mean of all four blocks are below the quantization threshold is the image block classified as a simplification block. This mechanism ensures that only image blocks with minimal variation and similarity to surrounding already encoded blocks are simplified, thereby saving encoding resources and improving encoding efficiency.

[0132] Conversely, if the statistical characteristics of any small block exceed the quantization threshold, the entire image block is marked as a non-sectional block and requires more detailed encoding processing. This method improves the accuracy of encoding decisions and ensures that the encoding process is both efficient and maintains video quality.

[0133] Step S3213: Perform discrete cosine transform on the residual block of each small block, determine its transform coefficients according to the quantization parameters, compare and score the transform coefficients of each small block with the threshold matrix, and confirm that the current image block is a truncated block when the score is lower than the preset threshold; otherwise, the current image block is a non-truncated block.

[0134] To further confirm whether the current image block belongs to a sectional block, a Discrete Cosine Transform (DCT) can be performed on the residual block of each small block. The resulting transform coefficients are quantized according to the quantization parameters, and the transform coefficients that reflect the frequency components of pixel value changes within the small block are calculated to determine their contribution to the coding cost.

[0135] Specifically, the quantization parameter (QP) determines the quantization level of the transform coefficients, affecting the quality and number of bits of the encoded image. A lower QP value means a lower quantization threshold, allowing more detail to be retained; while a higher QP value means a higher quantization threshold, leading to more loss of detail, but reducing the number of bits required for encoding. Based on the quantization parameter, the encoder quantizes the transform coefficients to obtain the quantized transform coefficient matrix.

[0136] Next, the encoder compares and scores the transform coefficients of each quantized block against a threshold matrix. The threshold matrix contains multiple thresholds determined based on the quantization parameters, corresponding to transform coefficients of different frequency components. The encoder compares the transform coefficients of each block with the corresponding threshold in the threshold matrix and calculates a score. This score can be the sum of squares of the deviations between the transform coefficients and the thresholds, or a weighted score calculated based on a specific algorithm.

[0137] To obtain an overall score, the encoder aggregates the scores of all smaller blocks. The aggregation method can be a simple summation or a weighted summation based on the importance of the smaller blocks. For example, if an image patch is divided into four smaller blocks, each with a score, the encoder can add these four scores to obtain the overall score for the entire image patch. If this overall score is below a preset threshold, it indicates that the transform coefficients of the entire image patch are small, resulting in low encoding cost. Therefore, the image patch can be considered a segmented block and can be simplified for encoding. If the overall score is above the preset threshold, the image patch is considered not to be a segmented block and requires more detailed encoding processing.

[0138] This method provides the encoder with a clear decision-making basis through quantization of transformation coefficients and a threshold matrix-based scoring mechanism, enabling it to efficiently identify shard blocks, thereby optimizing the allocation of coding resources and improving coding efficiency. Especially in live streaming scenarios with high real-time requirements, this mechanism can significantly improve coding performance and user experience.

[0139] This embodiment achieves a significant improvement in coding efficiency and optimization of video quality through refined luminance component processing and sectional block determination mechanisms, and has a variety of technical advantages, including but not limited to:

[0140] First, by dividing the luminance component into small blocks and performing statistical feature analysis, the changes within image blocks can be accurately captured, thus more accurately assessing the coding cost and complexity of each small block. This allows the encoder to reduce unnecessary coding steps while maintaining video quality, especially when image blocks have small changes and are similar to surrounding already coded blocks. It can effectively identify sectional blocks, thereby saving coding resources and improving coding speed.

[0141] Secondly, by performing discrete cosine transform and quantization on the residual blocks of each small block, and employing a scoring mechanism based on a threshold matrix, the encoder can comprehensively consider the frequency components and quantization parameters of the image blocks, further optimizing the decision-making process for sectional blocks. This not only improves the accuracy of sectional block recognition but also enhances the encoder's adaptability to different video content and motion characteristics, making coding decisions more flexible and efficient.

[0142] Finally, the overall score is derived by aggregating the scores of all small blocks and compared with a preset threshold, which serves as the basis for determining whether an image block is a syntactic block. This aggregated scoring mechanism simplifies the decision-making process, reduces computational complexity, and ensures that the encoder can respond quickly in real-time demanding live streaming scenarios, providing a seamless viewing experience.

[0143] Based on any embodiment of the method in this application, determining whether the current image block belongs to a simplified coded simplification block based on the overall chroma components of the current image block includes:

[0144] Step S3221: Determine the corresponding quantization threshold based on the quantization parameters used to calculate the coding cost index;

[0145] The chromaticity component, comprising two parts, U and V, represents the blue and red information of the chromaticity, respectively, and describes the color variations of the image. When determining whether the current image patch belongs to a simplifiable coded block based on the chromaticity components, the quantization threshold must first be determined according to the quantization parameters.

[0146] As revealed earlier, the quantization threshold, set based on quantization parameters, is used to evaluate the statistical characteristics of chroma components, such as variance or mean. These statistical characteristics reflect the degree of variation in pixel values ​​within the chroma components and are important indicators for determining whether an image patch has high coding cost. Specifically, a lower QP value corresponds to a higher quantization threshold, allowing more details to be preserved; while a higher QP value corresponds to a lower quantization threshold, resulting in more detail loss to reduce the number of encoded bits. The quantization threshold can be determined experimentally or dynamically adjusted based on historical coding data and the characteristics of the video content.

[0147] Step S3222: Determine the statistical characteristics of the chroma components of the current image block, and compare the statistical characteristics with the quantization threshold. If the statistical characteristics are less than the quantization threshold, the current image block is a sectional block; otherwise, the current image block is a non-sectional block.

[0148] To determine whether the current image patch belongs to a simplifiable block, the encoder needs to perform a detailed statistical feature analysis of the chroma components. Specifically, the encoder first calculates the statistical characteristics of the chroma components, commonly including variance and mean. Variance measures the distribution range of pixel values ​​within the chroma component, while the mean reflects the average level of pixel values. These statistical characteristics can intuitively reflect the degree of variation in the chroma components and are important criteria for evaluating the encoding cost and complexity of image patches.

[0149] After calculating the statistical characteristics of the chrominance components, the encoder compares these characteristics with a pre-set quantization threshold. The quantization threshold, determined based on quantization parameters, reflects the level of statistical characteristics the chrominance components should achieve at a specific quantization level. If the statistical characteristics of the chrominance components are below the quantization threshold, it indicates that the chrominance variation is small, and the current image patch is chromatically similar to surrounding encoded image patches. Therefore, this image patch can be considered a simplification patch and can be simplified into encoding. This simplified encoding typically records only necessary information, such as simplification markers and motion vectors, without performing full encoding processing, thereby reducing the number of encoded bits and improving encoding speed.

[0150] Conversely, if the statistical characteristics of the chroma components exceed the quantization threshold, it means that the chroma variation is large, and the current image patch may contain important color information, requiring detailed encoding to preserve this information. In this case, the encoder marks the current image patch as a non-drilled patch and processes it according to the standard encoding process.

[0151] Through this judgment mechanism based on statistical features and quantization thresholds, the encoder can efficiently identify image blocks suitable for simplified encoding, optimize the allocation of encoding resources, improve encoding efficiency, and maintain video quality. This method is particularly suitable for live video streaming, where real-time performance and encoding efficiency are especially important, ensuring smooth transmission and high-quality playback of video content.

[0152] The technical advantage of this embodiment, which uses chroma components to make the final confirmation of whether a current image block belongs to a simplification block, lies in its ability to allow the encoder to accurately assess the degree of color change in the image block and make a decision on whether to perform simplified encoding accordingly. This method effectively identifies image blocks with minimal color change and similarity to surrounding encoded image blocks by calculating the statistical characteristics of the chroma components, such as variance and mean, and comparing them with a quantization threshold, thus enabling simplified encoding. This process not only reduces unnecessary encoding bits and improves encoding efficiency but also alleviates the computational burden on the encoder. Especially in live video streams with high real-time requirements, this optimization ensures smooth transmission and high-quality playback of video content while reducing network bandwidth demands and improving user experience. Furthermore, this method helps reduce storage space usage while maintaining video quality, allowing video content providers to transmit more content with limited resources, thus improving the overall economic efficiency and practicality of the video encoding solution.

[0153] Based on any embodiment of the method in this application, when the image block is not a sectional block, the predictive coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal predictive coding mode determined by the decision, including:

[0154] Step S3310: Perform inter-frame prediction of the current image block based on the reference frame, corresponding to various prediction modes, select the best prediction mode according to the prediction results, and determine the residual information corresponding to the best prediction mode.

[0155] For a current image patch that is determined not to belong to a sectional block, a detailed prediction mode decision, also known as predictive coding mode decision, is required. First, the encoder performs inter-frame prediction corresponding to various prediction modes for the current image patch based on a reference frame. Specifically, previously encoded frames (reference frames) can be used to predict the content of the current frame, with the aim of finding the best matching region to generate a prediction block. The selection of inter-frame prediction modes includes, but is not limited to, motion-compensated prediction, global motion prediction, and block-based prediction.

[0156] During inter-frame prediction, a wide variety of prediction modes are available. Taking the AV1 protocol as an example, as an efficient video coding standard, it may involve multiple predictive coding modes when making inter-frame prediction decisions to adapt to different video content and scenarios. Specifically, AV1 supports, but is not limited to, the following main prediction modes: Global Motion Prediction, Affine Motion Prediction, Overlapping Block Motion Compensation (OBMC), Wedge Prediction, Directional Prediction, and prediction modes based on block sizes. These modes can be used individually or in combination to adapt to complex motion and texture variations. AV1 is designed to allow the encoder to flexibly select the most suitable prediction mode based on the characteristics of the video content to achieve optimal coding efficiency and video quality. It is evident that the AV1 protocol specification supports a rich variety of predictive coding modes, covering the needs of most video coding scenarios. However, when encoding each image block in detail, it is necessary to select the best predictive coding mode corresponding to the image block. This decision-making mechanism can be implemented using the prediction mechanism inherent in the AV1 protocol specification. Other protocols that encode based on image blocks follow the same principle.

[0157] Specifically, according to the relevant protocol specifications, when performing inter-frame prediction, the encoder calculates residual blocks in multiple prediction modes and evaluates the coding cost of these residual blocks. For example, the motion-compensated prediction mode estimates motion vectors to find the region in the reference frame most similar to the current image block, generates a prediction block, and calculates the prediction error. This process may include using algorithms such as full search, diamond search, and three-step search to accurately estimate motion vectors.

[0158] Based on the prediction results, the encoder selects the optimal prediction mode, that is, the mode that produces the smallest prediction residual among all candidate prediction modes. After determining the optimal prediction mode, the encoder determines the corresponding residual information. This information reflects the difference between the current image patch and the best predicted patch, and is data that needs to be focused on in subsequent encoding processes.

[0159] Step S3320: Transform and encode the residual information to obtain quantized transformation coefficients;

[0160] After the residual information of the current image patch is determined according to the best prediction mode, the encoder will perform transform coding on the residual information to obtain the quantized transform coefficients. Transform coding is a key step in the video coding process. Its purpose is to transform the residual information from the spatial domain to the frequency domain. Commonly used and optional transform methods include Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT).

[0161] During transform coding, the encoder first applies a transform to the residual information to convert the correlation in the image data into frequency components, making the energy more concentrated and facilitating subsequent quantization steps. After the transform, the encoder quantizes the transformed coefficients according to preset quantization parameters. Quantization reduces the precision of the transform coefficients, thereby reducing the number of encoded bits. The number and distribution of non-zero coefficients in the quantized transform coefficients determine the coding efficiency.

[0162] Step S3330: Encode the quantized transformation coefficients and, in conjunction with the prediction mode information corresponding to the best prediction mode, write the encoding result into the live stream.

[0163] Once the residual information has been transformed and encoded, and the quantized transform coefficients are obtained, these coefficients can be encoded and combined with information from the best prediction mode to write the encoded result into the live stream. The quantized transform coefficients contain energy distribution information of image patches in the frequency domain. This information is crucial data in the encoding process because it directly affects encoding efficiency and image quality.

[0164] The encoder first performs entropy coding on the quantized transform coefficients to further compress the data and reduce the number of encoded bits. Entropy coding is a lossless compression technique that allocates code lengths based on the probability distribution of the transform coefficients, resulting in shorter codes for frequently occurring coefficients and longer codes for less frequent coefficients. Common entropy coding methods include Huffman coding and arithmetic coding.

[0165] During entropy coding, the encoder selects the most suitable coding scheme based on the importance and distribution of the transform coefficients. For example, the encoder assigns shorter codes to high-frequency coefficients with concentrated energy to ensure coding efficiency, while assigning longer codes to low-frequency coefficients with dispersed energy. Furthermore, the encoder records the location information of non-zero coefficients, as this information is crucial for reconstructing image patches at the decoding end.

[0166] Combining the information from the optimal prediction mode, the encoder writes the encoded transform coefficients along with the prediction mode information into the live stream. The prediction mode information includes the type of the selected prediction mode and related parameters, such as motion vectors or prediction direction. This information guides the decoder on how to use reference frames to reconstruct the current image patch, ensuring the continuity and consistency of the video content.

[0167] Finally, the encoder packages the encoded transform coefficients and prediction mode information to form a video stream suitable for network transmission. This video stream contains all the necessary encoded data and can be accurately parsed by the decoder at the receiving end to reconstruct the original video frames. In this way, while maintaining video quality, encoding efficiency is optimized, network bandwidth consumption is reduced, and the transmission efficiency and playback smoothness of live video streams are improved.

[0168] The live video image encoding method in this embodiment achieves a significant improvement in encoding efficiency and video quality through accurate inter-frame prediction and residual information processing, and has several technical advantages, including but not limited to:

[0169] First, by performing inter-frame predictions corresponding to multiple prediction modes on the current image patch and selecting the optimal prediction mode, prediction errors are effectively reduced, thereby lowering the number of encoded bits. This allows the system to adapt to complex changes in video content, finding the most suitable prediction mode for both fast-moving scenes and richly detailed textures, ensuring both encoding accuracy and efficiency.

[0170] Secondly, by performing transform coding and quantization on the residual information, the spatial correlation in the image data is converted into frequency components, making the energy more concentrated and facilitating subsequent quantization and coding steps. This transform coding not only improves coding efficiency but also helps maintain video quality, especially when dealing with high dynamic range and complex scenes, better preserving image details.

[0171] Furthermore, lossless compression of the quantized transform coefficients using entropy coding further reduces the number of encoded bits. The introduction of entropy coding makes the encoding process more efficient because it allocates the encoding length based on the probability of the transform coefficients occurring, optimizing the use of encoding resources.

[0172] Finally, the encoded transform coefficients and the information of the best prediction mode are written into the live stream, ensuring the integrity and decodeability of the video stream. This not only improves the transmission efficiency of the live video stream but also guarantees smooth playback, significantly enhancing the user experience, especially in environments with limited network bandwidth or computing resources.

[0173] Please see Figure 7According to one aspect of this application, a live video image encoding apparatus includes a mode activation module 3100, a truncation encoding module 3200, a default encoding module 3300, and a transmission and display module 3400. The mode activation module 3100 is configured to determine whether to enable a fast decision mode based on the encoding cost difference between the current image block in the current video frame of the live stream and its already encoded surrounding image blocks. The truncation encoding module 3200 is configured to, when the fast decision mode is enabled, progressively determine whether the current image block belongs to a truncation block that can be simplified for encoding based on the luminance and chrominance components of the current image block. If it belongs to a truncation block, the current image block is encoded into the corresponding truncation information in the live stream. The default encoding module 3300 is configured to, when it does not belong to a truncation block, determine the predictive encoding mode of the current image block and encode the current image block into the live stream according to the determined optimal predictive encoding mode. The transmission and display module 3400 is configured to transmit the encoded live stream to a live broadcast room for display.

[0174] Based on any embodiment of the device in this application, the mode activation module 3100 includes: a perimeter acquisition module, configured to determine the surrounding image blocks adjacent to the current image block position, and acquire the determined encoding cost indicators of multiple surrounding image blocks that have completed encoding; a value setting module, configured to match corresponding weights to the determined encoding cost indicators of the multiple surrounding image blocks according to the position information of the surrounding image blocks in the video frame, and fuse them into a cost measurement threshold; and a mode judgment module, configured to determine the encoding cost indicator of the current image block, compare the encoding cost indicator of the current image block with the cost measurement threshold, and exit the fast decision mode when it is greater than the cost measurement threshold, otherwise, enter the fast decision mode.

[0175] Based on any embodiment of the apparatus in this application, the mode determination module includes: a motion estimation module, configured to perform motion estimation on the current image block based on a reference frame to obtain a prediction block; a residual transformation module, configured to calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to quantization parameters to obtain a transform coefficient matrix; and an index calculation module, configured to calculate the sum of squares of the non-zero coefficients in the transform coefficient matrix as the transform coefficient energy, which is used as the coding cost index of the current image block.

[0176] Based on any embodiment of the device in this application, the simplification encoding module 3200 includes: a luminance judgment module, configured to divide the current image block into multiple small blocks based on the luminance component, and determine whether the current image block belongs to a simplifiable simplification block based on the multiple small blocks; a chrominance judgment module, configured to, when the current image block is determined to belong to a simplification block based on the luminance component, continue to determine whether the current image block belongs to a simplifiable simplification block based on the overall chrominance component of the current image block; a fast encoding module, configured to, when the current image block is determined to belong to a simplification block based on the chrominance component, encode the current image block into the corresponding simplification information in the live stream; and a standard encoding module, configured to, if the current image block does not belong to a simplification block, run the default encoding module 3300.

[0177] Based on any embodiment of the device in this application, the brightness judgment module includes: a segmentation definition module, configured to determine a corresponding quantization threshold and threshold matrix according to the quantization parameters used to calculate the coding cost index, and divide the brightness component of the current image block into multiple small blocks; a statistical comparison module, configured to determine the statistical characteristics of each small block of the brightness component of the current image block, compare the statistical characteristics with the quantization threshold, and if the statistical characteristics of each small block are all less than the quantization threshold, then continue to the next step; otherwise, the current image block is a non-recursive block; and a transform scoring module, configured to perform discrete cosine transform on the residual block of each small block, determine its transform coefficients according to the quantization parameters, compare the transform coefficients of each small block with the threshold matrix, and score the current image block if the score is lower than a preset threshold, otherwise, the current image block is a non-recursive block.

[0178] Based on any embodiment of the device in this application, the chroma determination module includes: a threshold determination module, configured to determine a corresponding quantization threshold according to the quantization parameters used to calculate the coding cost index; and a statistical analysis module, configured to determine the statistical characteristics of the chroma components of the current image block, compare the statistical characteristics with the quantization threshold, and if the statistical characteristics are less than the quantization threshold, then the current image block is a sectional block; otherwise, the current image block is a non-sectional block.

[0179] Based on any embodiment of the apparatus in this application, the default encoding module 3300 includes: a mode determination module, configured to perform inter-frame prediction of the current image block corresponding to multiple prediction modes based on a reference frame, select the best prediction mode according to the prediction result, and determine the residual information corresponding to the best prediction mode; a quantization transformation module, configured to perform transformation encoding on the residual information to obtain quantized transformation coefficients; and an optimal encoding module, configured to encode the quantized transformation coefficients and, in conjunction with the prediction mode information corresponding to the best prediction mode, write the encoding result into the live stream.

[0180] Another embodiment of this application also provides a live video image encoding device. For example... Figure 8 The diagram shows the internal structure of a live video image encoding device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a live video image encoding method.

[0181] The processor of this live video image encoding device provides computing and control capabilities, supporting the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the live video image encoding method of this application. The network interface of the device is used for communication with a terminal.

[0182] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the live video image encoding device to which the solution of this application is applied. A specific live video image encoding device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] In this embodiment, the processor is used to execute... Figure 7 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the live video image encoding device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0184] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the live video image encoding method of any embodiment of this application.

[0185] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0187] In summary, this application optimizes the coding decision-making process, reduces unnecessary calculations, and accelerates the coding speed. At the same time, through precise threshold block judgment and selection of the best predictive coding mode, it maintains video quality, achieving a dual improvement in the efficiency and quality of live video image coding. For online live streaming platforms, this can yield significant economies of scale.

Claims

1. A live video image encoding method, characterized in that, include: Based on the difference in encoding cost between the current image block and its already encoded surrounding image blocks in the current video frame of the live stream, it is determined whether to enable the fast decision mode. This includes: identifying the surrounding image blocks that are immediately adjacent to the current image block, obtaining the determined encoding cost indicators of multiple surrounding image blocks that have already been encoded; matching corresponding weights to the determined encoding cost indicators of the multiple surrounding image blocks based on the position information of the surrounding image blocks in the video frame, and merging them into a cost measurement threshold; determining the encoding cost indicator of the current image block, comparing the encoding cost indicator of the current image block with the cost measurement threshold, and exiting the fast decision mode if it is greater than the cost measurement threshold, otherwise entering the fast decision mode. When the fast decision mode is enabled, the current image block is progressively determined to be a simplified simplification block based on the luminance and chrominance components of the current image block. If it is a simplification block, the current image block is encoded as the corresponding simplification information in the live stream. When the image block does not belong to a sectional block, the predictive coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the best predictive coding mode determined by the decision. The encoded live stream is then transmitted to the live streaming room for display.

2. The live video image encoding method according to claim 1, characterized in that, Determine the coding cost metrics for the current image patch, including: Motion estimation is performed on the current image block based on the reference frame to obtain the predicted block; Calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to the quantization parameters to obtain the transform coefficient matrix; The sum of squares of the non-zero coefficients in the transform coefficient matrix is ​​calculated as the transform coefficient energy, which serves as the coding cost metric for the current image block.

3. The live video image encoding method according to claim 1, characterized in that, Based on the luminance and chrominance components of the current image block, it is progressively determined whether the current image block belongs to a simplified simplification block. If it belongs to a simplification block, the current image block is encoded into the corresponding simplification information in the live stream, including: The current image block is divided into multiple smaller blocks based on the brightness component of the current image block. Based on these smaller blocks, it is determined whether the current image block belongs to a simplified encoding block. When it is determined that the current image block belongs to a simplification block based on the luminance component, it is further determined whether the current image block belongs to a simplification block based on the overall chrominance component of the current image block. When the current image block is determined to belong to a montage block based on the chroma components, the current image block is encoded into the corresponding montage information in the live stream; If the current image block does not belong to a sectional block, the step of deciding the predictive coding mode of the current image block is performed, and the current image block is encoded into the live stream according to the best predictive coding mode determined by the decision.

4. The live video image encoding method according to claim 3, characterized in that, The current image block is divided into multiple smaller blocks based on its luminance components. Based on these smaller blocks, it is determined whether the current image block belongs to a simplified encoding block, including: Based on the quantization parameters used to calculate the coding cost index, the corresponding quantization threshold and threshold matrix are determined, and the brightness component of the current image block is divided into multiple small blocks. Determine the statistical characteristics of each small block of the brightness component of the current image block, compare the statistical characteristics with the quantization threshold, and if the statistical characteristics of each small block are all less than the quantization threshold, continue to the next step; otherwise, the current image block is a non-section block. A discrete cosine transform is performed on the residual block of each small block, and its transform coefficients are determined according to the quantization parameters. The transform coefficients of each small block are compared and scored with the threshold matrix. When the score is lower than the preset threshold, the current image block is confirmed as a truncated block; otherwise, the current image block is a non-truncated block.

5. The live video image encoding method according to claim 3, characterized in that, Determine whether the current image block belongs to a simplified coded block based on the overall chroma components of the current image block, including: Determine the corresponding quantization threshold based on the quantization parameters used to calculate the coding cost index; Determine the statistical characteristics of the chroma components of the current image block, and compare the statistical characteristics with the quantization threshold. If the statistical characteristics are less than the quantization threshold, the current image block is a sectional block; otherwise, the current image block is a non-sectional block.

6. The live video image encoding method according to any one of claims 1 to 5, characterized in that, When the image block does not belong to a sectional block, the predictive coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal predictive coding mode determined by the decision, including: Based on the reference frame, perform inter-frame predictions for the current image patch using various prediction modes, select the best prediction mode based on the prediction results, and determine the residual information corresponding to the best prediction mode. The residual information is transformed and encoded to obtain quantized transform coefficients; The quantized transformation coefficients are encoded and combined with the prediction mode information corresponding to the best prediction mode, and the encoded result is written into the live stream.

7. A live video image encoding device, characterized in that, include: The mode activation module is configured to determine whether to enable the fast decision mode based on the difference in encoding cost between the current image block and its encoded surrounding image blocks in the current video frame of the live stream. This includes: a surrounding image acquisition module, configured to identify the surrounding image blocks adjacent to the current image block and acquire the determined encoding cost indicators of multiple encoded surrounding image blocks; a value setting module, configured to match corresponding weights to the determined encoding cost indicators of the multiple surrounding image blocks based on the position information of the surrounding image blocks in the video frame, and fuse them into a cost measurement threshold; and a mode judgment module, configured to determine the encoding cost indicator of the current image block, compare the encoding cost indicator of the current image block with the cost measurement threshold, and exit the fast decision mode if the current image block's encoding cost indicator is greater than the cost measurement threshold; otherwise, enter the fast decision mode. The simplification encoding module is configured to, when the fast decision mode is enabled, progressively determine whether the current image block belongs to a simplification block based on the luminance component and chrominance component of the current image block. If it belongs to a simplification block, the current image block is encoded into the corresponding simplification information in the live stream. The default encoding module is configured to determine the predictive encoding mode of the current image block when it does not belong to a sectional block, and encode the current image block into the live stream according to the optimal predictive encoding mode determined by the decision; The transmission and display module is configured to transmit the encoded live stream to the live broadcast room for display.

8. The live video image encoding device according to claim 7, characterized in that, The pattern determination module includes: The motion estimation module is configured to perform motion estimation on the current image block based on a reference frame to obtain a predicted block. The residual transformation module is configured to calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to the quantization parameters to obtain the transformation coefficient matrix. The index calculation module is configured to calculate the sum of squares of the non-zero coefficients in the transform coefficient matrix as the transform coefficient energy, which is used as the coding cost index for the current image block.

9. A live video image encoding device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

Images