Live video image coding method and device, equipment and medium

Abstract

The invention relates to the field of network live broadcast, and discloses a live broadcast video image coding method and device, equipment and a medium, and the method comprises the steps: determining whether to start a rapid decision mode or not according to the coding cost difference between a current image block in a current video frame of a live broadcast stream and a coded peripheral image block; when the fast decision mode is started, progressively judging whether the current image block belongs to a link block capable of simplifying coding or not according to the brightness component and the chrominance component of the current image block, and when the current image block belongs to the link block, coding the current image block into corresponding link information in the live stream; when the current image block does not belong to the joint block, deciding a predictive coding mode of the current image block, and coding the current image block into the live stream according to the optimal predictive coding mode determined by the decision; and transmitting the coded live broadcast stream to a live broadcast room for display. According to the invention, the coding speed is accelerated by reducing unnecessary coding calculation, the video quality is maintained, and the coding efficiency and quality of the live video image are improved.

Description

Technical Field

[0001] The present application relates to network live broadcast technology, and in particular to a live video image encoding method and its device, equipment, and medium. Background Art

[0002] Online live streaming platforms need to process massive amounts of video data in real time, and the encoding technology of video data affects the efficient use of each node device and network resources in the entire live streaming transmission path. Specifically in the field of live video image encoding, traditional encoding methods often face the problem of balancing efficiency and quality. These methods usually require detailed encoding processing of each image block in the video frame in the live stream, including complex inter-frame prediction, transformation, quantization, and entropy coding steps to ensure video quality. However, this comprehensive processing method will result in high computing overhead and delay in scenarios with high real-time requirements such as live streaming, affecting the fluency of live streaming and user experience.

[0003] The efficiency of traditional encoding methods is particularly prominent when dealing with high dynamic range or complex scenes. Since each image block needs to be carefully analyzed and processed, the encoder must reduce the bit rate as much as possible while maintaining video quality, which is a challenge in a resource-constrained live broadcast environment. In addition, traditional methods still perform a full encoding process when processing image blocks with a high degree of similarity to the reference frame, which causes unnecessary computational waste and reduces encoding efficiency.

[0004] To address these issues, we can try to improve coding efficiency by introducing fast decision modes. These modes simplify the coding process and reduce the processing steps of image blocks, in order to increase coding speed and reduce latency while ensuring acceptable video quality. However, these fast decision modes often lack a detailed analysis of the content of image blocks, resulting in an inability to accurately determine whether an image block really needs to be simplified in coding in some cases, or excessive loss of video quality when simplifying the coding. Summary of the invention

[0005] The purpose of the present application is to solve the above-mentioned problems and to provide a live video image encoding method and its corresponding device, equipment, and non-volatile readable storage medium.

[0006] According to one aspect of the present application, a live video image encoding method is provided, comprising the following steps:

[0007] Determining whether to enable a fast decision mode according to a difference in encoding costs between a current image block and encoded surrounding image blocks in a current video frame of a live stream;

[0008] When the quick decision mode is enabled, whether the current image block belongs to an abbreviated block that can be simplified for encoding is progressively determined based on the brightness component and the chrominance component of the current image block, and when the current image block belongs to an abbreviated block, the current image block is encoded as corresponding abbreviated information in the live stream;

[0009] When the image block does not belong to the omitted block, a prediction coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal prediction coding mode determined by the decision;

[0010] The encoded live stream is transmitted to the live broadcast room for display.

[0011] According to another aspect of the present application, a live video image encoding device is provided, comprising:

[0012] a mode enabling module configured to determine whether to enable a fast decision mode based on a difference in encoding costs between a current image block and its encoded surrounding image blocks in a current video frame of a live stream;

[0013] an abbreviation encoding module, configured to, when the quick decision mode is enabled, progressively determine whether the current image block belongs to an abbreviation block that can be simplified in encoding based on the brightness component and the chrominance component of the current image block, and if the current image block belongs to an abbreviation block, encode the current image block into corresponding abbreviation information in the live stream;

[0014] A default encoding module is configured to decide a prediction encoding mode for a current image block when the image block does not belong to an abbreviated block, and encode the current image block into the live stream according to the best prediction encoding mode determined by the decision;

[0015] The transmission and display module is configured to transmit the encoded live stream to the live broadcast room for display.

[0016] According to another aspect of the present application, a live video image encoding device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the live video image encoding method described in the present application.

[0017] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the live video image encoding method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0018] Compared with the traditional technology, this application has made a series of improvements and achieved rich technical effects, including but not limited to:

[0019] First, by comparing the difference in coding costs between the current image block and its encoded surrounding image blocks, the present application intelligently determines whether to enable the quick decision mode, and can quickly identify image blocks that are highly similar to the surrounding encoded blocks, thereby avoiding unnecessary detailed coding processing for these blocks, saving computing resources, and reducing coding delays.

[0020] Secondly, when the fast decision mode is enabled, the present application further progressively determines whether the image block belongs to an abbreviated block that can be simplified in coding based on the brightness component and chrominance component of the current image block, thereby using quantization parameters and statistical features to accurately evaluate the content of the image block, ensuring that only those blocks that really need simplified coding will be marked as abbreviated blocks. For image blocks that do not belong to abbreviated blocks, the best prediction coding mode will still be decided, and the image block will be encoded into the live stream according to this mode, ensuring the video quality. This ensures the operating efficiency of the node device responsible for encoding and the real-time transmission of the live stream.

[0021] In addition, the live stream encoded by the technical solution of this application is transmitted to the live broadcast room for display, ensuring the real-time transmission and high-quality playback of the live content. In this way, this application not only improves the encoding efficiency, but also improves the viewing experience of live video, especially in an environment with limited network bandwidth or limited computing resources, showing significant advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is an exemplary network architecture suitable for applying the live video image encoding method of the present application;

[0023] Figure 2 A flowchart of an embodiment of a live video image encoding method of the present application;

[0024] Figure 3 A schematic diagram of a process for determining whether to enter a quick decision mode in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a process for determining a coding cost indicator in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a flow chart for progressively determining whether a current image block is an abbreviated block in an embodiment of the present application;

[0027] Figure 6 This is a flow chart of determining whether a current image block is an abbreviated block based on a small block of a brightness component in an embodiment of the present application;

[0028] Figure 7 This is a principle block diagram of the live video image encoding device of the present application;

[0029] Figure 8A schematic diagram of the structure of a live video image encoding device used in this application. DETAILED DESCRIPTION

[0030] Before introducing the specific embodiments of the technical solution of the present application in detail, a network architecture and application scenarios suitable for supporting the implementation of the technical solution of the present application are first disclosed.

[0031] The technical solution of this application is applicable to the field of live video image coding, and is particularly applicable to scenarios that require high-efficiency coding and real-time transmission. In this context, the technical solution of this application can be applied in a typical online live broadcast platform, such as Figure 1 As shown, the platform is composed of a live broadcast server 83, a media server 85, a terminal device 90 of the anchor user and a terminal device 92 of the audience user.

[0032] The live broadcast server 83 is responsible for managing the creation of the live broadcast room and the reception and distribution of the live broadcast stream. As the central node of the live broadcast stream, it receives the live broadcast stream from the anchor user terminal device 90 and distributes it to the media server 85 and the audience user terminal device 92. The live broadcast server 83 is also responsible for handling the management tasks of the live broadcast room, such as user authentication, live broadcast room settings, etc.

[0033] The media server 85 is responsible for encoding, transcoding and storing the live stream. It performs necessary transcoding processing on the live stream according to the network conditions and device capabilities of the audience users to ensure that the live stream can be transmitted to the audience in an appropriate format and quality. The media server 85 is also responsible for storing the live stream for playback or other subsequent processing.

[0034] The anchor user's terminal device 90 is the source of the live content. The anchor captures the video through the terminal device 90 and sends the original video stream to the live server 83. The anchor user's terminal device 90 can be a smart phone, tablet computer, laptop computer or professional camera equipment, which is connected to the live server 83 via the Internet.

[0035] The terminal device 92 of the audience user is the receiving end of the live content. The audience watches the live broadcast through the terminal device 92, which can be a smart phone, a tablet computer, a personal computer or a smart TV. The terminal device 92 of the audience user is connected to the live broadcast server 83 through the Internet, receives the live broadcast stream and plays it.

[0036] The transmission process of the live stream is as follows: the anchor user's terminal device 90 captures the video and notifies the live server 83, which distributes the live stream to the media server 85. The media server 85 encodes and transcodes the live stream, and then sends the network address of the processed live stream back to the live server 83, which then transmits the network address of the live stream to the viewer user's terminal device 92, so that the terminal device 92 can load and play the live stream according to the network address.

[0037] Programming according to the live video image encoding method of the present application can be implemented as a computer program product, and the computer program product of the present application can be selectively deployed in the anchor user's terminal device 90 or the media server 85. The program product contains the algorithm and logic required to implement the technical solution of the present application, and can intelligently determine whether to enable the fast decision mode based on the difference in encoding cost between the current image block and its encoded surrounding image blocks in the current video frame of the live stream, and perform judgment and encoding processing of the omitted blocks accordingly.

[0038] The technical solution of this application is applicable to protocols based on image blocks for encoding, and is particularly applicable to scenarios based on the AV1 protocol for encoding, because AV1 involves large system overhead and time consumption when making decisions on prediction modes. Through the technical solution of this application, it is possible to improve encoding efficiency while maintaining video quality, reduce transmission delays in live streaming, and provide users with a smoother live viewing experience.

[0039] See also Figure 2 According to a live video image encoding method provided by the present application, it can be implemented as a computer program product, which is installed and run in various node devices of the network live broadcast platform, such as the terminal device of the anchor user and the media server, to be responsible for implementing efficient encoding processing of video frames in the live broadcast stream. In some embodiments of the method, the following steps are included:

[0040] Step S3100: determining whether to enable the fast decision mode according to the difference in coding cost between the current image block and its encoded surrounding image blocks in the current video frame of the live stream;

[0041] In the process of live video image encoding, the video frame data required for live stream encoding must first be obtained. The video frame data usually exists in YUV format, where Y represents the brightness component and UV represents the chrominance component. The brightness component contains most of the energy of the image and has the greatest impact on the visual perception of the human eye, while the chrominance component contains the color information of the image.

[0042] During the inter-frame prediction process of the encoding process, the encoder uses previously encoded reference frames to predict the content of the current video frame. Reference frames are previously encoded video frames that can be compared with the current video frame to find changes in the image. This prediction is based on the fact that typically only a small amount of content changes between consecutive frames in a video, while most of the content remains the same. By comparing the current video frame to the reference frames, the encoder can determine motion vectors, which describe the movement of objects or scenes in the image.

[0043] Before executing this step, the current video frame has been divided into multiple sub-blocks, a process called image block partitioning. Each image block is treated as an independent unit for processing. The purpose of this is to improve coding efficiency, because different areas of video content may require different coding strategies and corresponding prediction modes to achieve optimal coding efficiency. For example, fast-moving areas may require smaller blocks to capture details, while static areas may use larger blocks to reduce the number of coding bits.

[0044] Since the live broadcast scene has strict requirements on real-time performance, the encoder must process a large amount of video data in a limited time. The traditional encoding method requires detailed encoding processing for each image block, including complex inter-frame prediction, transformation, quantization and entropy coding steps. This will lead to high computational overhead and delay in scenes with high real-time requirements such as live broadcast, affecting the fluency and user experience of live broadcast. Therefore, by introducing the fast decision mode, the encoder can quickly identify those image blocks with high similarity to the surrounding encoded blocks, avoid unnecessary detailed encoding processing for these blocks, thereby saving computing resources, reducing encoding time, and improving encoding speed, ensuring that the live broadcast content can be transmitted in real time, and meeting the requirements of the live broadcast scene for efficient encoding and smooth playback. To this end, the encoder obtains the current video frame and divides it into multiple image blocks. Before encoding each image block, it can use the image block as the current image block and identify whether it is suitable for the fast decision mode by identifying the difference in encoding cost between it and the surrounding image blocks.

[0045] To determine whether to enable the fast decision mode, the encoder compares the coding cost of the current image block with the encoded surrounding image blocks. This comparison can be achieved by calculating the statistical difference between the two, for example, by comparing their variance, mean or other statistical features. If the difference in coding cost between the current image block and the surrounding encoded image blocks is small enough, it indicates that the current image block may not require detailed coding processing, so the fast decision mode can be enabled to improve coding efficiency. For details, please refer to the following embodiments:

[0046] In one embodiment, the encoder compares the difference in coding costs between the current image block and the encoded surrounding image blocks by calculating the rate-distortion optimization (RDO) cost between them. In this process, the encoder first performs inter-frame prediction on the current image block to obtain a prediction block, and then calculates the prediction error. Based on the prediction error, the encoder estimates the number of coding bits and reconstruction error under different coding modes, and selects the mode with the lowest RDO cost as a reference. If the RDO cost of the current image block is slightly different from the RDO cost of the surrounding encoded image blocks, such as lower than a preset threshold, this indicates that the current image block and the surrounding image blocks are similar in coding cost, so the encoder can enable a fast decision mode to reduce detailed coding processing steps.

[0047] In another embodiment, the encoder uses the Sutter transform (SATD) as an indicator to measure the difference in coding costs. For the current image block and its prediction block, the encoder calculates the SATD value of the residual block, which is a measure that takes into account the sum of the absolute differences of the transformed coefficients. By comparing the SATD values ​​of the current image block and the surrounding encoded image blocks, the encoder can evaluate the difference in coding costs between them. If the SATD value is low, such as below a preset threshold, it indicates that the difference between the current image block and the surrounding image blocks in the transform domain is small, and the encoder can consider that the current image block is suitable for using the fast decision mode.

[0048] In another embodiment, the encoder combines the mean square error (SSD) and the energy of the transform coefficients to evaluate the coding cost difference. The encoder first calculates the SSD value between the current image block and the prediction block, and then transforms the residual block determined by comparing the current image block with the prediction block, such as discrete cosine transform (DCT), and calculates the energy of the transform coefficients. By comparing the SSD and transform coefficient energy of the current image block with the surrounding encoded image blocks, the encoder can more comprehensively evaluate the coding cost difference. If these values ​​are all below a preset threshold, the encoder can determine that the current image block is suitable for the fast decision mode, thereby improving coding efficiency.

[0049] The positional relationship between the current image block and its surrounding image blocks is determined based on the division strategy of the video frame. Usually, a video frame is divided into multiple coding blocks during encoding, and each coding block has its own position and index. The surrounding image blocks of the current image block refer to the image blocks adjacent to it in the video frame, including image blocks in all directions in the nine-square grid centered on the current image block. These surrounding image blocks share edges with the current image block, so the difference in encoding costs between them can be used as an important basis for determining whether to enable the fast decision mode. During the encoding process, the encoding of each image block can be multi-threaded and processed in parallel. Based on this, the encoder can use these positional relationships to obtain the encoding cost index of the surrounding image blocks that have been encoded, and compare them with the cost index of the current image block to decide whether to use the fast decision mode to encode the current image block, thereby optimizing the encoding efficiency and video quality.

[0050] Step S3200: when the quick decision mode is enabled, whether the current image block belongs to an abbreviated block that can be simplified for encoding is progressively determined based on the luminance component and the chrominance component of the current image block, and if the current image block belongs to an abbreviated block, the current image block is encoded as corresponding abbreviated information in the live stream;

[0051] After determining that the current image block is suitable for the fast decision mode, it is possible to progressively determine whether the current image block is an abbreviated block that can be simplified for encoding based on the brightness component and chrominance component of the current image block. The brightness component is the main energy carrier of the image, and its encoding decision has a significant impact on the overall encoding efficiency and quality. Therefore, the encoder first evaluates the brightness component of the current image block.

[0052] There are two optional implementation methods for judging the brightness component: whole block judgment and small block judgment. In the whole block judgment embodiment, the encoder regards the current image block as a whole, directly performs statistical feature analysis on its brightness component, such as calculating the variance or mean, and compares these statistical features with the threshold value set based on the quantization parameter. If the statistical feature of the brightness component is lower than the threshold, it indicates that the brightness of the current image block does not change much and may be similar to the surrounding encoded image blocks, so that the current image block can be considered to be an abbreviated block.

[0053] In contrast, in the embodiment of judging by small blocks, the encoder further subdivides the brightness component into a plurality of small blocks, and respectively performs statistical feature analysis similar to the previous embodiment on each small block. This method is suitable for areas with rich image details or complex motion, and can more finely identify changes in image blocks, thereby more accurately judging whether it is an abbreviated block. For each small block, if its statistical features are all lower than the quantization threshold, it is considered that the entire brightness component belongs to the abbreviated block.

[0054] When it is determined based on the luminance component that the current image block does not belong to an omitted block, the process may jump to step S3300 . When it is determined to be an omitted block, it is necessary to further determine based on the chrominance component whether the current image block belongs to an omitted block.

[0055] In judging the chrominance component, the encoder regards the chrominance component as a whole, because the chrominance information usually contains lower energy than the brightness component, and the human eye is less sensitive to chrominance than brightness. The encoder calculates the statistical characteristics of the chrominance component, which can be any statistical characteristics suitable for measuring the overall characteristics of the image, such as variance or mean, and compares it with the threshold mapped to the quantization parameter applied during encoding. If the statistical characteristics of the chrominance component are lower than the threshold, it indicates that the chrominance change is small, and the current image block can be confirmed as belonging to the abbreviated block. Otherwise, the current layer block belongs to the non-abbreviated block, and the execution of step S3300 is jumped.

[0056] Based on the judgment results of the brightness and chrominance components, if the current image block is confirmed to be an abbreviated block, the encoder will simplify the encoding and only record the necessary abbreviated information, such as abbreviated markers and motion vectors, and encode them into the live stream instead of performing a complete encoding process. This simplified encoding method greatly reduces the number of encoding bits while maintaining video quality, especially in scenarios with high real-time requirements such as live broadcasts, which can significantly improve encoding efficiency and the transmission speed of live streams.

[0057] Step S3300: When the image block does not belong to an abbreviated block, a prediction coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal prediction coding mode determined by the decision;

[0058] When it is determined in step S3200 that the current image block does not belong to the omitted block, the prediction coding mode of the current image block is decided according to the standard process preset by the coding protocol. The selection of the prediction coding mode is based on the analysis of the content of the current image block, and the purpose is to find the best prediction mode to minimize the coding residual, thereby optimizing the coding efficiency and video quality.

[0059] When deciding the prediction coding mode, the encoder considers multiple prediction modes, including but not limited to intra-frame prediction, inter-frame prediction, and their combination. Intra-frame prediction mode uses the texture information inside the image block for prediction, while inter-frame prediction mode predicts based on the motion information of similar areas in the reference frame. The encoder evaluates the effect of each prediction mode, usually by calculating the prediction residual and the corresponding rate-distortion cost for comparison.

[0060] In a specific implementation, the encoder first performs inter-frame prediction corresponding to multiple prediction modes on the current image block based on the reference frame. During this process, the encoder calculates the residual blocks under each prediction mode and evaluates the coding cost of these residual blocks. For example, the encoder may use motion estimation technology to determine the best matching area and calculate the corresponding motion vector. Then, the encoder selects the best prediction mode based on the prediction results, which can produce the smallest prediction residual and the lowest coding cost.

[0061] After determining the best prediction mode, the encoder determines the residual information corresponding to that mode. This residual information reflects the difference between the current image block and the predicted block, and is the data that needs to be focused on in the subsequent encoding process. Next, the encoder transforms the residual information, including discrete cosine transform (DCT) and quantization steps to further compress the data and prepare for entropy coding.

[0062] Finally, the encoder encodes the quantized transform coefficients and writes the encoded results into the live stream in combination with the prediction mode information corresponding to the best prediction mode. This step ensures that the encoded data can be correctly transmitted and decoded to reconstruct high-quality video content at the receiving end.

[0063] Step S3400: Transmit the encoded live stream to the live broadcast room for display.

[0064] Depending on the device where the encoding is performed, such as the anchor user's terminal device or the media server, the corresponding encoder needs to transmit the encoded video stream to the live broadcast room for display. Specifically, the encoder needs to package and transmit the encoded data to ensure that the encoded video stream is effectively transmitted to the audience user's terminal device as a live stream through the network and played and displayed in the live broadcast room's graphical user interface. The encoded video stream contains all the necessary information for reconstructing the video frame at the decoding end, including transform coefficients, prediction mode, motion vectors, and abbreviation information.

[0065] During the transmission process, the encoder organizes the data of the coding block in a specific format to form a video stream suitable for network transmission. These data include transform coefficients, prediction modes, motion vectors, and abbreviation information. For image blocks marked as abbreviation blocks, the encoder only transmits simplified abbreviation information, such as abbreviation marks and motion vectors, without including complete coding data. In this way, the size of the video stream can be reduced, thereby reducing network bandwidth usage and transmission delay.

[0066] At the decoding end, that is, at the terminal device of the viewer user, after receiving the video stream, the decoder reconstructs the video frame based on the data transmitted by the encoder. For the abbreviated block, the decoder will use the abbreviated information, combined with the motion vector and the reference frame, to reconstruct the content of the image block. For example, if an image block is marked as an abbreviated block, the decoder will directly copy the corresponding block from the reference frame, or perform appropriate motion compensation in combination with the motion vector to restore the original content of the image block.

[0067] This processing method allows the decoder to reduce the amount of calculation and processing time without sacrificing video quality, especially when processing abbreviated blocks, because abbreviated blocks do not require complex transformation and quantization. This method is particularly important in live broadcast scenarios because it can significantly improve the transmission efficiency and real-time performance of video streams, providing users with a smooth viewing experience.

[0068] By implementing the above embodiments, the present application effectively solves the technical problems existing in the prior art and can achieve significant technical advantages, including but not limited to:

[0069] First, this application intelligently determines whether to enable the fast decision mode by comparing the difference in encoding cost between the current image block and the encoded surrounding image blocks. This strategy enables the encoder to quickly identify those image blocks that are highly similar to the surrounding encoded blocks, thereby avoiding unnecessary detailed encoding processing for these blocks. This intelligent decision-making mechanism not only saves computing resources, but also reduces encoding time, improves encoding speed, ensures that live content can be transmitted in real time, and meets the needs of live broadcast scenarios for efficient encoding and smooth playback.

[0070] Secondly, in the field of live video image coding, this application achieves a dual improvement in live video image coding efficiency and quality through accurate abbreviation block judgment and selection of the best prediction coding mode. This method can achieve efficient and reliable video coding in resource-limited environments, such as mobile devices or scenarios with limited network bandwidth, providing users with a smoother live viewing experience.

[0071] In addition, this application achieves a dual improvement in the efficiency and quality of live video image encoding by reducing unnecessary encoding calculations and increasing encoding speed while maintaining video quality, which can bring significant economies of scale to online live broadcast platforms. In this way, while maintaining the efficiency and stability of live streaming transmission, network resources can be maximized to provide users with a smoother and higher-quality live broadcast experience, bringing platform-level benefits.

[0072] Based on any embodiment of the method of this application, please refer to Figure 3, according to the difference in encoding cost between the current image block and its encoded surrounding image blocks in the current video frame of the live stream, determine whether to enable the fast decision mode, including:

[0073] Step S3110: determining surrounding image blocks that are immediately adjacent to the current image block, and obtaining determined coding cost indicators of a plurality of surrounding image blocks that have been coded;

[0074] The encoder first determines the position of the current image block in the video frame and identifies the surrounding image blocks that are directly adjacent to it. These surrounding image blocks can be image blocks that are horizontally or vertically adjacent to the current image block, or image blocks that are diagonally adjacent to the current image block, depending on the video frame division strategy and the design of the encoder. The encoder collects the coding cost indicators of these surrounding image blocks, which may include but are not limited to any one or more of the number of transform coefficients, the size of the quantization parameter, the number of coding bits, etc., in order to measure the coding cost.

[0075] The acquisition of the coding cost index of the surrounding image blocks can be achieved through pre-stored coding information, which has been calculated and recorded when the surrounding image blocks are coded. For example, if a surrounding image block generates fewer transform coefficients during coding, its coding cost index will be relatively low; conversely, if it generates more transform coefficients, its coding cost index will be relatively high.

[0076] Step S3120: matching corresponding weights to the determined coding cost indicators of the plurality of surrounding image blocks according to the position information of the surrounding image blocks in the video frame, and fusing them into a cost measurement threshold;

[0077] Taking into account the influence of the position of the image block in the video frame on its coding cost, the present embodiment performs weighted processing on the coding cost indicators of the surrounding image blocks to merge into a cost measurement threshold. Specifically, the encoder assigns different weights to the coding cost indicators of the surrounding image blocks according to their specific positions in the video frame. For example, the image blocks located at the edge of the video frame may differ from the image blocks in the central area in motion and texture, and therefore have different influences on the coding cost. The image blocks in the edge area may have a higher coding cost due to the influence of the frame boundary, while the image blocks in the central area may have a lower coding cost due to more repeated textures.

[0078] To accurately reflect this positional influence, the encoder assigns a weight to the coding cost indicator of each surrounding image block, which can be preset based on the relative position of the image block to the current image block and their relative importance in the video frame. For example, surrounding image blocks that share more edges with the current image block may be assigned higher weights because they have a greater impact on the coding decision of the current image block. The weights can be determined based on pre-set rules or automatically adjusted based on historical coding data through a learning algorithm.

[0079] Based on the weights assigned to the coding cost indicators of all surrounding image blocks, the encoder combines these weighted indicators to calculate a comprehensive cost measurement threshold. This threshold takes into account the coding costs of surrounding image blocks and their location influence, providing a more accurate reference for whether to enable the fast decision mode for the current image block.

[0080] In one embodiment, corresponding permissions and cost measurement thresholds may be determined for the surrounding image blocks respectively. Accordingly, an exemplary cost measurement threshold calculation formula is as follows:

[0081]

[0082] Where: Threshold is the cost measurement threshold used to decide whether to enable the fast decision mode. N is the number of surrounding image blocks. i is the weight of the ith surrounding image block, reflecting the influence of the image block on the coding cost of the current image block. i It is the coding cost indicator of the i-th surrounding image block, which can be the number of transform coefficients, the size of quantization parameters, the number of coding bits, etc.

[0083] This formula takes into account the coding cost of surrounding image blocks and their position, providing a more accurate reference for whether to enable the fast decision mode for the current image block. In this way, the encoder can more reasonably balance coding efficiency and video quality, especially in situations with high real-time requirements such as live broadcasting. This accurate cost assessment is crucial to maintaining the smoothness and stability of the live stream.

[0084] In another embodiment, the encoding cost indicators of all encoded surrounding image blocks can be averaged, and a preset weight can be assigned to the average value according to the position of the current image block in the video frame to obtain the corresponding cost measurement threshold. This can simplify the calculation complexity while still reflecting the impact of the surrounding image blocks on the decision-making of the current image block.

[0085] Step S3130, determine the coding cost index of the current image block, compare the coding cost index of the current image block with the cost measurement threshold, and when it is greater than the cost measurement threshold, exit the quick decision mode, otherwise, enter the quick decision mode.

[0086] The decision of whether to enable the fast decision mode for the current image block is based on the comparison between the coding cost index of the current image block and the cost measurement threshold. Specifically, the encoder first calculates the coding cost index of the current image block, which reflects the coding complexity and resource consumption of the current image block.

[0087] After calculating the coding cost index of the current image block, the encoder compares it with the previously calculated cost measurement threshold. The cost measurement threshold is a reference value that comprehensively considers the coding cost and position influence of surrounding image blocks, which provides a benchmark for coding decisions. If the coding cost index of the current image block is greater than the cost measurement threshold, it means that the coding cost of the current image block is relatively high and requires more complex coding processing, so the encoder decides not to enable the fast decision mode, but continue to perform the standard detailed coding process.

[0088] On the contrary, if the coding cost index of the current image block is less than or equal to the cost measurement threshold, the encoder believes that the coding cost of the current image block is low and can be processed in the fast decision mode. This mode allows the encoder to skip some detailed encoding steps, thereby speeding up the encoding speed and reducing the consumption of computing resources. The fast decision mode is particularly suitable for image blocks that are highly similar to the surrounding encoded image blocks, because they do not require additional detailed encoding to achieve satisfactory encoding results.

[0089] The unique advantage that can be achieved by this embodiment is that by matching different weights for different surrounding image blocks and comprehensively considering the coding cost and position influence of the surrounding image blocks, an accurate reference is provided for whether to enable the fast decision mode for the current image block. This comprehensive evaluation enables the encoder to more reasonably balance coding efficiency and video quality, and optimizes the transmission efficiency of the live stream. In addition, by comparing the coding cost index of the current image block with the cost measurement threshold, those image blocks that need to be simplified in coding can be further accurately identified, thereby reducing the number of coding bits and improving the coding speed while ensuring video quality.

[0090] Based on any embodiment of the method of this application, please refer to Figure 4 , determine the coding cost index of the current image block, including:

[0091] Step S3131, performing motion estimation on the current image block based on the reference frame to obtain a prediction block;

[0092] Motion estimation determines the best matching area of ​​the current block in the reference frame by analyzing the similarity between the current image block and the previously encoded reference frame. Specifically, the encoder searches for areas in the reference frame that are similar to the current image block and calculates the corresponding motion vectors, which describe the displacement of the current image block relative to the reference frame.

[0093] When implementing motion estimation, the encoder can choose from a variety of motion estimation methods, such as full search, fast search, or block-based motion estimation. The full search method searches all possible positions of the reference frame to find the best matching block, but the amount of calculation is large; the fast search method reduces the amount of calculation by limiting the search range or using heuristic algorithms to increase the speed of motion estimation. The block-based motion estimation method divides the image into multiple small blocks and performs motion estimation separately, thereby improving coding efficiency.

[0094] A prediction block is obtained through motion estimation. The prediction block is generated based on the best matching area found in the reference frame. The generation of the prediction block takes advantage of the temporal correlation between consecutive frames in the video. Usually, only a small part of the content changes between consecutive frames in the video, while most of the content remains unchanged. Therefore, through motion estimation, the encoder can effectively use the information of the reference frame and reduce the encoding complexity of the current image block.

[0095] Step S3132: Calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to the quantization parameter to obtain a transform coefficient matrix;

[0096] The residual block is the difference information between the current image block and the predicted block obtained through motion estimation. It can be determined by subtracting the predicted block from the current image block. The residual block reflects the deviation between the current image block and the corresponding area in the predicted frame. This deviation information is the data that needs to be focused on during the encoding process.

[0097] After the residual block is determined, a discrete cosine transform (DCT) is performed on the residual block to convert the residual block from the spatial domain to the frequency domain to reveal the frequency components of the image data. After the DCT transform, the encoder quantizes the transformed coefficients according to the preset quantization parameters to obtain a transform coefficient matrix.

[0098] The quantization process reduces the number of coded bits by reducing the precision of the transform coefficients, which also affects the video quality. The quantization parameter can be preset or dynamically generated. The quantization parameter directly affects the coding efficiency and video quality. A higher quantization parameter will result in more information loss, while a lower quantization parameter can retain more detailed information but increase the number of coded bits.

[0099] Step S3133: Calculate the square sum of non-zero coefficients in the transform coefficient matrix as the transform coefficient energy, which is used as the coding cost indicator of the current image block.

[0100] After quantization is completed, the encoder calculates the sum of the squares of the non-zero coefficients in the transform coefficient matrix. This value is called the transform coefficient energy, which serves as an indicator of the coding cost of the current image block. The transform coefficient energy reflects the complexity of the residual block in the transform domain. The more non-zero coefficients there are, the larger the sum of squares is, indicating that the residual block contains more detailed information and requires more bits for encoding. This indicator provides the encoder with a reference for measuring the coding cost of the current image block, which helps the encoder decide whether to enable the fast decision mode for the image block.

[0101] It is not difficult to understand that the coding cost index of each surrounding image block of the current image block is also determined according to the above process. In this way, the encoder can reasonably allocate coding resources according to the content complexity of the image block and optimize the coding efficiency and video quality.

[0102] This embodiment achieves dual optimization of coding efficiency and video quality through accurate motion estimation and residual block processing, and has obvious technical advantages, including but not limited to:

[0103] First, using motion estimation technology, the encoder can accurately capture the temporal correlation between consecutive frames in the video and effectively predict the content of the current image block by finding the best matching area in the reference frame. This process not only reduces the encoding complexity, but also improves the accuracy of the prediction, especially when there is fast motion or complex scene changes in the video content.

[0104] Secondly, by performing discrete cosine transform and quantization on the residual block, the encoder can convert the data in the spatial domain to the frequency domain and accurately control the number of coding bits according to the quantization parameter. This allows the encoder to dynamically adjust the quantization parameter according to the actual complexity of the video content and achieve a reasonable allocation of coding resources. For example, in areas where the video content changes less, a higher quantization parameter can be used to reduce the number of coding bits; while in areas with rich details, a lower quantization parameter is used to retain more detail information.

[0105] Finally, by calculating the transform coefficient energy as a coding cost indicator, the encoder can quantify the coding cost of each image block and decide whether to enable the fast decision mode based on this. This decision mechanism allows the encoder to quickly identify those image blocks that are highly similar to the surrounding coded blocks, avoiding unnecessary detailed coding processing, thereby saving computing resources and reducing coding delays.

[0106] Based on any embodiment of the method of this application, please refer to Figure 5, successively judging whether the current image block belongs to an abbreviated block that can be simplified for encoding according to the brightness component and the chrominance component of the current image block, and when it belongs to an abbreviated block, encoding the current image block into corresponding abbreviated information in the live stream, including:

[0107] Step S3210: divide the current image block into multiple small blocks based on the brightness component, and determine whether the current image block is an abbreviated block that can be simplified for encoding based on the multiple small blocks;

[0108] The luminance component, as the main energy carrier of the image, contains most of the visual information, so its encoding decision has a significant impact on the overall encoding efficiency and quality. In the specific implementation, the encoder first divides the luminance component of the current image block into multiple small blocks. For example, the luminance component of a larger image block such as 64x64 can be evenly divided into several 4x4 or 8x8 small blocks, and each small block will be independently processed for subsequent encoding.

[0109] In one embodiment, the encoder performs statistical feature analysis on the small blocks of each brightness component. Common statistical features include variance, mean, etc. These features can reflect the degree of change of pixel values ​​in the small blocks. By comparing these statistical features with the quantization threshold mapped to the preset quantization parameter, the encoder can evaluate the coding cost and complexity of each small block. If the statistical feature of a small block is lower than the threshold, it indicates that the small block has not changed much and may be similar to the surrounding encoded small blocks. Therefore, it can be considered that the small block belongs to the omitted block. Otherwise, as long as the statistical feature of a small block exceeds the threshold, the entire current image block is considered to be a non-omitted block.

[0110] Step S3220: when it is determined that the current image block belongs to an abbreviated block according to the luminance component, it is further determined whether the current image block belongs to an abbreviated block that can be simplified in coding based on the chrominance component of the current image block as a whole;

[0111] When the current image block is determined to be an abridged block based on the luminance component, the chrominance component of the current image block needs to be judged as a whole to further confirm whether the image block can be simplified for encoding. The chrominance component usually contains the color information of the image, and its impact on visual perception is relatively smaller than that of the luminance component, but it still plays an important role in image quality. Therefore, the encoder will evaluate the chrominance component of the current image block to ensure the accuracy of the encoding decision.

[0112] In this process, the encoder first extracts the chrominance component data of the current image block, which is usually expressed in UV format, where U and V represent the blue and red components of chrominance respectively. Since the energy of chrominance information is usually lower than that of brightness component, the encoder treats the chrominance component as a whole instead of further dividing it into small blocks. This processing method can simplify calculation and reduce encoding complexity.

[0113] Similar to step S3210, the encoder will then calculate the statistical features of the chrominance components, such as variance and mean. These statistical features can reflect the degree of change of the chrominance components. If the variance or mean of the chrominance components is lower than the preset quantization threshold, it indicates that the chrominance changes little, and the current image block may be similar to the surrounding encoded image blocks in chrominance, so it can be considered that the image block belongs to the abbreviated block, otherwise it is considered to be a non-abbreviated block.

[0114] Step S3230: when it is determined according to the chrominance component that the current image block belongs to an abbreviated block, the current image block is encoded as corresponding abbreviated information in the live stream;

[0115] If the current image block is confirmed to be an abbreviation block based on the judgment result of the chrominance component, the encoder will simplify the encoding. The simplified encoding process usually only records necessary information, such as abbreviation marks and motion vectors, without performing complete encoding processing. Simplified coded image blocks can be quickly reconstructed at the decoding end because they require less processing, which can reduce decoding delays and improve the responsiveness of video playback. In a live broadcast environment, this fast encoding and decoding capability is particularly important for providing a seamless viewing experience, especially when network conditions fluctuate or computing resources are limited. This method can significantly reduce the number of encoded bits while maintaining video quality, especially in scenarios with high real-time requirements such as live broadcasts, which can improve encoding efficiency and smoothness.

[0116] Step S3240: If the current image block does not belong to an omitted block, the step of deciding the prediction coding mode of the current image block and encoding the current image block into the live stream according to the optimal prediction coding mode determined by the decision is executed.

[0117] If any one of the analysis results of the brightness component and the chrominance component of the current image block indicates that the image block does not belong to the omitted block, the encoder can jump to the decision stage and start from step S3300 to determine the best prediction coding mode for the image block and then encode the current image block. Please refer to the description of step S3300 for details.

[0118] This embodiment introduces a dual-layer mechanism of luminance component and chrominance component to determine the omitted blocks, and achieves unique and significant technical advantages, including but not limited to:

[0119] First, by subdividing the brightness component and analyzing the statistical characteristics of each small block independently, the changes in the image blocks can be captured in a refined manner. This subdivision mechanism allows the encoder to evaluate the image content at a more detailed level, thereby more accurately identifying those omitted blocks that do not actually need to be encoded in detail. For example, even if the entire image block looks similar to the reference frame at a macro level, if the statistical characteristics of one of the small blocks exceed the preset threshold, indicating that the small block has changed significantly, the encoder will treat the entire image block as a non-omitted block to ensure that no important detail information is missed.

[0120] Secondly, as long as any small block of the luminance component is judged to be a non-ellipsis block, the entire current image block is considered to be a non-ellipsis block. This approach effectively simplifies the decision-making process, reduces the computational burden caused by over-segmentation, and avoids the need to perform chrominance component analysis on each small block separately, thereby improving coding efficiency. In particular, when processing image blocks with significant brightness changes but small chrominance changes, decisions can be made quickly, saving valuable coding resources and time.

[0121] In general, the advantage of implementing this omitted block judgment mechanism is that it combines the analysis results of the luminance component and the chrominance component, providing an efficient and accurate method for identifying omitted blocks. This method not only improves coding efficiency and reduces computational complexity, but also ensures video quality, especially in situations with high real-time requirements such as live broadcasts, which can significantly improve the transmission speed and playback smoothness of video streams. In addition, by reducing unnecessary encoding steps and bit counts, this method also helps save network bandwidth, optimize storage resources, and provide users with a smoother and higher-quality live viewing experience.

[0122] Based on any embodiment of the method of this application, please refer to Figure 6 , dividing the current image block into multiple small blocks based on the brightness component, and judging whether the current image block belongs to an abbreviated block that can be simplified in coding based on the multiple small blocks, including:

[0123] Step S3211, determining a corresponding quantization threshold and a threshold matrix according to the quantization parameter used to calculate the coding cost index, and dividing the brightness component of the current image block into a plurality of small blocks;

[0124] Determining whether an image block is an abbreviated block that can be simplified for coding is the key to improving coding efficiency. First, the encoder needs to determine the quantization threshold and threshold matrix based on the quantization parameter. These parameters and matrices are important bases for evaluating the coding cost of the image block. The quantization parameter (QP) is a key factor in controlling the coding quality in video coding. It directly affects the quantization process of the transform coefficients, and thus affects the quality of the encoded image and the number of bits required.

[0125] Specifically, the encoder sets a quantization threshold based on the preset quantization parameter. This quantization threshold is used to evaluate the statistical characteristics of the brightness component of the image block, such as variance or mean. These statistical characteristics can reflect the degree of change of pixel values ​​within the image block and are important indicators for determining whether the image block has a high coding cost. The threshold matrix contains multiple thresholds corresponding to the quantization parameter, which are used in the subsequent transform coefficient scoring process.

[0126] Regarding the quantization threshold, in the encoder, a series of quantization thresholds are predefined according to the QP value. These thresholds reflect the level that the statistical characteristics of the image block should reach at different quantization levels. For example, a lower QP value corresponds to a higher quantization threshold, allowing more details to be retained; while a higher QP value corresponds to a lower quantization threshold, resulting in more details being lost to reduce the number of encoded bits. The quantization threshold can be determined experimentally or dynamically adjusted based on historical encoding data and the characteristics of the video content.

[0127] The setting of the threshold matrix is ​​more complicated. It contains multiple thresholds corresponding to the quantization parameters for the subsequent transform coefficient scoring process. The threshold matrix can be customized according to the different characteristics of the video content and the encoding goals. For example, for fast-moving video content, the threshold matrix can set a lower threshold to capture more motion details; while for static or less detailed video content, the threshold matrix can set a higher threshold to reduce the number of unnecessary encoding bits. Each element of the threshold matrix corresponds to a possible value of the transform coefficient. The encoder will use this matrix to determine whether the transform coefficient is large enough to require detailed encoding. The setting of the threshold matrix can also be optimized through machine learning methods based on a large amount of encoding experimental data to achieve the best balance between coding efficiency and video quality.

[0128] Next, the encoder divides the brightness component of the current image block into multiple small blocks, which can be 4x4 or 8x8 pixels in size. This division allows the encoder to perform a more refined analysis of the image block, because different areas of the video content may require different encoding strategies. For example, a large image block may contain multiple areas with different motion or texture changes. By dividing it into small blocks, the encoder can analyze each area independently, thereby more accurately evaluating the encoding cost of each area.

[0129] Step S3212, determining the statistical characteristics of each small block of the brightness component of the current image block, comparing the statistical characteristics with the quantization threshold, if the statistical characteristics of each small block are less than the quantization threshold, proceeding to the next step, otherwise the current image block is a non-abbreviated block;

[0130] After subdividing the brightness component of the current image block, the next step is to determine the statistical characteristics of these small blocks and compare them with the quantization threshold. Specifically, the encoder calculates the statistical characteristics of each small block of the brightness component. Common statistical characteristics include variance and mean. Variance can measure the distribution range of pixel values ​​within a small block, while the mean reflects the average level of pixel values. These statistical characteristics can reflect the degree of change of pixel values ​​within a small block, and are an important basis for evaluating whether an image block needs detailed encoding.

[0131] After calculating the statistical features of each small block, the encoder compares these features with the quantization threshold. If the statistical features of all small blocks are lower than the quantization threshold, it means that these small blocks have not changed much and may be similar to the surrounding encoded small blocks, so the entire current image block can be considered to be an abbreviated block. This comparison process is actually evaluating the coding cost and complexity of the image block to determine whether the image block needs to be simplified. For example, if an image block is divided into four 4x4 small blocks, the encoder will calculate the variance and mean of these four small blocks respectively and compare them with the quantization threshold. Only when the variance and mean of all four small blocks are lower than the quantization threshold, the image block will be determined as an abbreviated block. Such a mechanism ensures that only those image blocks with small changes and similar to the surrounding encoded blocks will be simplified, thereby saving coding resources and improving coding efficiency.

[0132] On the contrary, if the statistical characteristics of any small block exceed the quantization threshold, the entire image block will be marked as a non-elliptical block and require more detailed encoding processing. This approach improves the accuracy of encoding decisions and ensures that the encoding process is both efficient and maintains video quality.

[0133] Step S3213, perform discrete cosine transform on the residual block of each small block, determine its transform coefficient according to the quantization parameter, compare and score the transform coefficient of each small block with the threshold matrix, and when the score is lower than the preset threshold, confirm that the current image block is an abbreviated block, otherwise the current image block is a non-abbreviated block.

[0134] To further confirm whether the current image block belongs to an omitted block, discrete cosine transform (DCT) can be performed on the residual block of each small block, and the obtained transform coefficients are quantized according to the quantization parameters. The transform coefficients of the frequency components reflecting the changes in pixel values ​​within the small block are calculated to determine their contribution to the coding cost.

[0135] Specifically, the quantization parameter (QP) determines the quantization level of the transform coefficients, affecting the quality and number of bits of the encoded image. A lower QP value means a lower quantization threshold, allowing more detail information to be retained; a higher QP value means a higher quantization threshold, resulting in more detail loss, but can reduce the number of encoded bits. Based on the quantization parameter, the encoder quantizes the transform coefficients to obtain a quantized transform coefficient matrix.

[0136] Next, the encoder compares the quantized transform coefficients of each small block with the threshold matrix for scoring. The threshold matrix contains multiple thresholds determined based on the quantization parameters, which correspond to the transform coefficients of different frequency components. The encoder compares the transform coefficients of each small block with the corresponding thresholds in the threshold matrix and calculates a score value. This score value can be the sum of the squares of the deviations between the transform coefficients and the threshold, or it can be a weighted score calculated based on a specific algorithm.

[0137] In order to obtain an overall score, the encoder aggregates the score values ​​of all small blocks. The aggregation method can be a simple summation or a weighted summation based on the importance of the small blocks. For example, if an image block is divided into four small blocks, each of which has a score value, the encoder can add up these four score values ​​to obtain the overall score of the entire image block. If this overall score is lower than the preset threshold, it indicates that the transformation coefficient of the entire image block is small and the encoding cost is low. Therefore, it can be considered that the image block belongs to an abbreviated block and can be simplified for encoding. If the overall score is higher than the preset threshold, it is considered that the image block does not belong to an abbreviated block and requires more detailed encoding processing.

[0138] This method provides the encoder with a clear decision-making basis through quantized transform coefficients and a scoring mechanism based on a threshold matrix, enabling it to efficiently identify omitted blocks, thereby optimizing the allocation of encoding resources and improving encoding efficiency. Especially in live broadcast scenarios with high real-time requirements, this mechanism can significantly improve encoding performance and user experience.

[0139] This embodiment achieves significant improvement in coding efficiency and optimization of video quality through refined luminance component processing and omitted block judgment mechanism, and has multiple technical advantages, including but not limited to:

[0140] First, by dividing the luminance component into small blocks and analyzing its statistical features, the changes within the image block can be accurately captured, thereby more accurately evaluating the coding cost and complexity of each small block. It allows the encoder to reduce unnecessary coding steps while maintaining video quality, especially when the image block changes little and is similar to the surrounding coded blocks, it can effectively identify the omitted blocks, thereby saving coding resources and increasing the coding speed.

[0141] Secondly, by performing discrete cosine transform and quantization on the residual block of each small block, as well as a scoring mechanism based on a threshold matrix, the encoder can comprehensively consider the frequency components and quantization parameters of the image block and further optimize the judgment process of the omitted block. This not only improves the accuracy of the omitted block recognition, but also enhances the encoder's adaptability to different video content and motion characteristics, making the encoding decision more flexible and efficient.

[0142] Finally, the overall score is obtained by aggregating the scores of all small blocks and comparing it with the preset threshold to determine whether the image block is an abbreviated block. This aggregate scoring mechanism simplifies the decision-making process and reduces the computational complexity, while ensuring that the encoder can respond quickly in live broadcast scenarios with high real-time requirements and provide a seamless viewing experience.

[0143] On the basis of any embodiment of the method of the present application, judging whether the current image block is an abbreviated block that can be simplified for encoding based on the chrominance component of the current image block as a whole includes:

[0144] Step S3221, determining a corresponding quantization threshold according to a quantization parameter used to calculate a coding cost index;

[0145] The chrominance component includes two components, U and V, which represent the blue and red information of the chrominance respectively, and they describe the color change of the image. When judging whether the current image block belongs to an abbreviated block that can be simplified for encoding based on the chrominance component, it is first necessary to determine the quantization threshold based on the quantization parameter.

[0146] As previously disclosed, the quantization threshold is set based on the quantization parameter and is used to evaluate the statistical characteristics of the chrominance component, such as variance or mean. These statistical characteristics can reflect the degree of change in the pixel values ​​within the chrominance component and are important indicators for determining whether an image block has a high coding cost. Specifically, a lower QP value corresponds to a higher quantization threshold, allowing more details to be retained; while a higher QP value corresponds to a lower quantization threshold, resulting in more details being lost to reduce the number of encoded bits. The setting of the quantization threshold can be determined experimentally or dynamically adjusted based on historical coding data and the characteristics of the video content.

[0147] Step S3222, determine the statistical characteristics of the chrominance component of the current image block, compare the statistical characteristics with the quantization threshold, if the statistical characteristics are less than the quantization threshold, the current image block is an abbreviated block, otherwise the current image block is a non-abbreviated block.

[0148] In order to determine whether the current image block is an abbreviated block that can be simplified for encoding, the encoder needs to perform a detailed statistical feature analysis of the chrominance component. Specifically, the encoder first calculates the statistical features of the chrominance component. Common statistical features include variance and mean. Variance measures the distribution range of pixel values ​​within the chrominance component, while the mean reflects the average level of pixel values. These statistical features can intuitively reflect the degree of change of the chrominance component and are an important basis for evaluating the coding cost and complexity of image blocks.

[0149] After calculating the statistical characteristics of the chrominance components, the encoder compares these characteristics with the pre-set quantization threshold. The quantization threshold is determined based on the quantization parameter and reflects the level that the statistical characteristics of the chrominance components should reach at a specific quantization level. If the statistical characteristics of the chrominance components are lower than the quantization threshold, this indicates that the chrominance changes little and the current image block is similar in chrominance to the surrounding encoded image blocks. Therefore, it can be considered that the image block belongs to an abbreviation block and can be simplified for encoding. This simplified encoding usually only records necessary information, such as abbreviation marks and motion vectors, without performing a complete encoding process, thereby reducing the number of encoding bits and increasing the encoding speed.

[0150] On the contrary, if the statistical characteristics of the chrominance component exceed the quantization threshold, it means that the chrominance changes greatly and the current image block may contain important color information, which needs to be encoded in detail to retain this information. In this case, the encoder marks the current image block as a non-abbreviated block and processes it according to the standard encoding process.

[0151] Through this judgment mechanism based on statistical features and quantization thresholds, the encoder can efficiently identify image blocks suitable for simplified encoding, optimize the allocation of encoding resources, improve encoding efficiency, and maintain video quality. This method is particularly suitable for live video streaming, where real-time performance and encoding efficiency are particularly important, and can ensure smooth transmission and high-quality playback of video content.

[0152] The technical advantage of this embodiment of finally confirming whether the current image block belongs to the abbreviated block based on the chrominance component is that it allows the encoder to accurately evaluate the degree of color change of the image block and make a decision on whether to simplify the encoding based on this. This method effectively identifies those image blocks with little color change and similar to the surrounding encoded image blocks by calculating the statistical characteristics of the chrominance component, such as variance and mean, and comparing them with the quantization threshold, so that simplified encoding can be performed. Such processing not only reduces the number of unnecessary coding bits and improves coding efficiency, but also reduces the computational burden of the encoder, especially in live video streams with high real-time requirements. This optimization can ensure smooth transmission and high-quality playback of video content, while reducing the demand for network bandwidth and improving user experience. In addition, this method also helps to reduce the storage space occupied while maintaining video quality, so that video content providers can transmit more content with limited resources, improving the economic benefits and practicality of the overall video encoding solution.

[0153] On the basis of any embodiment of the method of the present application, when it does not belong to the omitted block, deciding the prediction coding mode of the current image block, and encoding the current image block into the live stream according to the best prediction coding mode determined by the decision, including:

[0154] Step S3310: performing inter-frame prediction corresponding to multiple prediction modes on the current image block based on the reference frame, selecting the best prediction mode according to the prediction results, and determining residual information corresponding to the best prediction mode;

[0155] For the current image block that is determined not to be an omitted block, a detailed prediction mode, also known as a predictive coding mode decision, is required. First, the encoder performs inter-frame prediction corresponding to multiple prediction modes on the current image block based on the reference frame. Specifically, the previously encoded frame (reference frame) can be used to predict the content of the current frame, with the goal of finding the best matching area to generate a prediction block. The selection of inter-frame prediction modes includes but is not limited to motion compensation prediction, global motion prediction, block-based prediction, etc.

[0156] There are many optional prediction modes for inter-frame prediction. Taking the AV1 protocol as an example, as an efficient video coding standard, it may involve multiple prediction coding modes when making inter-frame prediction decisions to adapt to different video content and scenes. Specifically, AV1 supports but is not limited to the following main prediction modes: global motion prediction, affine motion prediction, overlapped block motion compensation (OBMC), wedge prediction, directional prediction, and block-based prediction modes of various sizes. These modes can be used alone or in combination to adapt to complex motion and texture changes. The design of AV1 allows the encoder to flexibly select the most appropriate prediction mode according to the characteristics of the video content to achieve the best coding efficiency and video quality. It can be seen that the AV1 protocol specification supports a wide variety of prediction coding modes that can cover the needs of most video coding scenarios, but when encoding each image block in detail, it is necessary to select the best one as the prediction coding mode corresponding to the image block. This decision-making mechanism can be implemented with the help of the prediction mechanism inherent in the AV1 protocol specification. The same is true for other protocols based on image block encoding.

[0157] Specifically, according to the corresponding protocol specifications, when performing inter-frame prediction, the encoder calculates the residual blocks under multiple prediction modes and evaluates the coding costs of these residual blocks. For example, the motion compensation prediction mode estimates the motion vector to find the area in the reference frame that is most similar to the current image block, generates a prediction block, and calculates the prediction error. This process can include using algorithms such as full search, diamond search, and three-step search to accurately estimate the motion vector.

[0158] According to the prediction results, the encoder selects the best prediction mode, that is, among all candidate prediction modes, the mode that can produce the smallest prediction residual is selected as the best prediction mode. After determining the best prediction mode, the encoder will determine the corresponding residual information, which reflects the difference between the current image block and the best prediction block and is the data that needs to be focused on in the subsequent encoding process.

[0159] Step S3320: transform-encode the residual information to obtain quantized transform coefficients;

[0160] After the current image block determines its residual information according to the best prediction mode, the encoder will transform the residual information to obtain the quantized transform coefficients. Transform coding is a key step in the video encoding process. Its purpose is to convert the residual information from the spatial domain to the frequency domain. Commonly used optional transform methods include discrete cosine transform (DCT) and discrete wavelet transform (DWT).

[0161] When transform coding is performed, the encoder first applies a transform to the residual information in order to convert the correlation in the image data into frequency components, making the energy more concentrated and convenient for the subsequent quantization step. After the transform, the encoder quantizes the transformed coefficients according to the preset quantization parameters, and reduces the precision of the transform coefficients through quantization, thereby reducing the number of encoding bits. The number and distribution of non-zero coefficients in the quantized transform coefficients will determine the efficiency of the encoding.

[0162] Step S3330: encode the quantized transform coefficients, and write the encoding results into the live stream in combination with the prediction mode information corresponding to the optimal prediction mode.

[0163] When the transform coding of the residual information is completed and the quantized transform coefficients are obtained, these coefficients can be encoded and combined with the information of the best prediction mode, and the encoding results are written into the live stream. The quantized transform coefficients contain the energy distribution information of the image block in the frequency domain, which is the key data in the encoding process because they directly affect the encoding efficiency and image quality.

[0164] The encoder first performs entropy coding on the quantized transform coefficients to further compress the data and reduce the number of encoded bits. Entropy coding is a lossless compression technique that allocates code lengths based on the probability distribution of transform coefficients, so that coefficients that appear frequently have shorter codes, while coefficients that appear less frequently have longer codes. Common optional entropy coding methods include Huffman coding and arithmetic coding.

[0165] During the entropy coding process, the encoder selects the most appropriate coding scheme based on the importance and distribution of the transform coefficients. For example, for high-frequency coefficients with concentrated energy, the encoder assigns shorter codes to ensure coding efficiency, while for low-frequency coefficients with dispersed energy, longer codes are assigned. In addition, the encoder also records the location information of those non-zero coefficients, because this information is crucial for the decoder to reconstruct the image block.

[0166] Combined with the information of the best prediction mode, the encoder writes the encoding results of the transform coefficients together with the prediction mode information into the live stream. The prediction mode information includes the type of the selected prediction mode and related parameters, such as motion vector or prediction direction. This information guides the decoder on how to use the reference frame to reconstruct the current image block, ensuring the continuity and consistency of the video content.

[0167] Finally, the encoder packages the encoded transform coefficients and prediction mode information to form a video stream suitable for network transmission. This video stream contains all the necessary encoded data and can be accurately parsed by the decoder at the receiving end to reconstruct the original video frame. In this way, while maintaining video quality, it is possible to optimize encoding efficiency, reduce network bandwidth usage, and improve the transmission efficiency and playback smoothness of live video streams.

[0168] The live video image encoding method of this embodiment achieves significant improvement in encoding efficiency and video quality through accurate inter-frame prediction and residual information processing, and has multiple technical advantages, including but not limited to:

[0169] First, by performing inter-frame prediction corresponding to multiple prediction modes on the current image block and selecting the best prediction mode, the prediction error is effectively reduced, thereby reducing the number of coding bits. This can adapt to the complex changes in video content, whether it is a fast-moving scene or a detailed texture, and can find the most appropriate prediction mode to ensure the accuracy and efficiency of coding.

[0170] Secondly, by transform coding and quantizing the residual information, the spatial correlation in the image data is converted into frequency components, making the energy more concentrated and convenient for subsequent quantization and encoding steps. This transform coding not only improves coding efficiency, but also helps maintain video quality, especially when dealing with high dynamic range and complex scenes, it can better preserve image details.

[0171] Furthermore, entropy coding is used to compress the quantized transform coefficients losslessly, further reducing the number of coding bits. The introduction of entropy coding makes the coding process more efficient because it allocates the code length according to the probability of occurrence of the transform coefficients, optimizing the use of coding resources.

[0172] Finally, the encoded transform coefficients and the information of the best prediction mode are written into the live stream, ensuring the integrity and decodability of the video stream. This not only improves the transmission efficiency of the live video stream, but also ensures the smoothness of the playback, which can significantly improve the user experience, especially in environments with limited network bandwidth or computing resources.

[0173] See also Figure 7According to one aspect of the present application, a live video image encoding device is provided, comprising a mode enabling module 3100, an abbreviated encoding module 3200, a default encoding module 3300, and a transmission display module 3400, wherein the mode enabling module 3100 is configured to determine whether to enable a fast decision mode according to a difference in encoding cost between a current image block and its encoded surrounding image blocks in a current video frame of a live stream; the abbreviated encoding module 3200 is configured to, when enabling the fast decision mode, progressively determine whether the current image block belongs to an abbreviated block that can be simplified in encoding according to the luminance component and the chrominance component of the current image block, and when it belongs to an abbreviated block, encode the current image block into the corresponding abbreviated information in the live stream; the default encoding module 3300 is configured to, when it does not belong to an abbreviated block, decide a prediction encoding mode for the current image block, and encode the current image block into the live stream according to the best prediction encoding mode determined by the decision; the transmission display module 3400 is configured to transmit the encoded live stream to a live broadcast room for display.

[0174] Based on any embodiment of the device of the present application, the mode enabling module 3100 includes: a surrounding acquisition module, configured to determine the surrounding image blocks adjacent to the current image block position, and obtain the determined coding cost indicators of multiple surrounding image blocks that have been encoded; a value setting module, configured to match the corresponding weights for the determined coding cost indicators of the multiple surrounding image blocks according to the position information of the surrounding image blocks in the video frame, and merge them into a cost measurement threshold; a mode judgment module, configured to determine the coding cost indicator of the current image block, compare the coding cost indicator of the current image block with the cost measurement threshold, and when it is greater than the cost measurement threshold, exit the quick decision mode, otherwise, enter the quick decision mode.

[0175] Based on any embodiment of the device of the present application, the mode judgment module includes: a motion estimation module, which is configured to perform motion estimation on the current image block based on a reference frame to obtain a prediction block; a residual transformation module, which is configured to calculate the residual block between the current image block and the prediction block, perform discrete cosine transformation on the residual block, and quantize it according to a quantization parameter to obtain a transformation coefficient matrix; an indicator calculation module, which is configured to calculate the sum of the squares of non-zero coefficients in the transformation coefficient matrix as the transformation coefficient energy, as a coding cost indicator for the current image block.

[0176] Based on any embodiment of the device of the present application, the abridged coding module 3200 includes: a brightness judgment module, which is configured to divide the current image block into multiple small blocks based on the brightness component, and judge whether the current image block belongs to an abridged block that can be simplified in coding based on the multiple small blocks; a chrominance judgment module, which is configured to, when judging that the current image block belongs to an abridged block based on the brightness component, continue to judge whether the current image block belongs to an abridged block based on the chrominance component of the current image block as a whole; a fast coding module, which is configured to, when judging that the current image block belongs to an abridged block based on the chrominance component, encode the current image block into the corresponding abridged information in the live stream; and a standard coding module, which is configured to run the default coding module 3300 if the current image block does not belong to an abridged block.

[0177] Based on any embodiment of the device of the present application, the brightness judgment module includes: a definition segmentation module, which is configured to determine the corresponding quantization threshold and threshold matrix according to the quantization parameter used to calculate the coding cost index, and divide the brightness component of the current image block into multiple small blocks; a statistical comparison module, which is configured to determine the statistical characteristics of each small block of the brightness component of the current image block, and compare the statistical characteristics with the quantization threshold. If the statistical characteristics of each small block are less than the quantization threshold, proceed to the next step, otherwise the current image block is a non-omitted block; a transform scoring module, which is configured to perform discrete cosine transform on the residual block of each small block, determine its transform coefficient according to the quantization parameter, and compare the transform coefficient of each small block with the threshold matrix for scoring. When the score is lower than the preset threshold, the current image block is confirmed to be an omit block, otherwise the current image block is a non-omitted block.

[0178] Based on any embodiment of the device of the present application, the chrominance judgment module includes: a threshold determination module, which is configured to determine the corresponding quantization threshold based on the quantization parameter used to calculate the coding cost index; a statistical analysis module, which is configured to determine the statistical characteristics of the chrominance component of the current image block, and compare the statistical characteristics with the quantization threshold. If the statistical characteristics are less than the quantization threshold, the current image block is an abbreviated block, otherwise the current image block is a non-abbreviated block.

[0179] Based on any embodiment of the device of the present application, the default encoding module 3300 includes: a mode determination module, which is configured to perform inter-frame prediction corresponding to multiple prediction modes on the current image block based on the reference frame, select the best prediction mode according to the prediction result, and determine the residual information corresponding to the best prediction mode; a quantization transformation module, which is configured to transform the residual information to obtain a quantized transformation coefficient; an optimal encoding module, which is configured to encode the quantized transformation coefficient, and write the encoding result into the live stream in combination with the prediction mode information corresponding to the best prediction mode.

[0180] Another embodiment of the present application also provides a live video image encoding device. Figure 8 As shown, a schematic diagram of the internal structure of a live video image encoding device. The live video image encoding device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable non-volatile readable storage medium of the live video image encoding device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a live video image encoding method.

[0181] The processor of the live video image encoding device is used to provide computing and control capabilities to support the operation of the entire live video image encoding device. The memory of the live video image encoding device may store computer-readable instructions, which, when executed by the processor, may enable the processor to execute the live video image encoding method of the present application. The network interface of the live video image encoding device is used to connect and communicate with a terminal.

[0182] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the live video image encoding device to which the scheme of the present application is applied. The specific live video image encoding device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In this embodiment, the processor is used to execute Figure 7 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the live video image encoding device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0184] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live video image encoding method of any embodiment of the present application.

[0185] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0187] In summary, this application reduces unnecessary calculations and speeds up encoding by optimizing the encoding decision process. At the same time, through precise omission block judgment and selection of the best prediction encoding mode, it maintains video quality and achieves a double improvement in live video image encoding efficiency and quality. For online live broadcast platforms, it can reap significant economies of scale.

Claims

1. A method for encoding live video images, characterized in that: include: Determining whether to enable a fast decision mode according to a difference in encoding costs between a current image block and encoded surrounding image blocks in a current video frame of a live stream; When the quick decision mode is enabled, whether the current image block belongs to an abbreviated block that can be simplified for encoding is progressively determined based on the brightness component and the chrominance component of the current image block, and when the current image block belongs to an abbreviated block, the current image block is encoded as corresponding abbreviated information in the live stream; When the image block does not belong to the omitted block, a prediction coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal prediction coding mode determined by the decision; The encoded live stream is transmitted to the live broadcast room for display.

2. The live video image encoding method according to claim 1, characterized in that: Determine whether to enable the fast decision mode according to the difference in encoding cost between the current image block and its encoded surrounding image blocks in the current video frame of the live stream, including: Determine surrounding image blocks that are immediately adjacent to the current image block, and obtain determined coding cost indicators of a plurality of surrounding image blocks that have been coded; According to the position information of the surrounding image blocks in the video frame, the determined encoding cost indicators of the plurality of surrounding image blocks are matched with corresponding weights, and the determined encoding cost indicators are merged into a cost measurement threshold; Determine the coding cost index of the current image block, compare the coding cost index of the current image block with the cost measurement threshold, and when it is greater than the cost measurement threshold, exit the fast decision mode, otherwise, enter the fast decision mode.

3. The live video image encoding method according to claim 2, characterized in that: Determine the coding cost index of the current image block, including: Perform motion estimation on the current image block based on the reference frame to obtain a prediction block; Calculate the residual block between the current image block and the prediction block, perform discrete cosine transform on the residual block, and quantize it according to the quantization parameter to obtain a transform coefficient matrix; The square sum of non-zero coefficients in the transform coefficient matrix is ​​calculated as the transform coefficient energy, which is used as the coding cost indicator of the current image block.

4. The live video image encoding method according to claim 1, characterized in that: The method comprises: determining whether the current image block belongs to an abbreviated block that can be simplified for encoding according to the brightness component and the chrominance component of the current image block in sequence, and encoding the current image block into corresponding abbreviated information in the live stream when the current image block belongs to the abbreviated block. Dividing the current image block into multiple small blocks based on the brightness component, and judging whether the current image block belongs to an abbreviated block that can be simplified in coding based on the multiple small blocks; When it is determined that the current image block belongs to an abbreviated block according to the luminance component, it is further determined whether the current image block belongs to an abbreviated block that can be simplified in coding based on the chrominance component of the current image block as a whole; When it is determined according to the chrominance component that the current image block belongs to an abbreviated block, encoding the current image block into corresponding abbreviated information in the live stream; If the current image block does not belong to the omitted block, the step of deciding the prediction coding mode of the current image block and encoding the current image block into the live stream according to the best prediction coding mode determined by the decision is executed.

5. The live video image encoding method according to claim 4, characterized in that: Based on the brightness component of the current image block, a plurality of small blocks are obtained by dividing the current image block, and based on the plurality of small blocks, it is determined whether the current image block belongs to an abbreviated block that can be simplified in coding, including: Determine a corresponding quantization threshold and a threshold matrix according to the quantization parameter used to calculate the coding cost index, and divide the brightness component of the current image block into a plurality of small blocks; Determine the statistical characteristics of each small block of the brightness component of the current image block, compare the statistical characteristics with the quantization threshold, if the statistical characteristics of each small block are less than the quantization threshold, proceed to the next step, otherwise the current image block is a non-abbreviated block; The residual block of each small block is subjected to discrete cosine transform, and its transform coefficient is determined according to the quantization parameter. The transform coefficient of each small block is compared with the threshold matrix for scoring. When the score is lower than the preset threshold, the current image block is confirmed to be an abbreviated block, otherwise the current image block is a non-abbreviated block.

6. The live video image encoding method according to claim 4, characterized in that: Based on the chrominance component of the current image block, it is judged whether the current image block belongs to an abbreviated block that can be simplified in coding, including: Determine a corresponding quantization threshold value according to a quantization parameter used to calculate the coding cost indicator; The statistical feature of the chrominance component of the current image block is determined, and the statistical feature is compared with the quantization threshold. If the statistical feature is less than the quantization threshold, the current image block is an abbreviated block, otherwise the current image block is a non-abbreviated block.

7. The live video image encoding method according to any one of claims 1 to 6, characterized in that: When the image block does not belong to the omitted block, a prediction coding mode of the current image block is determined, and the current image block is encoded into the live stream according to the optimal prediction coding mode determined by the decision, including: Based on the reference frame, inter-frame prediction corresponding to multiple prediction modes is performed on the current image block, the best prediction mode is selected according to the prediction results, and residual information corresponding to the best prediction mode is determined; Performing transform coding on the residual information to obtain quantized transform coefficients; The quantized transform coefficients are encoded, and combined with the prediction mode information corresponding to the optimal prediction mode, the encoding result is written into the live broadcast stream.

8. A live video image encoding device, characterized in that: include: a mode enabling module configured to determine whether to enable a fast decision mode based on a difference in encoding costs between a current image block and its encoded surrounding image blocks in a current video frame of a live stream; an abbreviation encoding module, configured to, when the quick decision mode is enabled, progressively determine whether the current image block belongs to an abbreviation block that can be simplified in encoding based on the brightness component and the chrominance component of the current image block, and if the current image block belongs to an abbreviation block, encode the current image block into corresponding abbreviation information in the live stream; A default encoding module is configured to decide a prediction encoding mode for a current image block when the image block does not belong to an abbreviated block, and encode the current image block into the live stream according to the best prediction encoding mode determined by the decision; The transmission and display module is configured to transmit the encoded live stream to the live broadcast room for display.

9. A live video image encoding device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images