Multi-person screen mirroring-based show live broadcast adaptive code rate control method

Abstract

This invention relates to the field of video encoding and decoding technology, specifically to an adaptive bitrate control method for multi-person live streaming. It solves the technical problem that existing technologies cannot dynamically adjust bitrate allocation when total bandwidth is limited, resulting in a poor overall visual experience in multi-person live streaming. The method includes: determining the resource contention weight of each input stream based on the source-end encoding compression rate, audio energy value, and spatial proportion; correcting the image texture complexity based on the resource contention weight to determine the target encoding complexity of each input stream; determining the original texture complexity of each input stream based on the image texture complexity and spatial proportion; and determining the offset value of the quantization parameters used to adjust the encoding region of each input stream based on the difference between the target encoding complexity and the original texture complexity of each input stream. This invention is applicable to video bitrate control scenarios.

Description

Technical Field

[0001] This invention relates to the field of video encoding technology, and more specifically to an adaptive bitrate control method for live streaming of shows based on multi-person screen sharing. Background Technology

[0002] With the popularization of multi-person real-time interactive live streaming, the media processing unit, as a core component, undertakes the task of receiving multiple heterogeneous video streams, decoding and splicing them, and re-encoding and distributing them to the audience. In actual engineering scenarios, the various video sources participating in the multi-screen live streaming are usually in different network and device environments. Due to network congestion or limited device performance, some streaming ends upload raw images that contain a large amount of quantization noise and ringing effects caused by high compression rates. Existing microcontroller unit merging and transcoding solutions typically adopt a bitrate allocation strategy based on global pixel domain texture. This type of strategy lacks the ability to perceive the quality of the input source and often mistakenly identifies blocky noise at the source as high-frequency texture details and allocates a large number of encoding bits. At the same time, it cannot perceive the audio activity status, resulting in the inability to dynamically adjust the bitrate allocation when the total bandwidth is limited, leading to a poor overall visual experience in multi-person multi-screen live streaming. Summary of the Invention

[0003] To address the technical problem that existing solutions cannot detect audio activity, resulting in an inability to dynamically adjust bitrate allocation when total bandwidth is limited, thus leading to a poor overall visual experience in multi-person connected live streaming, this invention aims to provide an adaptive bitrate control method for live streaming of shows based on multi-person connected screens. The specific technical solution adopted is as follows: In a first aspect, the present invention provides an adaptive bitrate control method for live streaming of shows based on multi-person screen sharing. The method includes: obtaining the source-end encoding compression rate before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the combined screen for each input stream participating in the screen sharing; determining the resource contention weight of each input stream based on the source-end encoding compression rate, the audio energy value, and the spatial proportion; correcting the image texture complexity based on the resource contention weight to determine the target encoding complexity of each input stream; determining the original texture complexity of each input stream based on the image texture complexity and the spatial proportion; and determining the offset value of the quantization parameter used to adjust the encoding region of each input stream based on the difference between the target encoding complexity of each input stream and the original texture complexity of each input stream.

[0004] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: using the synthesis time of the combined screen output frame as a reference, retrieving video frames whose display timestamps match the synthesis time from the receive buffer of each input stream; before decoding the video frames, performing bitstream parsing on the video elementary stream corresponding to the video frames, and determining the quantization parameters used by the video frames during encoding at the streaming end as the source-end encoding compression rate; dividing the decoded video frames into macroblocks, and determining the image texture complexity based on the sum of the absolute transform differences of all macroblocks in the entire frame; extracting audio sampling data within a preset time window before the synthesis time, and determining the audio energy value based on the root mean square energy value of the audio sampling data; and determining the space ratio based on the number of macroblocks occupied by the pixel size of the input stream in the combined screen image.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the root mean square energy value as the current audio energy value; adding the current audio energy value to a historical energy queue of a preset length and removing the earliest historical data from the historical energy queue; and determining the audio energy value as the arithmetic mean of all energy values ​​in the historical energy queue.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: normalizing the source-end coding compression rate, audio energy value, and spatial proportion to determine the source quality component, service concern component, and spatial weight component; determining the resource competition weight based on the relative proximity of the source quality component, service concern component, and spatial weight component to the first and second reference vectors; the first reference vector is composed of the normalized maximum values ​​of each dimension; and the second reference vector is composed of the normalized minimum values ​​of each dimension.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the maximum and minimum values ​​of the source-end coding compression ratio, the maximum and minimum values ​​of the audio energy value, and the maximum and minimum values ​​of the spatial proportion for each input stream; for the source-end coding compression ratio dimension, if the difference between the maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the source-end coding compression ratio dimension is set to a preset intermediate value; otherwise, a reverse normalization method is used to determine the source quality component; for the audio energy value dimension, if the difference between its maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the audio energy value dimension is set to a preset intermediate value; otherwise, a forward normalization method is used to determine the service-related component; for the spatial proportion dimension, if the difference between its maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the spatial proportion dimension is set to a preset intermediate value; otherwise, a forward normalization method is used to determine the spatial weight component.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: constructing a flow state feature vector composed of a source quality component, a service concern component, and a spatial weight component; calculating a first distance between the flow state feature vector and a first reference vector; calculating a second distance between the flow state feature vector and a second reference vector; and determining the resource competition weight based on the ratio of the second distance to the sum of the first and second distances.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the total amount of original texture based on the image texture complexity and spatial proportion; determining a nonlinear gain factor based on resource contention weights and a preset focus adjustment factor; the nonlinear gain factor is used to characterize the degree of nonlinear amplification or attenuation of the resource contention weights; and determining the target encoding complexity based on the total amount of original texture and the nonlinear gain factor.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: determining the weighted target allocation ratio of each input stream based on the target encoding complexity; determining the original texture complexity ratio of each input stream based on the image texture complexity and spatial proportion; and determining the offset value based on the ratio of the weighted target allocation ratio to the original texture complexity ratio.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the method specifically includes: comparing the ratio with a preset lower threshold and a preset upper threshold; if the ratio is less than the lower threshold, adjusting the ratio to the lower threshold; if the ratio is greater than the upper threshold, adjusting the ratio to the upper threshold; determining an offset value based on the adjusted ratio and a preset rate-distortion relationship; the rate-distortion relationship is used to characterize the correspondence between changes in the bitrate allocation ratio and the adjustment amount of the quantization parameters.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: constructing a two-dimensional offset matrix with the same macroblock resolution as the combined screen video frame; filling the offset value into the macroblock position covered by the corresponding input stream in the two-dimensional offset matrix based on the layout coordinates of each input stream in the combined screen image; sending the two-dimensional offset matrix and the combined screen video frame to be encoded to the encoder, instructing the encoder to superimpose the corresponding offset value on the basic quantization parameters.

[0013] The present invention has the following beneficial effects: This invention first acquires multi-dimensional feature data, including source-end encoding compression rate, audio energy value, image texture complexity, and spatial proportion. Then, based on these features, it determines resource contention weights, corrects to obtain the target encoding complexity, determines the original texture complexity, and finally generates a quantization parameter offset value based on the difference between the two, constructing a complete adaptive bitrate control mechanism. It can comprehensively perceive multi-dimensional information such as the source quality of the input stream, service focus, and image proportion, enabling differentiated resource allocation for different input streams. This solves the technical problem of existing solutions failing to perceive audio activity status, resulting in an inability to dynamically adjust bitrate allocation when total bandwidth is limited, leading to a poor overall visual experience in multi-person live streaming. Attached Figure Description

[0014] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating an embodiment of the adaptive bitrate control method for live streaming of a show based on multi-person screen sharing provided by the present invention. Figure 2 This is a schematic diagram of the structure of an adaptive bitrate control device for live streaming of shows based on multi-person screen sharing, provided in an embodiment of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the adaptive bitrate control method for live streaming based on multi-person multi-screen interaction proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] The specific scheme of the adaptive bitrate control method for live streaming based on multi-person screen connection provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0019] Please see Figure 1The diagram shows a flowchart of an adaptive bitrate control method for live streaming based on multi-person screen sharing, which is provided by an embodiment of the present invention. The method includes the following steps S101-S105, which will be described in detail below.

[0020] S101. Obtain the source-end encoding compression rate of each input stream participating in the screen merging before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the screen merging image.

[0021] In one possible implementation, using the synthesis time of the combined output frame as a reference, video frames whose display timestamps match the synthesis time are retrieved from the receive buffer of each input stream and used as valid frames for the current processing cycle. For the retrieved video frames, before sending them to the decoder, the bitstream of the corresponding video elementary stream is parsed to extract the quantization parameters used in the streaming encoding. These quantization parameters are determined as the source-end coding compression ratio. The source-end coding compression ratio characterizes the compression degree of the original source signal at the streaming end; a larger value indicates more severe source-end compression, more quantization noise and distortion in the image, and poorer source quality. The decoded video frames are divided into macroblocks, and the image texture complexity is determined based on the statistical values ​​of the frequency domain transform coefficients of all macroblocks in the entire frame. Image texture complexity characterizes the degree of pixel change in the image content; a larger value indicates richer image details, naturally requiring more bit resources for encoding. Audio sampling data within a preset time window before the synthesis time is extracted, and the audio energy value is determined based on the energy statistics of the audio sampling data. The audio energy value characterizes the loudness of the corresponding broadcaster's voice within the current time window. A higher value indicates a more active broadcaster and higher engagement in the current live stream interaction. To eliminate the impact of transient fluctuations in the audio signal on image quality, the audio energy value can be processed using temporal smoothing. Based on the pixel size of the input stream in the combined screen image, the number of macroblocks it occupies is determined as its spatial proportion. The spatial proportion characterizes the visible area of ​​the stream in the viewer's field of vision; a higher value indicates a greater weight of influence on the subjective visual experience.

[0022] S102. Based on the source-end coding compression rate, audio energy value, and spatial proportion, determine the resource contention weight of each input stream.

[0023] In one possible implementation, the source-end coding compression rate, audio energy value, and spatial proportion are normalized to obtain the corresponding source quality component, service concern component, and spatial weight component. Resource contention weights are determined based on the relative proximity of these components to a first and a second benchmark vector. The first benchmark vector consists of the normalized maximum values ​​of each dimension, and the second benchmark vector consists of the normalized minimum values ​​of each dimension.

[0024] S103. Based on resource contention weights, the image texture complexity is corrected to determine the target encoding complexity of each input stream.

[0025] In one possible implementation, the total amount of original textures is first determined based on the image texture complexity and spatial proportion. This total amount of original textures represents the base bitrate requirement of the path stream without intervention. Then, a nonlinear gain factor is determined based on the resource contention weights and a preset focus adjustment factor. This nonlinear gain factor is used to nonlinearly amplify or attenuate the resource contention weights. Finally, the total amount of original textures is multiplied by the nonlinear gain factor to obtain the target coding complexity.

[0026] S104. Determine the original texture complexity of each input stream based on the image texture complexity and space ratio.

[0027] In one possible implementation, the original texture complexity is determined by both the image texture complexity and the spatial proportion, specifically by the product of the image texture complexity and the spatial proportion. This original texture complexity reflects the bitrate requirement naturally allocated by the encoder based on purely physical features and serves as a benchmark for subsequent comparison with the corrected target encoding complexity.

[0028] For example, for each input stream, its image texture complexity is multiplied by the number of macroblocks it occupies to obtain the original texture complexity. The larger this value is, the more bit resources the stream requires at the physical level.

[0029] S105. Based on the difference between the target encoding complexity of each input stream and the original texture complexity of each input stream, determine the offset value of the quantization parameter used to adjust the encoding region of each input stream.

[0030] In one possible implementation, firstly, the weighted target allocation ratio for each input stream is determined based on the target encoding complexity; that is, the proportion of the target encoding complexity to the sum of the target encoding complexities of all streams. Simultaneously, the proportion of the original texture complexity for each input stream is determined based on the original texture complexity; that is, the proportion of the original texture complexity to the sum of the original texture complexities of all streams. Then, the offset value of the quantization parameter is determined based on the ratio of the weighted target allocation ratio to the original texture complexity ratio.

[0031] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment acquires multi-dimensional feature data, including source-end encoding compression rate, audio energy value, image texture complexity, and spatial proportion. Based on these features, it determines resource contention weights, corrects and obtains the target encoding complexity, determines the original texture complexity, and finally generates a quantization parameter offset value based on the difference between the two, thus constructing a complete adaptive bitrate control mechanism. It can comprehensively perceive multi-dimensional information such as the source quality of the input stream, service focus, and image proportion, achieving differentiated resource allocation for different input streams. This solves the technical problem of existing solutions being unable to perceive audio activity status, resulting in an inability to dynamically adjust bitrate allocation when total bandwidth is limited, leading to a poor overall visual experience in multi-person live streaming.

[0032] In one possible implementation, the process of obtaining the source-end encoding compression rate of each input stream participating in the screen merging before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the screen merging image can be specifically implemented through the following S201-S205, which will be explained in detail below.

[0033] S201. Based on the composite time of the combined output frame, retrieve the video frame whose display timestamp matches the composite time from the receive buffer of each input stream.

[0034] In one possible implementation, when preparing to synthesize a combined video frame with a display timestamp of a specific moment, for each input stream, the receive buffer of that stream is traversed to find the video frame data packet whose display timestamp is not greater than the synthesis moment and whose absolute time difference with the synthesis moment is the smallest. If the search is successful, the frame is locked as a valid frame for the current processing cycle; if there is no matching data within the current time window, the stream is marked as invalid and will not participate in subsequent calculations.

[0035] For example, the compositing time can be the display timestamp of the current combined frame to be composited. By comparing the display timestamps of each input stream video frame with the compositing time, the multiple streams are aligned in time, eliminating timing drift caused by network transmission jitter.

[0036] S202. Before decoding the video frame, perform bitstream parsing on the video basic stream corresponding to the video frame, and determine the quantization parameters used when encoding the video frame at the streaming end as the source-end encoding compression rate.

[0037] In one possible implementation, before sending the retrieved video frame to the decoder, a bitstream parser parses the video elementary stream to extract the quantization parameters used during encoding at the streaming end. These quantization parameters are encapsulated as metadata tags, bound to the compressed file to be decoded, and passed along with the data stream to subsequent stages.

[0038] For example, for H.264 / AVC bitstreams, the average quantization parameter of the frame can be calculated by parsing the quantization parameter information in the slice header and the initialization quantization parameter information in the image parameter set; for H.265 / HEVC bitstreams, the quantization parameter information in the slice header can be parsed. If the input stream supports the pass-through of supplementary enhancement information, the source-end statistical information carried therein can also be read directly.

[0039] S203. Divide the decoded video frame into macroblocks and determine the image texture complexity based on the sum of the absolute transform differences of all macroblocks in the whole frame.

[0040] In one possible implementation, after the video stream is decoded and restored to pixel data, the image is divided into macroblock units of a preset size, the frequency domain transform coefficient statistics of each macroblock are calculated, and then the statistics of all macroblocks in the whole frame are accumulated and normalized to obtain the image texture complexity.

[0041] For example, an image can be divided into 16×16 pixel macroblocks. The sum of the absolute transform differences of each macroblock is calculated, which involves summing the absolute values ​​of the transformed macroblock pixels. The sum of the absolute transform differences of all macroblocks in the entire frame is then accumulated and divided by the total number of macroblocks to obtain the normalized mean, which is used as the image texture complexity. The larger this value, the more complex the image texture and the richer the details.

[0042] S204. Extract audio sampling data within a preset time window before the synthesis time, and determine the audio energy value based on the root mean square energy value of the audio sampling data.

[0043] In one possible implementation, audio sampling data within a preset time window before the synthesis time is extracted, its root mean square energy value is calculated, and it is then subjected to temporal smoothing to eliminate transient fluctuations, thus obtaining the final audio energy value.

[0044] For example, audio sampling data from a certain time period (e.g., 400 milliseconds) prior to the synthesis time can be extracted, and its root mean square energy value can be calculated as the raw energy. To smooth out fluctuations, a historical energy queue of a preset length is maintained. The current root mean square energy value is added to the queue, and the oldest historical data in the queue is removed. The arithmetic mean of all energy values ​​in the queue is calculated as the final audio energy value. The larger this value, the louder and more stable the corresponding broadcaster's voice.

[0045] S205. Determine the space ratio based on the number of macroblocks occupied by the pixel size of the input stream in the combined screen.

[0046] In one possible implementation, the current screen layout configuration is read, the pixel width and pixel height of the input stream in the final canvas are obtained, and the number of macroblocks it occupies is calculated based on the macroblock size as the space ratio.

[0047] For example, the macroblock size is typically 16×16 pixels. The pixel width is divided by 16 and rounded up, and the pixel height is divided by 16 and rounded up. The two results are then multiplied to obtain the total number of macroblocks occupied by that stream. The larger this value, the larger the visible area of ​​that stream in the combined screen, and the higher its weight in influencing the viewer's visual experience.

[0048] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific acquisition method of multi-dimensional feature data. By performing temporal alignment based on the synthesis time of the combined output frame, the temporal consistency of feature data of each input stream is ensured; by performing bitstream parsing on the video basic stream before decoding to obtain the source-end encoding compression rate, the original encoding information of the streaming end is preserved; by dividing the decoded video frames into macroblocks and determining the image texture complexity based on the sum of absolute transform differences, the complexity of the image content is accurately characterized; by extracting audio sampling data within a preset time window and determining the audio energy value based on the root mean square energy value, the audio activity state is effectively reflected; by determining the spatial proportion based on the number of macroblocks occupied by the pixel size, the visual weight of each stream in the combined screen is accurately quantified. This provides a precise, synchronous, and multi-dimensional data foundation for subsequent resource contention weight calculation.

[0049] In one possible implementation, the process of extracting audio sampling data within a preset time window before the synthesis time and determining the audio energy value based on the root mean square energy value of the audio sampling data can be specifically implemented through the following S301-S303, which will be explained in detail below.

[0050] S301. Determine the root mean square energy value as the current audio energy value.

[0051] In one possible implementation, the root mean square energy value of the audio sampled data within the preset time window is calculated, and this root mean square energy value is used as the instantaneous audio energy value at the current moment, and recorded as the current audio energy value.

[0052] For example, for a segment of audio sample data, the sum of the squares of each sample value is calculated, divided by the number of sample points, and then the square root is taken to obtain the root mean square energy value, which directly reflects the overall loudness level of the audio signal within that time window.

[0053] S302. Add the current audio energy value to the historical energy queue of a preset length, and remove the earliest historical data from the historical energy queue.

[0054] One possible implementation involves maintaining a historical energy queue of a preset length to store audio energy values ​​across multiple consecutive processing cycles. In each processing round, the current audio energy value is added to the tail of the queue, while the oldest historical data value is removed from the head of the queue, keeping the queue length constant.

[0055] For example, the queue length can be set to 10, corresponding to a time span of approximately 400 milliseconds. Each frame is processed by enqueuing the current energy value and simultaneously popping the oldest data from the front of the queue, forming a sliding window-style data update mechanism.

[0056] S303. The arithmetic mean of all energy values ​​in the historical energy queue is determined as the audio energy value.

[0057] In one possible implementation, after each queue update, the arithmetic mean of all energy values ​​in the current queue is calculated, and this average is used as the final output audio energy value to characterize the stable audio activity of the stream at the current moment.

[0058] For example, the average value is obtained by summing all 10 energy values ​​in the queue and dividing by 10. This average value effectively filters out instantaneous spikes and fluctuations in the audio signal, reflecting the level of speech activity over a longer time scale.

[0059] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific method for determining the audio energy value. By adding the current root mean square energy value to a historical energy queue of a preset length and removing the oldest historical data, a sliding window-style audio energy smoothing mechanism is constructed; by calculating the arithmetic mean of all energy values ​​in the historical energy queue as the final audio energy value, the impact of instantaneous fluctuations in the audio signal on bitrate allocation is effectively eliminated. This avoids drastic fluctuations in video quality caused by millisecond-level transient changes in audio energy, achieving stable image quality focus guided by audio, and improving the viewer's subjective viewing experience.

[0060] In one possible implementation, the process of determining the resource contention weight of each input stream based on the source-end coding compression rate, audio energy value, and spatial proportion can be specifically implemented through the following S401-S402, which will be explained in detail below.

[0061] S401. Normalize the source coding compression rate, audio energy value, and spatial proportion to determine the source quality component, service concern component, and spatial weight component.

[0062] In one possible implementation, the source-end coding compression rate, audio energy value, and spatial proportion of each input stream are normalized to eliminate the differences between different physical dimensions, and the original values ​​of each dimension are mapped to a unified numerical range to obtain the corresponding source quality component, service concern component, and spatial weight component.

[0063] For example, for the source coding compression rate, since its value is negatively correlated with the source quality, it can be processed using reverse normalization, so that the stream with better source quality receives a higher source quality component; for the audio energy value and spatial proportion, since its value is positively correlated with business attention and visual importance, it can be processed using forward normalization, so that the stream with more active audio and a larger proportion of the picture receives a higher business attention component and spatial weight component.

[0064] S402. Based on the relative proximity of the source quality component, service focus component, and spatial weight component to the first and second benchmark vectors, determine the resource competition weight.

[0065] In one possible implementation, the source quality component, service focus component, and spatial weight component of each input stream are combined to form the stream's feature vector. The first benchmark vector consists of the normalized maximum values ​​of each dimension, representing the ideal state where the source is clearest, the sound is loudest, and the image is largest. The second benchmark vector consists of the normalized minimum values ​​of each dimension, representing the poor state where the source is most blurry, silent, and the image is smallest. The resource contention weight of the stream is determined by calculating the relative proximity between the stream's feature vector and the first and second benchmark vectors.

[0066] For example, the deviation between the feature vector and the first reference vector, and the proximity to the second reference vector, can be calculated separately. The smaller the deviation from the first reference vector and the larger the proximity to the second reference vector, the higher the resource contention weight of the flow. This weight value comprehensively reflects the overall quality of the flow in three dimensions: source quality, service attention, and spatial proportion.

[0067] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific method for determining resource contention weights. By normalizing the source-end coding compression rate, audio energy value, and spatial proportion, the dimensional differences of data in different dimensions are eliminated, enabling fair comparison of each input stream under the same standard. By introducing a first reference vector composed of the maximum normalized values ​​of each dimension and a second reference vector composed of the minimum normalized values ​​of each dimension, an objective reference coordinate system is established. By calculating the relative proximity between the feature components of each input stream and the two reference vectors, the resource contention capability of each input stream in the multi-dimensional feature space is comprehensively quantified. This achieves a joint evaluation of source quality, service focus, and spatial weights, providing a scientific basis for subsequent differentiated bitrate allocation.

[0068] In one possible implementation, the process of normalizing the source coding compression rate, audio energy value, and spatial proportion to determine the source quality component, service concern component, and spatial weight component can be specifically implemented through the following S501-S504, which will be explained in detail below.

[0069] S501. Determine the maximum and minimum values ​​of the source-end coding compression rate, the maximum and minimum values ​​of the audio energy value, and the maximum and minimum values ​​of the spatial proportion for each input stream.

[0070] One possible implementation involves iterating through all current valid input streams, calculating the values ​​of the source-side coding compression rate, audio energy value, and spatial proportion at the current moment, and determining the maximum and minimum values ​​for each dimension.

[0071] For example, for the source-side encoding compression ratio, the maximum value of this parameter across all streams is denoted as... The minimum value is denoted as For audio energy values, the maximum value of this parameter across all streams is recorded as follows: The minimum value is denoted as For spatial proportion, the maximum value of this parameter across all flows is denoted as... The minimum value is denoted as These extreme values ​​reflect the distribution range of data in each dimension at the current moment.

[0072] S502. For the source-end coding compression ratio dimension, if the difference between the maximum and minimum values ​​is less than the preset tolerance threshold, the normalized value of all input streams in this dimension is set to the preset intermediate value; otherwise, the source quality component is determined by reverse normalization.

[0073] In one possible implementation, for the source-end coding compression ratio dimension, the difference between its maximum and minimum values ​​is first calculated, and this difference is compared with a preset tolerance threshold. If the difference is less than the tolerance threshold, it indicates that the source-end coding compression ratios of all streams at the current moment are extremely similar, and this dimension lacks discriminatory power. In this case, the normalized value of all input streams under this dimension is uniformly set to a preset intermediate value (e.g., 0.5), and this preset intermediate value is directly used as the source quality component of each input stream. If the difference is not less than the tolerance threshold, it indicates that this dimension has discriminatory power. Then, a reverse normalization method is used to determine the source quality component of each input stream based on the relationship between the source-end coding compression ratio of each stream and its maximum and minimum values. The reverse normalization method means that for the parameter of source-end coding compression ratio, which represents better source quality, the normalized source quality component is positively correlated with the source quality through a formula. That is, the stream with a smaller source-end coding compression ratio obtains a larger source quality component.

[0074] For example, source quality components Satisfy the following formula 1: Formula 1 in, The average source-end coding compression ratio of the i-th input stream represents the degree of compression of the original source at the push end. The larger the value, the worse the source quality. For all valid input streams at the current time The maximum value; For all valid input streams at the current time Minimum value; These are parameter tuning coefficients, and their values ​​should be extremely small positive numbers (e.g., 0.01) to avoid a denominator of 0. It reflects the difference between the source compression rate of the current stream and the worst source. The larger the value, the better the quality of the current stream source. This represents the range of source quality across all streams at the current moment; overall, inverse normalization is used, i.e. The smaller the molecule (the better the source quality), the larger the molecule. The closer to 1; conversely The larger, As it gets closer to 0, Formula 1 converts the source compression ratio into a dimensionless component that is positively correlated with the priority.

[0075] Optionally, considering the potential differences in the physical meaning and numerical distribution characteristics of the three dimensions, different preset tolerance thresholds can be set independently for each dimension in practical applications. For example, the tolerance threshold for the source-end coding compression rate dimension can be set relatively small (e.g., 1), because even small changes in the quantization parameters can reflect differences in source quality; the tolerance threshold for the audio energy value dimension can be set slightly larger (e.g., 1000), because the fluctuation range of audio energy is large, requiring a larger range to reflect effective discrimination. Specifically, it needs to be combined with the dynamic range calibration of the output audio data of the specific audio acquisition device; the tolerance threshold for the spatial proportion dimension can be set according to the granularity of the combined screen layout (e.g., 1, i.e., when they are completely equal, it is determined that there is no discrimination). When all stream screen sizes are the same, the range is zero, and at this time, the threshold judgment directly triggers boundary protection.

[0076] S503. For the audio energy value dimension, if the difference between its maximum and minimum values ​​is less than the preset tolerance threshold, then the normalized value of all input streams in the audio energy value dimension is set to the preset median value; otherwise, the forward normalization method is used to determine the business focus component.

[0077] In one possible implementation, for the audio energy value dimension, the difference between its maximum and minimum values ​​is first calculated, and this difference is compared with a preset tolerance threshold. If the difference is less than the tolerance threshold, it indicates that the audio energy values ​​of all streams at the current moment are extremely similar, and this dimension lacks discriminatory power. In this case, the normalized value of all input streams under this dimension is uniformly set to a preset intermediate value (e.g., 0.5), and this preset intermediate value is directly used as the service focus component of each input stream. If the difference is not less than the tolerance threshold, it indicates that this dimension has discriminatory power. Then, a forward normalization method is adopted, and the service focus component of each input stream is determined according to the relationship between the audio energy value of each stream and its maximum and minimum values. The forward normalization method means that for the parameter of audio energy value, which represents a higher level of service focus, a formula is used to make the normalized service focus component positively correlated with the audio energy value, that is, the stream with a higher audio energy value receives a larger service focus component.

[0078] For example, business focus components The following formula 2 is satisfied: Formula 2 in, The audio energy value of the i-th stream represents the voice activity level; the larger the value, the higher the business attention. For all valid input streams at the current time The maximum value; For all valid input streams at the current time Minimum value; These are parameter tuning coefficients, and their values ​​should be extremely small positive numbers (e.g., 0.01) to avoid a denominator of 0. This represents the difference between the current streaming audio energy and the minimum value; the larger the value, the more active the audio stream. It represents the range of audio energy values ​​in all streams at the current moment; Formula 2, through positive normalization, makes the stream with higher audio energy obtain a component closer to 1, and the stream with lower energy obtain a component closer to 0, thus realizing a positive correlation between audio activity and priority.

[0079] S504. For the spatial proportion dimension, if the difference between its maximum and minimum values ​​is less than the preset tolerance threshold, then the normalized values ​​of all input streams in the spatial proportion dimension are set to the preset median value; otherwise, the spatial weight components are determined by the forward normalization method.

[0080] In one possible implementation, for the spatial proportion dimension, the difference between its maximum and minimum values ​​is first calculated, and this difference is compared with a preset tolerance threshold. If the difference is less than the tolerance threshold, it indicates that the spatial proportion differences among all streams at the current moment are extremely small, and this dimension lacks discriminatory power. In this case, the normalized value of all input streams under this dimension is uniformly set to a preset median value (e.g., 0.5), and this preset median value is directly used as the spatial weight component of each input stream. If the difference is not less than the tolerance threshold, it indicates that this dimension has discriminatory power. Then, a forward normalization method is adopted, and the spatial weight component of each input stream is determined according to the relationship between the spatial proportion of each stream and its maximum and minimum values. The forward normalization method means that for the parameter of spatial proportion, which represents a larger visual weight, the normalized spatial weight component is positively correlated with the spatial proportion through a formula, that is, the stream with a larger screen proportion receives a larger spatial weight component.

[0081] For example, spatial weight components The following formula 3 is satisfied: Formula 3 in, The number of macroblocks occupied by the i-th stream in the combined screen image represents the screen proportion; For all valid input streams at the current time The maximum value; For all valid input streams at the current time Minimum value; These are parameter tuning coefficients, and their values ​​should be extremely small positive numbers (e.g., 0.01) to avoid a denominator of 0. This represents the difference between the number of macroblocks and the minimum value; a larger value indicates greater activity. This represents the range of macroblock counts across all streams at the current moment; Formula 3, through positive normalization, indicates the stream with the largest screen area... The closer it is to 1, the greater its visual impact on the audience, and the more bitrate should be allocated to it.

[0082] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific execution method of normalization processing. By separately calculating the maximum and minimum values ​​under each dimension, dynamic normalization based on the data distribution at the current moment is realized; by introducing a preset tolerance threshold to judge the data discrimination of each dimension, when the range of a certain dimension is too small, a preset intermediate value is used to replace it, effectively avoiding calculation anomalies caused by the lack of data discrimination; reverse normalization is used for the source-end encoding compression rate, and forward normalization is used for the audio energy value and spatial proportion, ensuring that the normalized values ​​of each dimension are positively correlated with resource priority. The robustness and accuracy of the normalization process are guaranteed, and effective normalization results can be stably output even in extreme scenarios.

[0083] In one possible implementation, the process of determining the resource competition weight based on the relative proximity of the source quality component, service concern component, and spatial weight component to the first and second reference vectors can be specifically implemented through the following S601-S604, which will be explained in detail below.

[0084] S601. Construct a flow state feature vector composed of source quality components, service concern components, and spatial weight components.

[0085] The stream state feature vector represents the multi-dimensional integrated state of each input stream at the current moment after normalization, specifically including the following three dimensions: Source quality dimension: characterized by the source quality component, reflecting the relative quality of the input stream in terms of source-side coding compression rate, i.e., the quality of the source image. Service focus dimension: characterized by the service focus component, reflecting the relative activity of the input stream in the current audio energy, i.e., whether it is the current focus of the speech. Visual weight dimension: characterized by the spatial weight component, reflecting the relative size of the area occupied by the input stream in the combined screen image, i.e., the weight of its impact on the viewer's visual experience.

[0086] In one possible implementation, the source quality component, service focus component, and spatial weight component obtained after normalization of each input stream are combined to construct a three-dimensional stream state feature vector. This feature vector comprehensively describes the combined state of the input stream at the current moment across the three dimensions of source quality, service focus, and visual spatial weight.

[0087] S602, Calculate the first distance between the flow state feature vector and the first reference vector.

[0088] In one possible implementation, the preset first reference vector is composed of the normalized maximum values ​​of each dimension, i.e., [1, 1, 1], representing the ideal state where the source is clearest, the sound is loudest, and the image is largest. The geometric distance between the flow state feature vector of each input stream and the first reference vector is calculated as the first distance, used to characterize the degree of deviation between the stream and the ideal state.

[0089] For example, Euclidean distance can be used for calculation, which involves calculating the square root of the sum of the squares of the differences between each of the three components of the flow state feature vector and 1. The smaller the first distance, the closer the input flow is to the ideal state, and the higher the overall quality.

[0090] S603, Calculate the second distance between the flow state feature vector and the second reference vector.

[0091] In one possible implementation, the preset second reference vector is composed of the normalized minimum values ​​of each dimension, i.e., [0, 0, 0], representing the worst state of the source, characterized by the most blurred, silent, and smallest image. The geometric distance between the flow state feature vector of each input stream and this second reference vector is calculated as the second distance, used to characterize the degree of proximity between the stream and the worst state.

[0092] For example, Euclidean distance can be used for calculation, which involves calculating the square root of the sum of the squares of the differences between each of the three components of the flow state feature vector and 0. The larger the second distance, the further the input flow is from the poor state, and the higher the overall quality.

[0093] S604. Determine the resource competition weight based on the ratio of the second distance to the sum of the first and second distances.

[0094] In one possible implementation, the second distance of each input stream is used as the numerator, and the sum of the first distance and the second distance is used as the denominator. The ratio between the two is calculated, and this ratio is determined as the resource contention weight of that input stream.

[0095] For example, the formula for calculating resource contention weight can be expressed as the second distance divided by the sum of the first and second distances. When the feature vector of a certain flow is closer to the ideal state, the first distance approaches 0, and the second distance is relatively large. At this time, the ratio approaches 1, indicating that the flow should receive the highest priority resource guarantee. When the feature vector is closer to the inferior state, the second distance approaches 0, and the ratio approaches 0, indicating that the flow belongs to an inefficient source and should be suppressed. In engineering implementation, a very small positive constant (such as 0.01) can be added to the denominator as a protection to avoid numerical instability caused by floating-point calculation precision.

[0096] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific calculation method of resource contention weight. By constructing a stream state feature vector composed of source quality components, service concern components, and spatial weight components, multi-dimensional features are fused into a unified mathematical expression; by calculating the first distance between this feature vector and the first reference vector and the second distance between it and the second reference vector, the deviation degree of each input stream from the ideal state and the poor state is quantified; by determining the resource contention weight based on the ratio of the second distance to the sum of the first distance and the second distance, an objective ranking of the relative merits of each input stream is achieved. This results in input streams with high overall quality receiving a weight close to 1, and input streams with low overall quality receiving a weight close to 0, providing a quantitative basis for the precise allocation of bitrate resources.

[0097] In one possible implementation, the process of correcting the image texture complexity based on resource contention weights and determining the target encoding complexity of each input stream can be specifically implemented through the following S701-S703, which will be explained in detail below.

[0098] S701. Determine the total amount of original textures based on the complexity of the image texture and the space ratio.

[0099] One possible implementation combines the image texture complexity of each input stream with its spatial proportion to determine the total original texture of that stream. The total original texture is used to characterize the bitrate requirement of that input stream based on purely physical image features without any business value intervention.

[0100] For example, the image texture complexity can be multiplied by the spatial proportion to obtain the original total texture. Image texture complexity reflects the richness of image details, while spatial proportion reflects the visual area of ​​the stream in the combined screen. The product of the two combines the factors of detail density and visual area, jointly determining the scale of bitrate resources that the encoder naturally tends to allocate based on the rate-distortion optimization principle. The larger the original total texture, the more bit resources that stream needs to consume at the physical level.

[0101] S702. Based on the resource competition weights and the preset focusing adjustment factor, determine the nonlinear gain factor; the nonlinear gain factor is used to characterize the degree of nonlinear amplification or attenuation of the resource competition weights.

[0102] In one possible implementation, the resource contention weight of each input stream is combined with a preset focusing adjustment factor to determine the nonlinear gain factor of that stream. The nonlinear gain factor is used to perform a nonlinear transformation on the resource contention weight, thereby achieving differentiated amplification or attenuation of streams with different priorities.

[0103] For example, the resource competition weights can be exponentially calculated using a focus adjustment factor as a parameter to obtain a nonlinear gain factor. The focus adjustment factor is a preset constant (e.g., 1) used to control the degree to which resource allocation is tilted towards high-value streams. When the focus adjustment factor is greater than 1, the exponential calculation amplifies the resource competition weights nonlinearly and differentially, so that the gain of high-weight streams is relatively maintained, while the gain of low-weight streams is further reduced, thereby widening the gap between different streams; when the focus adjustment factor is equal to 1, the nonlinear gain factor is equal to the resource competition weights themselves, which is equivalent to a linear mapping; when the focus adjustment factor is less than 1, the exponential calculation amplifies the resource competition weights nonlinearly, so that the gain of low-weight streams is relatively increased, which can be used in scenarios that need to take into account multiple streams.

[0104] S703. Determine the target encoding complexity based on the total amount of original texture and the nonlinear gain factor.

[0105] In one possible implementation, the total original texture amount of each input stream is combined with its nonlinear gain factor to determine the target coding complexity of that stream. The target coding complexity is used to characterize the effective bitrate requirement after business value weighting correction, and serves as the basis for subsequent comparison with the original texture complexity, thereby deriving the quantization parameter offset value.

[0106] For example, the target coding complexity can be obtained by multiplying the original total texture size by a nonlinear gain factor. For input streams with high resource contention weight (i.e., good overall quality), the nonlinear gain factor is relatively large, which amplifies the original total texture size, resulting in a target coding complexity higher than the original total texture size. This means that the business layer expects to allocate more bitrate resources to it. For input streams with low resource contention weight (i.e., poor overall quality), the nonlinear gain factor is relatively small, which attenuates the original total texture size, resulting in a target coding complexity lower than the original total texture size. This means that the business layer expects to suppress its bitrate consumption.

[0107] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific method for determining the target coding complexity. By determining the total amount of original texture based on the image texture complexity and spatial proportion, the bitrate requirement base of the input stream at the physical level is characterized; by determining the nonlinear gain factor based on the resource contention weight and the preset focus adjustment factor, the nonlinear amplification or attenuation of the resource contention weight is realized; by determining the target coding complexity based on the total amount of original texture and the nonlinear gain factor, the business value weight is injected into the bitrate requirement calculation in a nonlinear manner. This significantly improves the target coding complexity of high-value input streams and effectively suppresses the target coding complexity of low-value input streams, realizing the dynamic focus of resources on high-value areas.

[0108] In one possible implementation, the process of determining the offset value of the quantization parameter for adjusting the encoding region of each input stream based on the difference between the target encoding complexity of each input stream and the original texture complexity of each input stream can be specifically implemented through the following S801-S803, which will be described in detail below.

[0109] S801. Determine the weighted target allocation ratio for each input stream based on the target encoding complexity.

[0110] In one possible implementation, the target coding complexity of each input stream is normalized to determine the resource allocation ratio of that stream under the service expectation, which is then used as the weighted target allocation ratio. The weighted target allocation ratio is used to characterize the share of bitrate resources that the service layer believes that the stream should receive after corrections based on multi-dimensional features such as source quality, service attention, and spatial proportion.

[0111] For example, the sum of the target coding complexities of all valid input streams can be calculated, and then the target coding complexity of each stream can be divided by this sum to obtain the weighted target allocation percentage for that stream. This percentage is a dimensionless value, and the sum of the weighted target allocation percentages of all streams is 1. The larger the weighted target allocation percentage, the more bitrate resources the business layer expects that stream to receive; conversely, the smaller the weighted target allocation percentage, the less bitrate resources it is expected to receive.

[0112] S802. Determine the original texture complexity ratio of each input stream based on the image texture complexity and space ratio.

[0113] In one possible implementation, based on the image texture complexity and spatial proportion of each input stream, the proportion of bitrate resources that the encoder naturally tends to allocate based on rate-distortion optimization principles for that stream is determined as the original texture complexity proportion. The original texture complexity proportion is used to characterize the original allocation benchmark at the purely physical level.

[0114] For example, firstly, the original texture complexity of each stream is determined based on the image texture complexity and spatial proportion. Then, the sum of the original texture complexities of all valid input streams is calculated. Finally, the original texture complexity of each stream is divided by this sum to obtain the original texture complexity proportion of that stream. This proportion is also a dimensionless value, and the sum of the original texture complexity proportions of all streams is 1. The larger the original texture complexity proportion, the more bitrate resources the encoder naturally tends to allocate to that stream based on its physical characteristics.

[0115] For example, the proportion of original texture complexity The following formula 4 is satisfied: Formula 4 in, Let be the image texture complexity of the i-th input stream, used to characterize the degree of drastic pixel changes in the image content of that stream; The space occupied by the i-th input stream in the combined screen; This is a parameter tuning coefficient (e.g., 0.01) used to prevent numerical anomalies caused by zero denominator or zero numerator; N is the total number of valid input streams participating in screen merging at the current moment; The original texture total of the i-th input stream represents the base bitrate requirement of the stream based on pure physical image features without any business intervention. Summing the total amount of original textures for all valid input streams yields the sum of the total amount of original textures for all streams at the current time. Determine the proportion of the original texture complexity for the i-th flow. .

[0116] S803. Determine the offset value based on the ratio of the weighted target allocation ratio to the original texture complexity ratio.

[0117] In one possible implementation, the weighted target allocation ratio of each input stream is compared with the original texture complexity ratio, and the offset value used to adjust the quantization parameters of the corresponding coding region of that stream is determined based on the ratio of the two. In engineering implementation, to ensure the numerical stability of the calculation, a very small positive constant (such as 0.01) can be added as a denominator protection to the original texture complexity ratio.

[0118] For example, for each input stream, the ratio of its weighted target allocation percentage to its original texture complexity percentage is calculated. When this ratio is greater than 1, it indicates that the service expects the bitrate share of this stream to be higher than its natural tendency, requiring more bitrate resources to be allocated to this stream. In this case, a negative offset value should be generated to reduce quantization parameters and improve image quality. When the ratio is less than 1, it indicates that the service expects the bitrate share of this stream to be lower than its natural tendency, requiring suppression of bitrate consumption in this stream. In this case, a positive offset value should be generated to increase quantization parameters. When the ratio is equal to 1, it indicates that the service expectation is consistent with the natural tendency, and the offset value is 0, with no adjustment made. The specific magnitude of the offset value can be determined based on the ratio and a preset rate-distortion relationship, such that the further the ratio deviates from 1, the greater the adjustment.

[0119] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific method for determining the quantization parameter offset value. By determining the weighted target allocation ratio based on the target coding complexity, the ideal resource allocation ratio expected at the business level is characterized; by determining the original texture complexity ratio based on the image texture complexity and spatial ratio, the original resource allocation ratio naturally inclined by the encoder is characterized; by calculating the ratio of the two ratios to determine the offset value, the supply and demand difference between business expectations and natural inclination is quantified. A mapping relationship from business needs to underlying coding control is established. When the expected ratio is higher than the original ratio, a negative offset value is generated to improve image quality; when the expected ratio is lower than the original ratio, a positive offset value is generated to suppress bitrate consumption, thus achieving precise intervention in coding behavior.

[0120] In one possible implementation, the process of determining the offset value based on the ratio of the weighted target allocation ratio to the original texture complexity ratio can be specifically implemented through the following S901-S904, which will be explained in detail below.

[0121] S901. Compare the ratio with the preset lower threshold and the preset upper threshold.

[0122] In one possible implementation, after obtaining the ratio of the weighted target allocation ratio to the original texture complexity ratio for each input stream, this ratio is compared with a preset lower threshold and a preset upper threshold to determine whether the ratio is within a preset reasonable range. The lower and upper thresholds are two pre-defined positive constants used to define the safe range of the ratio.

[0123] Understandably, to prevent abnormal adjustment of quantization parameters due to ratios that are too small or too large, the following measures are taken: It is limited to a preset valid range. Specifically, a preset lower threshold is defined. and upper limit threshold ,like < Then let = ;like > Then let = Otherwise, keep constant.

[0124] For example, It can be set to 0.2. It can be set to 5.0. This range covers the supply and demand differences in most normal business scenarios. Even in extreme cases, the truncated ratio can still correctly reflect the adjustment direction, and the corresponding quantization parameter adjustment is limited to the range allowed by the encoder, ensuring encoding stability and smooth visual transition.

[0125] S902. If the ratio is less than the lower threshold, the ratio will be adjusted to the lower threshold.

[0126] In one possible implementation, when the comparison result shows that the ratio of a certain input stream is less than a preset lower threshold, the ratio of that stream is forcibly set to the lower threshold to avoid abnormal calculation of subsequent quantization parameter offsets due to an excessively small ratio.

[0127] S903. If the ratio is greater than the upper limit threshold, the ratio will be adjusted to the upper limit threshold.

[0128] In one possible implementation, when the comparison result shows that the ratio of a certain input stream is greater than a preset upper limit threshold, the ratio of that stream is forcibly set to the upper limit threshold to avoid abnormal calculation of subsequent quantization parameter offsets due to an excessively large ratio.

[0129] S904. Determine the offset value based on the adjusted ratio and the preset rate-distortion relationship.

[0130] Among them, the rate-distortion relationship is used to characterize the correspondence between changes in the code rate allocation ratio and the adjustment of quantization parameters.

[0131] In one possible implementation, the ratio after threshold adjustment is input into a preset rate-distortion relationship model, and the quantization parameter offset value corresponding to the ratio is determined according to the model. The rate-distortion relationship adopts the logarithmic mapping law commonly used in the field of video coding: doubling the bitrate corresponds to a decrease of 6 in the quantization parameter, and halving the bitrate corresponds to an increase of 6 in the quantization parameter.

[0132] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific execution method for determining the offset value based on the ratio. By comparing the ratio with preset lower and upper thresholds, ratios exceeding a reasonable range are truncated, avoiding logarithmic operation anomalies caused by ratios that are too small or too large; by introducing a rate-distortion relationship to characterize the correspondence between changes in the bit rate allocation ratio and the adjustment amount of quantization parameters, a quantization parameter adjustment model that conforms to the characteristics of the encoder is established; by further restricting the initial offset value of the quantization parameters, it is ensured that the adjusted offset value is within the smooth adjustment range allowed by the encoder. This guarantees the numerical stability of the offset value calculation and encoder compatibility, realizing a reliable conversion from algorithm decision to underlying instructions.

[0133] In one possible implementation, it is also necessary to instruct the encoder to superimpose the corresponding offset value on the basic quantization parameters. This process can be specifically implemented through the following S1001-S1003, which will be explained in detail below.

[0134] S1001. Construct a two-dimensional offset matrix that matches the macroblock resolution of the combined video frame.

[0135] In one possible implementation, after determining the quantization parameter offset for each input stream, a two-dimensional matrix is ​​constructed that matches the macroblock division of the combined video frame to be encoded. The number of rows and columns of this matrix corresponds to the number of macroblocks in the height and width directions of the combined video frame, respectively, and each element in the matrix is ​​used to store the quantization parameter offset value required for the corresponding macroblock position.

[0136] S1002. Based on the layout coordinates of each input stream in the combined screen, fill the offset value into the macroblock position covered by the corresponding input stream in the two-dimensional offset matrix.

[0137] In one possible implementation, the start and end coordinates of each input stream in the combined screen are determined based on its layout configuration. These pixel coordinates are then converted into macroblock coordinates to determine the row and column ranges of macroblocks covered by that stream. Finally, the quantization parameter offset value corresponding to that stream is filled into the positions of all macroblocks within the corresponding row and column ranges in a two-dimensional offset matrix.

[0138] S1003. Send the two-dimensional offset matrix and the combined video frame to be encoded to the encoder, instructing the encoder to superimpose the corresponding offset value on the basic quantization parameters.

[0139] In one possible implementation, the constructed two-dimensional offset matrix is ​​used as auxiliary data accompanying the frame and is synchronously sent to the encoder along with the combined video frames to be encoded. When performing macroblock-level rate-distortion optimization on the video frames to be encoded, the encoder reads the two-dimensional offset matrix. For each macroblock, it adds the corresponding offset value to the basic quantization parameters of the current frame to obtain the quantization parameters actually used for that macroblock, and encodes accordingly.

[0140] The technical solution provided by the above embodiments can bring at least the following beneficial effects: This embodiment further defines the specific execution method of sending the offset value to the encoder. By constructing a two-dimensional offset matrix with the same resolution as the macroblock of the combined screen video frame, a fine control grid covering the entire screen is established; by filling the calculated offset value into the macroblock position covered by the corresponding input stream based on the layout coordinates of each input stream in the combined screen, spatial dimension differential control is realized; by synchronously sending the two-dimensional offset matrix and the combined screen video frame to be encoded to the encoder, the encoder can superimpose the corresponding offset value during the macroblock-level rate-distortion optimization process. Without modifying the encoder kernel, precise intervention in the encoding behavior is achieved through external metadata injection, completing a complete closed-loop control from algorithm decision-making to underlying execution.

[0141] Please see Figure 2This document illustrates a schematic diagram of an adaptive bitrate control device 200 for live streaming based on multi-person screen sharing, according to an embodiment of the present invention. The device includes a communication unit 201 and a processing unit 202. The communication unit 201 is used to acquire the source-end encoding compression rate before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the combined screen for each input stream participating in the screen sharing. The processing unit 202 is used to determine the resource contention weight of each input stream based on the source-end encoding compression rate, audio energy value, and spatial proportion; to correct the image texture complexity based on the resource contention weight and determine the target encoding complexity of each input stream; to determine the original texture complexity of each input stream based on the image texture complexity and spatial proportion; and to determine the offset value of the quantization parameters used to adjust the encoding region of each input stream based on the difference between the target encoding complexity and the original texture complexity of each input stream.

[0142] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for adaptive bitrate control in live streaming of variety shows based on multi-person multi-screen interaction, characterized in that, The method includes: Obtain the source-end encoding compression rate of each input stream participating in the screen merging process before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the screen merging process; Based on the source-end coding compression rate, the audio energy value, and the spatial proportion, the resource contention weight of each input stream is determined; The image texture complexity is corrected based on the resource contention weight to determine the target encoding complexity of each input stream; The original texture complexity of each input stream is determined based on the image texture complexity and the space ratio. Based on the difference between the target encoding complexity of each input stream and the original texture complexity of each input stream, an offset value for the quantization parameter used to adjust the encoding region of each input stream is determined.

2. The adaptive bitrate control method for live streaming based on multi-person screen sharing as described in claim 1, characterized in that, The acquisition of the source-end encoding compression rate of each input stream participating in the screen merging before decoding, the audio energy value after decoding, the image texture complexity, and the spatial proportion in the screen merging image, includes: Based on the synthesis time of the combined output frame, retrieve the video frame whose display timestamp matches the synthesis time from the receive buffer of each input stream; Before decoding the video frame, the video basic stream corresponding to the video frame is parsed, and the quantization parameters used by the video frame when it is encoded at the streaming end are determined as the source-end encoding compression rate. The decoded video frame is divided into macroblocks, and the image texture complexity is determined based on the sum of the absolute transform differences of all macroblocks in the entire frame. Audio sampling data within a preset time window before the synthesis time is extracted, and the audio energy value is determined based on the root mean square energy value of the audio sampling data. The space ratio is determined based on the number of macroblocks occupied by the pixel size of the input stream in the combined screen image.

3. The adaptive bitrate control method for live streaming based on multi-person screen sharing as described in claim 2, characterized in that, The step of extracting audio sampling data within a preset time window before the synthesis time, and determining the audio energy value based on the root mean square energy value of the audio sampling data, includes: The root mean square energy value is determined as the current audio energy value; The current audio energy value is added to a historical energy queue of a preset length, and the earliest historical data in the historical energy queue is removed. The arithmetic mean of all energy values ​​in the historical energy queue is determined as the audio energy value.

4. The adaptive bitrate control method for live streaming based on multi-person screen sharing as described in claim 1, characterized in that, The determination of the resource contention weight for each input stream based on the source-end coding compression ratio, the audio energy value, and the spatial proportion includes: The source-end coding compression rate, the audio energy value, and the spatial proportion are normalized respectively to determine the source quality component, the service concern component, and the spatial weight component. Based on the relative proximity of the source quality component, the service focus component, and the spatial weight component to the first and second benchmark vectors, the resource competition weight is determined; the first benchmark vector is composed of the normalized maximum values ​​of each dimension; the second benchmark vector is composed of the normalized minimum values ​​of each dimension.

5. The adaptive bitrate control method for live streaming based on multi-person screen sharing as described in claim 4, characterized in that, The normalization process for the source-end coding compression ratio, the audio energy value, and the spatial proportion to determine the source quality component, service-related component, and spatial weight component includes: Determine the maximum and minimum values ​​of the source-end coding compression ratio, the maximum and minimum values ​​of the audio energy value, and the maximum and minimum values ​​of the spatial proportion for each input stream; For the source-end coding compression ratio dimension, if the difference between the maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the source-end coding compression ratio dimension is set to a preset intermediate value; otherwise, the source quality component is determined by reverse normalization. For the audio energy value dimension, if the difference between its maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the audio energy value dimension is set to the preset intermediate value; otherwise, the business focus component is determined by positive normalization. For the spatial proportion dimension, if the difference between its maximum and minimum values ​​is less than a preset tolerance threshold, then the normalized value of all input streams in the spatial proportion dimension is set to the preset intermediate value; otherwise, the spatial weight components are determined by positive normalization.

6. The adaptive bitrate control method for live streaming of a show based on multi-person multi-screen interaction as described in claim 4, characterized in that, The determination of the resource competition weight based on the relative proximity of the source quality component, the service focus component, and the spatial weight component to the first and second benchmark vectors includes: Construct a flow state feature vector composed of the source quality component, the service concern component, and the spatial weight component; Calculate the first distance between the flow state feature vector and the first reference vector; Calculate the second distance between the flow state feature vector and the second reference vector; The resource competition weight is determined based on the ratio of the second distance to the sum of the first distance and the second distance.

7. The adaptive bitrate control method for live streaming of a show based on multi-person connected screens as described in claim 1, characterized in that, The step of correcting the image texture complexity based on the resource contention weight to determine the target encoding complexity of each input stream includes: The total amount of original textures is determined based on the image texture complexity and the space ratio. Based on the resource competition weights and a preset focusing adjustment factor, a nonlinear gain factor is determined; the nonlinear gain factor is used to characterize the degree to which the resource competition weights are nonlinearly amplified or attenuated. The target encoding complexity is determined based on the total amount of the original texture and the nonlinear gain factor.

8. The adaptive bitrate control method for live streaming of a show based on multi-person connected screens as described in claim 1, characterized in that, The step of determining the offset value for adjusting the quantization parameter of the encoding region of each input stream based on the difference between the target encoding complexity of each input stream and the original texture complexity of each input stream includes: The weighted target allocation ratio of each input stream is determined based on the target coding complexity. The original texture complexity ratio of each input stream is determined based on the image texture complexity and the space ratio. The offset value is determined based on the ratio of the weighted target allocation ratio to the original texture complexity ratio.

9. The adaptive bitrate control method for live streaming of a show based on multi-person multi-screen interaction as described in claim 8, characterized in that, Determining the offset value based on the ratio of the weighted target allocation ratio to the original texture complexity ratio includes: The ratio is compared with a preset lower threshold and a preset upper threshold. If the ratio is less than the lower threshold, then the ratio is adjusted to the lower threshold. If the ratio is greater than the upper limit threshold, then the ratio is adjusted to the upper limit threshold; The offset value is determined based on the adjusted ratio and the preset rate-distortion relationship; the rate-distortion relationship is used to characterize the correspondence between the change in the code rate allocation ratio and the adjustment amount of the quantization parameter.

10. The adaptive bitrate control method for live streaming of a show based on multi-person connected screens according to any one of claims 1-9, characterized in that, The method further includes: Construct a two-dimensional offset matrix with the same macroblock resolution as the combined video frames; Based on the layout coordinates of each input stream in the combined screen, the offset value is filled into the macroblock position covered by the corresponding input stream in the two-dimensional offset matrix; The two-dimensional offset matrix and the combined video frame to be encoded are sent to the encoder, which is instructed to superimpose the corresponding offset value on the basic quantization parameters.

Images