Video encoder parameter dynamic adjustment method

Abstract

The invention relates to a method for dynamically adjusting parameters of a video encoder, which comprises the following steps of: acquiring a visual characteristic index corresponding to a video frame image in real time, and calculating an image complexity score of the image according to the visual characteristic index; the visual feature indexes comprise motion intensity, edge density and information entropy; acquiring a network index of a video transmission network in real time; the network indexes comprise available bandwidth, delay, jitter and packet loss rate; obtaining a current encoder control parameter according to the current image complexity score and the network index; the control parameters of the encoder comprise GOP, GP, FPS and Buffer Size; and adjusting the encoder according to the obtained control parameters of the encoder. The method is superior to an existing fixed parameter coding system in the aspects of image quality, delay control, system robustness and adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video transmission, and particularly to a method for dynamically adjusting parameters of a video encoder. BACKGROUND

[0002] With the development of industrial automation and remote control systems, high-definition video acquisition and low-latency transmission technologies are increasingly widely used in heavy industrial scenes such as port machinery, mine operations, manufacturing workshops, and rail transportation. In order to ensure the real-time perception of operators and intelligent decision-making of the system, the camera equipment not only needs to have high-definition image quality and image stabilization functions, but also must meet the transmission requirements of low delay, no freezing, and high robustness.

[0003] Currently, industrial camera systems mostly use industrial-grade cores with optical image stabilization functions to cooperate with encoding boards for image processing and output. However, in the prior art, the encoder parameters are mostly fixed configurations, and there is a lack of adaptive adjustment mechanisms for different image complexities and network states. Especially when facing large vibration disturbances, dramatic image changes, or unstable network bandwidth, traditional systems are prone to problems such as large image delay, freezing, sudden image quality degradation, or data loss, which seriously affect the stability and safety of remote operation and intelligent recognition.

[0004] In addition, although some systems attempt to introduce dynamic code rate control, most of them only adjust the code stream based on network layer information, do not optimize the overall encoding strategy in combination with image content features, and do not jointly dynamically control the GOP structure, QP parameters, frame rate, and cache duration.

[0005] Therefore, there is an urgent need for an intelligent encoding control method that can jointly model image content and network state and has the ability to output variable encoding parameters, in order to realize a more stable and efficient industrial video transmission system. SUMMARY

[0006] To solve the above problems, the present application provides a method for dynamically adjusting parameters of a video encoder.

[0007] The specific scheme is as follows:

[0008] A method for dynamically adjusting parameters of a video encoder, comprising the following steps:

[0009] S1: Real-time acquisition of visual feature indicators corresponding to video frame images, and calculation of image complexity scores of the images according to the visual feature indicators; the visual feature indicators include motion intensity, edge density, and information entropy;

[0010] S2: Real-time acquisition of network indicators of a video transmission network; the network indicators include available bandwidth, delay, jitter, and packet loss rate;

[0011] S3: obtaining current encoder control parameters according to the current image complexity score and the network indicators; the encoder control parameters include: GOP, GP, FPS and BufferSize;

[0012] S4: adjusting the encoder according to the obtained encoder control parameters.

[0013] Further, the image complexity score is a result of weighted summation of the characteristic values of various visual feature indicators.

[0014] Further, the characteristic values of the visual feature indicators are normalized before being weighted and summed.

[0015] Further, the encoder control parameters are obtained through a rule decision table, and after recording the encoder control parameter values corresponding to different combinations of the image complexity score and the network indicators in the rule decision table, the corresponding encoder control parameter values are found through table lookup according to the current image complexity score and the network indicators.

[0016] Further, the encoder control parameters are obtained through a data mapping method, and after constructing a solving formula of each encoder control parameter based on the image complexity score and the network indicators, the values of each encoder control parameter are calculated through the solving formula according to the current image complexity score and the network indicators.

[0017] Further, the encoder control parameters are obtained through a neural network model, the input of the model is set to the image complexity score and each network indicator, and the output of the model is set to the encoder control parameter, and after training the constructed neural network model, the current image complexity score and the network indicators are input into the trained neural network model to obtain the values of each encoder control parameter output by the model.

[0018] Further, when adjusting the encoder, it also includes: judging whether the difference between the obtained each encoder control parameter and the actual value of each encoder control parameter is greater than the corresponding difference threshold value, if not, it is not adjusted.

[0019] Further, in the step of adjusting the encoder according to the obtained encoder control parameters, it also includes: calculating the corresponding single frame encoding delay according to the obtained encoder control parameters, and adding the single frame encoding delay and the transmission delay to obtain the total delay of the encoding and sending process; judging whether the total delay is greater than the set maximum delay, if yes, adjusting the obtained encoder control parameters, and adjusting the encoder according to the adjusted encoder control parameters.

[0020] Further, in the adjusting of the obtained encoder control parameters, the adjusting method is: increasing QP, increasing GOP, and decreasing FPS.

[0021] The present application adopts the above technical solutions, and is superior to the existing fixed parameter coding system in image quality, delay control, system robustness, and adaptability, and is particularly suitable for application in industrial scenes such as port remote control, mine operation, intelligent manufacturing, track monitoring, and the like, which have strict requirements on video performance. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of the method of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To further illustrate the embodiments, the present application provides accompanying drawings. These drawings are part of the disclosure of the present application, and mainly serve to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. Those of ordinary skill in the art should be able to understand other possible implementations and advantages of the present application by referring to these contents.

[0024] The present application will be further described in conjunction with the accompanying drawings and specific embodiments.

[0025] The embodiment of the present application provides a video encoder parameter dynamic adjustment method, as shown in the accompanying drawings, the method comprises the following steps: Figure 1 As shown in the accompanying drawings, the method comprises the following steps:

[0026] S1: Real-time acquisition of the visual feature index corresponding to the video frame image, and calculation of the image complexity score of the image according to the visual feature index.

[0027] In this embodiment, the video stream is collected in real time by the camera core, and each frame of image collected is a video frame image. In some embodiments, the camera core can be selected as an industrial-grade high-performance image core to improve the quality of the collected video frame image, which has the following characteristics: high sensitivity and low illumination imaging capability, suitable for outdoor, night or unevenly illuminated environment; support Full HD (1920x1080) or 4K (3840x2160) high resolution output; frame rate up to 60fps120fps, meeting the continuous imaging requirements in dynamic scenes; built-in anti-shake and noise reduction module, automatically corrects and compensates the image when the camera body vibrates.

[0028] In order to accurately evaluate the difficulty of image encoding in the encoder, the visual feature index is selected to evaluate the complexity score of the image in this embodiment, and the visual feature index is set to include three visual features: motion intensity, edge density, and information entropy. The corresponding feature extraction process is as follows.

[0029] The video frame image at the current time (t time) is denoted as where H, W represent the height and width of the image respectively, and C represents the number of channels (usually 3 channels in YUV or RGB format).

[0030] (1) Motion intensity S motion (t)

[0031] Estimate the pixel motion vector field between consecutive frame images (I t-1 , t )(I t-1 representing the video frame image at the previous time (t-1 time)) by optical flow method (such as Lucas-Kanade or Farneback algorithm):

[0032] M t (x, y) = (u(x, y), v(x, y))

[0033] Define the motion intensity as the average of all pixel displacements within the image:

[0034]

[0035] where (x, y) represents the coordinates of a pixel in the image, and u and v represent the motion vectors of the pixel in the x-axis and y-axis directions of the image, respectively.

[0036] (2) Edge density S edge (t)

[0037] Apply Sobel or Laplacian algorithm for edge detection on frame image I t , and obtain the corresponding binary edge map E t (x, y) ∈ {0, 1}.

[0038] Define the edge density as the average of all pixel values in the binary edge map within the image:

[0039]

[0040] (3) Information entropy S entropy (t)

[0041] Convert frame image I t to a grayscale image, count the pixel value distribution p(i), and calculate the information entropy of the image according to the obtained pixel value distribution p(i):

[0042]

[0043] In this embodiment, the image complexity score is set as the weighted sum of the normalized results of the three types of visual feature indicators, and the corresponding calculation formula is:

[0044]

[0045] wherein C img (t) represents the frame image I t of the current moment; and are the normalized feature values of S motion (t), S edge (t) and S entropy (t), respectively; α, β, γ ∈ [0, 1] represent the weight coefficients of the corresponding visual feature indicators (α represents the weight coefficient of the motion intensity indicator, β represents the weight coefficient of the edge density indicator, and γ represents the weight coefficient of the information entropy indicator), and satisfy α + β + γ = 1 (for example, α = 0.4, β = 0.2, and γ = 0.4). It should be noted that in actual use, dynamic weight adjustment can be made according to the application scenario, for example, increasing the value of α in a mobile scenario, and increasing the value of γ in a detail recognition scenario.

[0046] The normalization process can use a commonly used normalization formula, such as:

[0047]

[0048] wherein S * (t) represents the original feature value of a certain type of visual feature indicator of the current frame, and the subscript * indicates the type of the visual feature indicator; and S max represent the minimum feature value and the maximum feature value of the visual feature indicator, respectively, which are usually obtained from system acquisition statistics (observation of the first several frames), preset empirical values (determined during development), or sliding window dynamic adjustment (robust adaptive normalization), etc.

[0049] S2: Real-time acquisition of network indicators of a video transmission network.

[0050] The video transmission network is the network used by the encoder to receive the video stream. The network indicators can be acquired in real time by a network monitoring module, and the network indicators are quantified by acquiring the transmission state of the current video data link. The network indicators used in this embodiment include four network indicators: available bandwidth B(t), delay L(t), jitter J(t), and packet loss rate P(t), and a network state vector V net (t) is constructed based on the network indicators, which is used to provide the adaptive control strategy basis for the encoder.

[0051] V net (t) = {B(t), L(t), J(t), P(t)}

[0052] The specific meanings and units of the network indicators are shown in Table 1.

[0053] Table 1

[0054] Symbol Meaning Unit B(t) Current link available bandwidth Mbps L(t) Average end-to-end latency ms J(t) Jitter, i.e. short-term variance of latency ms P(t) Packet loss rate Percentage (%)

[0055] In collecting network metrics, this embodiment adopts a combination of passive and active collection strategies, as detailed below.

[0056] (1) Bandwidth B(t)

[0057] In actual encoding output links, the effective video byte transmission rate per unit time is measured based on a sliding window:

[0058]

[0059] Among them, s i N(t) represents the size of the i-th data packet (in bits); i represents the sequence number of the data packet; N(t) represents the total number of data packets corresponding to the t-th frame; Δt represents the sampling period (usually 500ms).

[0060] (2) Delay L(t)

[0061] By periodically sending Ping (ICMP) or custom UDP echo packets using a real-time control terminal or monitoring node, the current link latency can be estimated using the round-trip time (RTT).

[0062]

[0063] Among them, RTT i This represents the round-trip time of the i-th data packet.

[0064] (3) J(t) jitter

[0065] Jitter is the variability of delay, defined as the mean absolute difference or variance of the transmission time of consecutive frames:

[0066]

[0067] or:

[0068]

[0069] Among them, L i and L i-1 These represent the actual network delays from the sender to the receiver for the i-th and (i-1)-th data packets (or video frames), respectively. This represents the average delay.

[0070] (4) Packet loss rate P(t)

[0071] Within each statistics window, the number N of data packets that should be received is calculated based on statistics. expect The actual number of N received recv Calculate packet loss rate:

[0072]

[0073] In this embodiment, the network state vector is sampled and updated every Δt seconds, and an exponential moving average (EMA) is used to smooth out sudden changes:

[0074]

[0075] Where λ∈(0,1] represents the smoothing factor, with a typical value of 0.6 to 0.8, used to balance real-time performance and noise immunity; and These represent the smoothed network state vectors at the current and previous time steps, respectively.

[0076] S3: Obtain the current encoder control parameters based on the current image complexity score and network metrics.

[0077] The encoder control parameters used in this embodiment include GOP, GP, FPS, and BufferSize.

[0078] GOP (Group of Pictures Length) is the keyframe interval, representing the number of frames between two consecutive I-frames (IntraFrames, keyframes), typically ranging from 8 to 60. A small GOP (e.g., 15) results in faster image recovery and stronger resistance to packet loss, but a higher bitrate; a large GOP (e.g., 60) results in higher compression ratio and bandwidth savings, but slower recovery and weaker resistance to packet loss.

[0079] QP (Quantization Parameter) is a core parameter controlling the compression intensity, affecting image quality and bitrate. Its value typically ranges from 0 to 51, where a value of 0 indicates extremely high image quality, and a value of 51 indicates extremely low image quality. A larger QP value results in lower image quality (blurred image, low bitrate (higher compression)); a smaller QP value results in higher image quality (clearer image, higher bitrate (less compression)).

[0080] FPS (Frames Per Second) is the frame rate, representing the number of video frames encoded and output per unit of time. The unit is frames per second, and the value range is usually 10 to 60. Common values ​​are 30fps, 25fps, 15fps, and 10fps. A higher FPS results in smoother video and a larger data volume; a lower FPS results in lower latency, a lower bitrate, and more resource savings.

[0081] BufferSize is the size of the encoding buffer, representing the buffer capacity used for temporary data storage within the encoder. The unit can be Byte, ms, or Frame, and the value range is typically 50ms to 500ms. A larger BufferSize results in higher latency, stronger jitter resistance, and is suitable for scenarios that tolerate latency; a smaller BufferSize results in faster response and lower latency, but is more sensitive to burst bit rate fluctuations.

[0082] The strategy function for obtaining encoder control parameters can be expressed as:

[0083] Params enc (t)=f(X t )=f(C img (t),B(t),L(t),J(t),P(t))

[0084] Params enc (t)={GOP(t),QP(t),FPS(t),BufferSize(t)}

[0085] Params enc (t) represents the set of encoder control parameters; X t ={C img The set of input variables is represented by (t), B(t), L(t), J(t), P(t).

[0086] The policy function can be solved using methods such as rule-based decision tables, mathematical mappings, and neural network models. The following sections will explain each method in detail.

[0087] (1) Rule-Based Mapping

[0088] After recording the encoder control parameter values ​​corresponding to different combinations of image complexity scores and network metrics in the rule decision table, the corresponding encoder control parameter values ​​are retrieved from the rule decision table based on the current image complexity score and network metrics using a lookup table method. In this embodiment, a rule decision table as shown in Table 2 is constructed. Since the image complexity score and network metrics in the rule decision table are set to three levels—high, medium, and low—after obtaining the values ​​of the image complexity score and network metrics in the aforementioned steps, the obtained values ​​are compared with the corresponding boundary values ​​that distinguish between the high, medium, and low levels (the boundary values ​​need to be preset). This allows the determination of the respective levels of the image complexity score and network metrics, and then the lookup table is performed based on these levels.

[0089] Table 2

[0090] [C img ]]> B L J P Encoding strategy example High High Low Low Low GOP=12, QP=22, FPS=60, Buffer=100 High Low High Medium Medium GOP=30, QP=32, FPS=25, Buffer=300 Medium Medium Medium Medium Medium GOP=20, QP=28, FPS=40, Buffer=200 Low High Medium Low Low GOP=50, QP=24, FPS=60, Buffer=200 Medium Low High High High GOP=40, QP=34, FPS=20, Buffer=400 Low Medium Medium High Medium GOP=60, QP=36, FPS=15, Buffer=500

[0091] (b) Mathematical mapping

[0092] After constructing the solution formulas for each encoder control parameter based on image complexity score and network metrics, the values ​​of each encoder control parameter are calculated using these formulas based on the current image complexity score and network metrics. The solution formulas for each encoder control parameter constructed in this embodiment are as follows.

[0093]

[0094] Where Q0, G0, F0, and Buf0 are the default baseline parameters, and α i ,β i ,γ i ,δ i This is an adjustable coefficient.

[0095] (3) Neural Network Model

[0096] By constructing a neural network model, setting the image complexity score and various network metrics as the inputs, and setting the encoder control parameters as the outputs, and after training the constructed neural network model, inputting the current image complexity score and network metrics into the trained neural network model, the values ​​of the encoder control parameters output by the model are obtained.

[0097] To better capture the nonlinear relationships in the high-dimensional state space, a lightweight network model f is designed in this embodiment. θ It uses a 12-layer MLP (Multilayer Perceptron) with a parameter count of <100KB.

[0098] Params enc (t)=f θ (C img (t),B(t),L(t),J(t),P(t))

[0099] Furthermore, to avoid video stream instability caused by frequent switching of encoder control parameters, this embodiment introduces parameter update constraints, namely, determining whether the difference between the obtained encoder control parameters and the actual values ​​of the obtained encoder control parameters is greater than the corresponding difference threshold. If it is not greater, no adjustment is made; if it is greater, adjustment is made (that is, the encoder control parameters are updated to the obtained encoder control parameters).

[0100] S4: Adjust the encoder according to the obtained encoder control parameters.

[0101] The encoder receives encoder control parameters GOP, GP, FPS, and BufferSize before the start of each frame encoding and updates its configuration accordingly.

[0102] Set the current GOP length to GOP(t), that is, insert one I-frame every GOP(t) frames;

[0103] Setting the quantization parameter of the current frame to QP(t) affects the coding compression ratio and image quality;

[0104] Set the output frame rate to FPS(t). If the frame rate is lower than the acquisition frame rate, a frame rate reduction operation needs to be performed.

[0105] Set the duration of the encoded output buffer, BufferSize(t), to limit the buffer lag window.

[0106] Each encoder control parameter can be sent to the hardware or software encoding unit via the encoder driver API, and parameter change events are recorded for debugging and tracking. After adjusting the encoder via the encoder control parameters (i.e., updating the configuration), the adjusted encoder processes the input video frame image I. t The video stream is compressed and encoded, and finally output as a compressed video stream.

[0107] Furthermore, in adjusting the encoder according to the obtained encoder control parameters, this embodiment further includes: calculating the corresponding single-frame encoding delay according to the obtained encoder control parameters, and adding the single-frame encoding delay to the transmission delay to obtain the total delay of the encoding and transmission process; determining whether the total delay is greater than the set maximum delay (set to 100ms in this embodiment); if so, adjusting the obtained encoder control parameters, and adjusting the encoder according to the adjusted encoder control parameters.

[0108] In this embodiment, a single-frame encoding delay T is set. encode (t) is calculated based on the four encoder control parameters, and the corresponding calculation formula is:

[0109] T encode (t)=f1(QP(t),C img (t),GOP(t),R)

[0110] Where R represents the resolution (e.g., 1920x1080); f1 represents the processing time function that is positively correlated with the encoding complexity.

[0111] The transmission process of video frames must meet the following constraints:

[0112]

[0113] Among them, T net (t) represents the transmission delay; This indicates the maximum acceptable delay (e.g., 40ms) allocated to the network transmission portion of the total delay budget to ensure that the overall end-to-end delay does not exceed the maximum delay.

[0114] Therefore, the total delay of the encoding and transmission process is:

[0115] T total (t)=T encode (t)+T net (t)

[0116] Among them, T total (t) represents the total delay.

[0117] In this embodiment, the obtained encoder control parameters are adjusted by increasing QP to reduce the bit rate, increasing GOP to reduce the I-frame frequency, and decreasing FPS to alleviate system pressure. The specific increases or decreases can be set by those skilled in the art according to actual applications, and are not limited here.

[0118] This embodiment can feed back the current delay to the adjustment of encoder control parameters by judging the total delay, thus forming a closed-loop adaptive control.

[0119] The embodiments of the present invention have the following beneficial effects:

[0120] (1) Significantly reduces image transmission latency

[0121] By constructing a joint perception model of image complexity and network state, the encoder's GOP structure, QP parameters, frame rate, and buffer size are dynamically adjusted to keep the overall system transmission delay within the maximum delay, thus meeting the requirements of industrial scenarios with extremely high real-time requirements, such as remote control of port machinery and robot operation.

[0122] (2) Improve image continuity and stability

[0123] To address video stuttering and frame skipping issues, this invention effectively adapts to network jitter and sudden changes by real-time controlling encoding buffers and frame rate output strategies, thereby achieving a smooth and continuous video experience.

[0124] (3) Enhance image quality adaptive capability

[0125] This embodiment combines image motion intensity, edge density, and texture information to automatically improve coding quality (reduce QP) for complex scenes, and automatically extend GOP and save bitrate when image changes are not drastic, achieving the optimal balance between quality and resources under different visual loads.

[0126] (4) Improve network anti-interference robustness

[0127] The encoding strategy not only responds to the image content, but also to network conditions such as bandwidth, latency, and packet loss rate in real time, and intelligently switches between encoding modes and transmission protocols to maintain strong network adaptability and jitter resistance under complex working conditions.

[0128] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for dynamically adjusting video encoder parameters, characterized in that, Includes the following steps: S1: Real-time acquisition of visual feature indicators corresponding to video frame images, and calculation of image complexity score based on visual feature indicators; Visual feature metrics include motion intensity, edge density, and information entropy; S2: Real-time acquisition of network metrics for video transmission networks; network metrics include: available bandwidth, latency, jitter, and packet loss rate; S3: Based on the current image complexity score and network metrics, obtain the current encoder control parameters; the encoder control parameters include: GOP, GP, FPS, and BufferSize; S4: Adjust the encoder according to the obtained encoder control parameters.

2. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: The image complexity score is the result of a weighted sum of the feature values ​​of various visual feature indicators.

3. The method for dynamically adjusting video encoder parameters according to claim 2, characterized in that: Before performing weighted summation, the eigenvalues ​​of each visual feature index are normalized.

4. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: The encoder control parameters are obtained through a rule decision table. After recording the encoder control parameter values ​​corresponding to different combinations of image complexity scores and network metrics in the rule decision table, the corresponding encoder control parameter values ​​are found from the rule decision table based on the current image complexity score and network metrics using a lookup table method.

5. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: The encoder control parameters are obtained through a data mapping method. After constructing the solution formulas for each encoder control parameter based on the image complexity score and network index, the values ​​of each encoder control parameter are calculated using the solution formulas based on the current image complexity score and network index.

6. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: The encoder control parameters are obtained through a neural network model. The neural network model is constructed by setting the image complexity score and various network metrics as inputs and setting the encoder control parameters as outputs. After training the constructed neural network model, the current image complexity score and network metrics are input into the trained neural network model to obtain the values ​​of the encoder control parameters output by the model.

7. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: When adjusting the encoder, the process also includes: determining whether the difference between the obtained encoder control parameters and the actual values ​​of the encoder control parameters is greater than the corresponding difference threshold; if not, no adjustment is made.

8. The method for dynamically adjusting video encoder parameters according to claim 1, characterized in that: The process of adjusting the encoder based on the obtained encoder control parameters also includes: calculating the corresponding single-frame encoding delay based on the obtained encoder control parameters, and adding the single-frame encoding delay to the transmission delay to obtain the total delay of the encoding and transmission process; determining whether the total delay is greater than the set maximum delay; if so, adjusting the obtained encoder control parameters, and adjusting the encoder based on the adjusted encoder control parameters.

9. The method for dynamically adjusting video encoder parameters according to claim 8, characterized in that: The obtained encoder control parameters are being adjusted using the following methods: increasing QP, increasing GOP, and decreasing FPS.

Images