Video frame encoding method, device, electronic device and medium

Abstract

The embodiment of the present application provides a video frame encoding method, which relates to the field of video processing technology. The method includes: obtaining a video frame to be encoded; determining a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame; reducing the bit rate of the target frame; adjusting the quantization parameter for the target frame according to a preset smoothness parameter and the remaining bit rate after the reduction; encoding the reduced video frame according to the quantization parameter; and allocating the reduced bit rate to at least one video frame to be encoded other than the target frame. The embodiment of the present application can reduce the peak bandwidth and the frame rate while ensuring that the video quality remains basically unchanged, thereby providing a better user experience.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video frame encoding method, device, electronic device and medium. Background Art

[0002] With the development of network technology, more and more users are using video to communicate online, such as one-on-one video chats, many-to-many video chats, video conferencing, and live streaming. However, in real-world applications, network and device issues can cause video freezes, a critical issue that impacts user experience. Video freezes are generally caused by scene changes in the video content. Since there is no temporal correlation between the previous and next frames after a scene change, the next frame can no longer be predicted based on the previous one, resulting in a potentially high bitrate for the next frame. For example, during a video conference, especially when the number of participants exceeds a certain threshold, the scene change can cause a sudden increase in the bitrate of the video frame, leading to high peak bandwidth and a high freeze rate.

[0003] Currently, the main way to solve video freezes is to skip video frames whose bit rate is greater than a set threshold and continue processing other frames.

[0004] However, in the current solution, the inventors found that although a simple frame skipping operation reduces the peak bandwidth and the freeze rate and improves the smoothness, the video quality is greatly degraded due to the direct skipping of the video frame. Summary of the Invention

[0005] The embodiment of the present application provides a video frame encoding method to reduce peak bandwidth and reduce freeze rate while ensuring video quality as much as possible.

[0006] Correspondingly, the embodiment of the present application also provides a video frame encoding device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses a video frame encoding method, which includes:

[0008] Get the video frame to be encoded;

[0009] Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0010] reducing the bit rate of the target frame;

[0011] Adjusting the quantization parameter for the target frame according to a preset fluency parameter and the remaining bit rate after the reduction;

[0012] encoding the reduced video frame according to the quantization parameter;

[0013] The reduced bit rate is allocated to at least one to-be-encoded video frame except the target frame, and the step of determining the target frame among the to-be-encoded video frames is performed.

[0014] The present application also discloses a rate control method, which includes:

[0015] Get the video frame to be encoded;

[0016] Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0017] reducing the bit rate of the target frame;

[0018] The reduced bit rate is allocated to at least one video frame to be encoded except the target frame, and the step of determining the target frame among the video frames to be encoded is performed. The embodiment of the present application also discloses a video frame encoding method, which includes:

[0019] Obtaining a first video stream sent by a video transmitter and obtaining a fluency parameter of a video receiver;

[0020] Decoding the first video stream to obtain a video frame to be encoded;

[0021] Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0022] reducing the bit rate of the target frame;

[0023] Adjusting the quantization parameter for the target frame according to a preset fluency parameter and the remaining bit rate after the reduction;

[0024] Adjusting a quantization parameter for the target frame according to the fluency parameter;

[0025] encoding the reduced video frame according to the quantization parameter, and sending the encoded second video stream to a video receiving end;

[0026] The reduced bit rate is allocated to at least one to-be-encoded video frame except the target frame, and the step of determining the target frame among the to-be-encoded video frames is performed.

[0027] The present application also discloses a method for processing video conference data, the method comprising:

[0028] Obtaining the video frame to be encoded in the video conference data;

[0029] Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0030] reducing the bit rate of the target frame;

[0031] The reduced bit rate is allocated to at least one to-be-encoded video frame except the target frame, and the step of determining the target frame among the to-be-encoded video frames is performed.

[0032] The present application also discloses a method for processing live video data, the method comprising:

[0033] Obtain the video frame to be encoded in the live video data;

[0034] Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0035] reducing the bit rate of the target frame;

[0036] The reduced bit rate is allocated to at least one to-be-encoded video frame except the target frame, and the step of determining the target frame among the to-be-encoded video frames is performed.

[0037] The embodiment of the present application further discloses a video frame encoding device, the device comprising:

[0038] A first video frame acquisition module, used to acquire a video frame to be encoded;

[0039] A first frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0040] A first bit rate reduction module, configured to reduce the bit rate of the target frame;

[0041] A first quantization parameter adjustment module, configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and a reduced residual bit rate;

[0042] a first encoding module, configured to encode the reduced video frame according to the quantization parameter;

[0043] The first bit rate reallocation module is configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and enter the first frame detection module.

[0044] The embodiment of the present application further discloses a video frame encoding device, the device comprising:

[0045] A second video frame acquisition module, used to acquire the video frame to be encoded;

[0046] A second frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0047] A second bit rate reduction module, configured to reduce the bit rate of the target frame;

[0048] The second bit rate reallocation module is configured to allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and execute the second frame detection module.

[0049] The present application also discloses a rate control device, which includes:

[0050] A third acquisition module is used to acquire the first video stream sent by the video sending end and acquire the fluency parameter of the video receiving end;

[0051] A decoding module, configured to decode the first video stream to obtain a video frame to be encoded;

[0052] A third frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0053] a third bit rate reduction module, configured to reduce the bit rate of the target frame;

[0054] a third quantization parameter adjustment module, configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and the reduced residual bit rate;

[0055] a third encoding module, configured to encode the reduced video frame according to the quantization parameter, and send the encoded second video stream to a video receiving end;

[0056] The third bit rate reallocation module is configured to allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and execute the third frame detection module.

[0057] The present application also discloses a device for processing video conference data, the device comprising:

[0058] A fourth acquisition module is used to acquire the video frame to be encoded in the video conference data;

[0059] a fourth frame detection module, configured to determine a target frame in the to-be-encoded video frame; the target frame comprises a scene-change type video frame;

[0060] a fourth bit rate reduction module, configured to reduce the bit rate of the target frame;

[0061] The fourth bit rate reallocation module is configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fourth frame detection module.

[0062] The present application also discloses a device for processing live video data, the device comprising:

[0063] A fifth acquisition module is used to acquire the video frame to be encoded in the live video data;

[0064] a fifth frame detection module, configured to determine a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame;

[0065] a fifth bit rate reduction module, configured to reduce the bit rate of the target frame;

[0066] The fifth bit rate reallocation module is configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fifth frame detection module.

[0067] An embodiment of the present application further discloses an electronic device, comprising: a processor; and a memory on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0068] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0069] Compared with the prior art, the embodiments of the present application have the following advantages:

[0070] In an embodiment of the present application, for a video frame to be encoded, it is first determined whether it is a scene-switch type video frame. If the video frame to be encoded is a scene-switch type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded. The above process is then performed on the next video to be encoded. In this way, during the processing of the video frame, the bit rate of the video frame that is itself a scene-switch type, or the video frame that becomes a scene-switch type after being assigned a new bit rate, will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene-switch type video frame is reduced and the bit rate of the normal video frame is increased, which can minimize the overall bit rate change. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not significantly reduce the evaluation of video quality, and can also avoid the problem of high peak bandwidth demand due to the excessive frame rate of the scene-switch type video frame. While ensuring that the video quality remains basically unchanged, the peak bandwidth and the frame rate can be reduced, thereby providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a system architecture diagram of a video frame encoding method according to an embodiment of the present application;

[0072] Figure 2 This is a system architecture diagram of another video frame encoding method according to an embodiment of the present application;

[0073] Figure 3 This is a flowchart of the steps of a video frame encoding method of the present application;

[0074] Figure 4 This is a flowchart of another video frame encoding method of the present application;

[0075] Figure 5 This is a flowchart of another video frame encoding method of the present application;

[0076] Figure 6 This is a flowchart of another method for processing video conference data of the present application;

[0077] Figure 7 This is a flowchart of another method for processing live video data of the present application;

[0078] Figure 8 This is a structural block diagram of an embodiment of a video frame encoding device of the present application;

[0079] Figure 9 This is a structural block diagram of another embodiment of a video frame encoding device of the present application;

[0080] Figure 10 This is a structural block diagram of an embodiment of a video frame encoding device of the present application;

[0081] Figure 11 This is a structural block diagram of an embodiment of a video conferencing data processing device of the present application;

[0082] Figure 12 This is a structural block diagram of an embodiment of a device for processing live video data of the present application;

[0083] Figure 13 It is a structural diagram of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0085] In order to better illustrate the embodiments of the present application, the embodiments of the present application first introduce the basic coding principles.

[0086] In actual use, there are a large number of encoders and decoders, such as the H series, H263, H264, H265, H266 and other types of encoders and decoders, and the MPEG (Moving Picture Experts Group) series, such as MPEG-2 and MPEG-4 encoders. Of course, there are other types of encoders and decoders. Although there are different types of encoders, their basic principles are similar. Before introducing the basic principles of video encoding, let's first introduce the relevant technical terms:

[0087] I frame: Intra-coded picture, which is encoded using only the information of this frame without referring to other frames.

[0088] P frame: Predictive-coded Picture (predictive coded image frame), uses the previous I frame or P frame to perform inter-frame predictive coding using motion prediction.

[0089] B frame: Bidirectionally predicted picture (bidirectionally predicted coded image frame), provides the highest compression ratio. It requires both the previous image frame (I frame or P frame) and the subsequent image frame (P frame), and uses motion prediction to perform inter-frame bidirectional predictive coding.

[0090] IDR frame: Instantaneous Decoding Refresh. This frame starts encoding from the IDR frame, recalculating a new frame sequence. The first I-frame in a frame sequence is called an IDR frame. All frames following an IDR frame cannot reference the content of any frame preceding it. In contrast, for a regular I-frame, the B-frames and P-frames following it can reference the I-frame preceding it.

[0091] GOP: Group of picture, refers to the distance between two I frames.

[0092] The basic process of video encoding is as follows:

[0093] Grouping: Group several frames of images into one group (GOP). To prevent motion changes, the number of frames should not be too large.

[0094] Define frames: define each frame image in each group into three types, namely IDR frame (optional), I frame, B frame (optional) and P frame;

[0095] Predicted frame: Use IDR frame or I frame as base frame, use I frame to predict P frame, and then use I frame and P frame to predict B frame (optional);

[0096] Coding: Finally, the I frame data and the predicted difference information are encoded.

[0097] QP: Quantizer Parameter, which reflects the compression of spatial detail. Smaller values result in finer quantization, higher image quality, and a longer bitrate. A low QP preserves most detail; a larger QP reduces detail, lowers the bitrate, but also increases image distortion and quality.

[0098] Bit rate: The size of data encoded by the encoder per second, in kbps.

[0099] Smoothness parameter: The smoothness of the video can be set in a range of 0 to 100, with 0 being the lowest smoothness and 100 being the highest smoothness. Low smoothness results in relatively good image quality, but may lead to lags in poor network conditions. High smoothness results in relatively poor image quality, but may lead to less lags in poor network conditions. The parameters for smoothness are not limited in this embodiment.

[0100] Complexity: SAD between the current frame and the reference frame. SAD refers to the sum of the absolute values of the differences between the pixels of the current coding block and the predicted block.

[0101] The following takes the H264 encoder as an example to introduce the encoding principle in detail:

[0102] (1) First, each video frame to be encoded in the cache is divided into macroblocks;

[0103] (2) Then, the video frames in the buffer are grouped to obtain a GOP, and the I frame (or DIR frame) and P frame in the GOP are determined. Alternatively, the I frame (or DIR frame), B frame, and P frame in the GOP are determined.

[0104] The video frame sequence is divided into GOP groups; the group length can be preset.

[0105] (3) Then, an intra-frame prediction mode is selected based on the QP for the I frame in the GOP, and an inter-frame prediction mode is selected based on the QP for the P frame or the B frame, to obtain the first data after preliminary compression. The first data corresponds to the first matrix of the video frame.

[0106] (4) Performing a DCT discrete cosine transform (DCT) on the first data; after the DCT transformation, the first matrix becomes a second matrix, and the values in the second matrix are smaller than the values in the first matrix.

[0107] (5) The transformed second data is quantized using QP. After quantizing the second matrix, a third matrix is obtained, in which most of the third matrix becomes 0 and a small part becomes a non-zero value.

[0108] (6) Encode the QP-quantized data, such as entropy coding.

[0109] After encoding is complete, the encoded video frame data can be sent or stored. After the encoded video frame data is sent, the data associated with the video frame data in the cache is deleted from the cache. This data includes the original video frame, its predicted frame, and the encoded video frame. The principles of other video encoders are similar and will not be detailed here.

[0110] Based on the above coding principle, the inventors found during the research process that in scenarios with very short latency requirements such as video conferencing, live video broadcasting, and video chatting, if the scene in the video content of the video sender switches during the video process, the bit rate of the video frame sent by the video sender will be very high, resulting in a very high peak bandwidth. The probability of video jams when the video sender sends video data increases, and the probability of video jams when the video receiver receives the video data also increases. Based on this problem, the inventors of this application found that the video frames can be processed at the video sender, such as reducing the bit rate of scene switching type video frames and increasing the bit rate of normal video frames, so as to ensure that the overall bit rate changes are small. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality by a lot, and can also avoid the problem of high peak bandwidth requirements due to the excessive frame rate of scene switching type video frames.

[0111] Reference Figure 1 , which shows a system architecture diagram of a video frame encoding method provided in an embodiment of the present application.

[0112] like Figure 1 , including multiple video clients and server 101, multiple video clients such as Figure 1 In the video clients 102, 103, and 104, each video client can receive video data from other video clients through the server 101, and can also send video data to other video clients through the server 101. The term "multiple" can be understood as meaning at least two. Each video client may or may not have a camera device. The video data may be data collected by the camera device, or data collected from content displayed on the system desktop after the video client shares the system desktop.

[0113] The video client can be a personal computer, notebook, smart phone or other device, and the embodiments of the present application do not limit it.

[0114] Each video client can serve as both a video transmitter and a video receiver.

[0115] In the embodiment of the present application, if each video client wants to perform functions such as video conferencing, video live broadcast, and video chat, it can first establish a communication connection between each other through the server.

[0116] Reference Figure 2 , which shows a system architecture diagram of a video frame encoding method provided in an embodiment of the present application. Figure 2 In the video frame encoding process of the present application, the video sending end 102 sends the encoded video frame to the server 102, and the server 102 then sends the encoded video frame to the video receiving end 103, which decodes the video frame and plays it. The specific process can be as follows:

[0117] A10, the video sending end displays the smoothness parameter input control.

[0118] A11, the video sending end displays the fluency parameter in response to the input operation of the fluency parameter input control, obtaining the fluency parameter.

[0119] like Figure 2 After the user triggers to enter the video conference, video live broadcast, video chat and other interfaces 110, the video sending end can display the fluency parameter input control in the interface 110. The fluency parameter input control can be as follows: Figure 2 The slider 111 includes a slider button 112, and the user can adjust the fluency parameter by dragging the slider button. The video sending end can obtain the final fluency parameter ws after the user adjusts the fluency parameter input control.

[0120] Of course, in the embodiment of the present application, the setting of the fluency parameters can also be performed in other interfaces, such as the system setting interface of the instant messaging client, etc., and the embodiment of the present application does not limit it.

[0121] A12: The video transmitter obtains the video frame to be encoded.

[0122] In an embodiment of the present application, after a user activates a mode such as video conferencing, live video streaming, or video chat, the camera of the video transmitting end may be called to capture video frames. Of course, if the user of the video transmitting end chooses to share the system desktop or designate a window with the video receiving end, the video transmitting end obtains the video frames captured by the camera as the video frames to be encoded, or obtains the video frames captured from the system desktop or the designated window as the video frames to be encoded. Of course, the video frames to be encoded may also be video frames obtained by combining the video frames of the camera and the video frames of the system desktop, or the video frames to be encoded may also be video frames obtained by combining the video frames of the camera and the video frames of the designated window.

[0123] It should be noted that users can also use videos in some pan-entertainment scenarios as the videos to be encoded. Such pan-entertainment scenarios include the transmission of movies and TV series, the transmission of videos in gaming scenarios, etc. The video transmitter needs to transmit movies and TV series, in-progress game videos, and in-game transmission videos. It is understandable that in the scenario of transmitting movies and TV series, the video transmitter can be the server, of course, it can also be the client used by the user, and the user's client sends the locally played video to other clients through the server.

[0124] In practical applications, the embodiment of the present application can store the video frame to be encoded in the buffer of the encoder. In practical applications, a video frame can be collected and then stored in the buffer in the order of collection.

[0125] Of course, the video frames to be encoded are stored in the cache in the form of video frame sequences.

[0126] Of course, if the video sending end does not have a camera or the camera is damaged, the video data with an empty value can be directly sent out.

[0127] It can be understood that in the embodiment of the present application, the video sending end can continuously obtain the video frames to be encoded, and then continuously and cyclically execute the subsequent processes A13-A18 on the received video frames to be encoded.

[0128] A13. The video sending end determines a target frame in the video frames to be encoded; the target frame includes a scene switching type video frame.

[0129] In an embodiment of the present application, for the video frames to be encoded in the cache, it can be determined frame by frame in chronological order whether the video frames to be encoded are scene switching type video frames.

[0130] It can be understood that the condition of the video frame belonging to the scene switching type may be that the bit rate of the to-be-encoded video frame is greater than a preset multiple of the average bit rate of all unencoded video frames in the cache.

[0131] Of course, it can be understood that the condition for a video frame belonging to the scene switching type may be that the complexity of the to-be-encoded video frame is greater than a preset multiple of the average complexity of all unencoded video frames in the cache.

[0132] In practical applications, since frame-by-frame processing is possible, the average complexity can be obtained by multiplying the average complexity of the previous round by a preset attenuation factor, and then adding the product of the complexity of the current frame by another attenuation factor. For example, the average complexity can be determined by formula (1):

[0133] FrameCmplxMean i =FrameCmplxMean i-1 *ratio+current frame complexity*(1-ratio) Formula (1)

[0134] Among them, FrameCmplxMean i is the average complexity corresponding to the i-th video frame, FrameCmplxMean i-1 is the average complexity corresponding to the i-1th frame, and ratio is the attenuation factor. The ratio value range can be [0.0, 1.0]. The ratio can be set based on actual test results, for example, 0.5.

[0135] In the first frame of the entire video stream, its average complexity is the same as the complexity of the current frame. Starting from the second frame, since the intra-frame complexity of the second frame begins to change, the average complexity corresponding to the second frame and subsequent frames also changes.

[0136] It is understood that the intra-frame complexity of the current frame in formula (1) can also be replaced by the single-frame complexity of SAD. In this case, for the first video frame of the entire video stream, its corresponding average complexity can be the intra-frame complexity of the frame.

[0137] Of course, if the code rate is used for calculation, the complexity of the above formula can be replaced by the corresponding code rate, and the embodiments of the present application are not limited thereto.

[0138] In the embodiment of the present application, since complexity can better represent the complexity of the image content, using complexity to determine the aforementioned target frame can be more accurate. In addition, the calculated complexity and average complexity can be directly used in the subsequent encoding process, saving calculation time and reducing video latency.

[0139] It can be understood that after the target frame is determined, the process proceeds to A15.

[0140] It is understandable that the video frame to be encoded that is not the target frame may be a normal video frame. The normal video frame may directly proceed to step A17, and the quantization parameter of the normal video frame may be calculated by the encoder according to the code control model.

[0141] A14: The video transmitter reduces the bit rate of the target frame.

[0142] In the embodiment of the present application, for a scene switching type video frame that meets the above conditions, the bit rate is high, and the embodiment of the present application can first reduce the bit rate of the target frame. After the reduction, A15 and A18 can be entered respectively.

[0143] In practical applications, the normal bit rate of the target frame can be obtained first, and the pre-trained bit rate redistribution model can be used to calculate the bit rate reduction for the target frame. Then, the reduction amount is subtracted from the normal bit rate to obtain the remaining bit rate after the reduction.

[0144] The normal bit rate of the video frame can be calculated based on the size of the video frame. Of course, the normal bit rate of the video frame can also be predicted based on the original bit rate allocation model of the encoder. This embodiment of the present application does not limit it.

[0145] A15: The video transmitter determines whether to skip frames. If yes, it proceeds to A13; if not, it proceeds to A16.

[0146] In the embodiment of the present application, a frame skipping decision can be made for the target frame after the bit rate reduction according to a preset adjustment mechanism. If frame skipping is still required after the bit rate reduction, it indicates that the network environment of the video transmitter is particularly poor, and the target frame is skipped without the subsequent encoding process. If no adjustment is required after the bit rate reduction, the subsequent encoding process is performed.

[0147] A16. The video transmitter adjusts the quantization parameter for the target frame according to the fluency parameter and the reduced remaining bit rate;

[0148] A17. The video sending end encodes the reduced video frame according to the quantization parameter;

[0149] In the embodiment of the present application, the quantization parameter QP is adjusted according to the user-selected smoothness parameter and the remaining bit rate, and then the encoder uses the QP to perform the encoding process (3)-(6) described above. In actual applications, the present application can set up a new encoding thread in addition to the original encoding thread of the encoder. The new encoding thread is used to perform the process A10-A18, and after obtaining the adjusted QP, the original encoder thread is called to perform encoding based on the adjusted QP.

[0150] A18. The video sending end allocates the reduced bit rate to at least one to-be-encoded video frame except the target frame, and executes A13.

[0151] In an embodiment of the present application, the bit rate reduced from the scene switching type video frame can be allocated to other video frames to be encoded.

[0152] In practical applications, the reduced bit rate can be allocated to one or more normal video frames, or the bit rate can be allocated to all video frames to be encoded, and then the above process is performed on each video frame to be encoded.

[0153] It can be understood that since the bit rate of the target frame is allocated to one or more video frames to be encoded, if other frames are normal frames and are additionally allocated bit rates, the bit rate of the frame may be too high, and it may have the attributes of the aforementioned target frame. Therefore, the present application can cyclically execute the processes A13-A18 for the video frames to be encoded, so that the video frames to be encoded that may become the target frame attributes after being allocated bit rates can continue to be identified as target frames, and then the bit rate of the frame can continue to be reduced and reallocated.

[0154] A19: Video frames encoded by the video transmitter.

[0155] like Figure 2 , the video sending end can send the encoded video frame to the server 101, and the server 101 forwards the encoded video frame to the video receiving end.

[0156] A20, the video receiving end obtains the encoded video frame;

[0157] A21, a video receiving end decodes the encoded video frame;

[0158] A22. The video receiving end plays the decoded video frame.

[0159] The video receiving end can receive the video frame data of the video sending end from the server end 101, and then decode the encoded video frame based on the decoder corresponding to the encoder, and then play the decoded video frame.

[0160] It should be noted that, in the embodiment of the present application, similar processing can also be performed on audio data. For audio, the audio encoding process is as follows:

[0161] 1. Obtain analog signal;

[0162] 2. Input the analog signal into the sub-band filter and vocal cord model respectively;

[0163] 3. Assign a bit rate to the audio frame based on the output of the subband filter and the output of the vocal cord model;

[0164] 4. Quantize the output of the subband filter according to the allocated bit rate;

[0165] 5. Encode the quantized results.

[0166] The encoded data can then be sent to the server.

[0167] In practical applications, target frame criteria can be set for audio data, such as when the bitrate of an audio frame exceeds a set threshold. Based on this criteria, each audio frame is then determined to be a target frame. If it is, the bitrate of the frame is reduced, and the reduced bitrate is then quantized and encoded. The reduced bitrate is then assigned to subsequent audio frames. If it is not a target frame, the frame is quantized and encoded again. This process is repeated until the acquired audio is complete.

[0168] Among them, the set threshold value can correspond to the network environment information, and different network environment information can preset different threshold values. Of course, the aforementioned concept of fluency can also be used for audio, and the target audio frame in the audio frame can be judged based on the fluency parameter, and the target audio frame can be quantified based on the fluency parameter. The specific embodiments of the present application do not limit it. In the embodiments of the present application, first, the concept of fluency parameter is proposed. The fluency parameter can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the input interface of the fluency parameter. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0169] Secondly, for a video frame to be encoded, first determine whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of video frame processing, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0170] Again, since the quantization parameters of the target frame are adjusted based on the smoothness parameters and the remaining bit rate after reduction, when encoding the target frame, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0171] In addition, embodiments of the present application can be applied to a video transmitter, where the video frames are processed as described above at the video source to make the transmitted video frames as smooth as possible. This allows for a low-definition video source, such as video conferencing, to achieve a low-definition video freeze rate while ensuring clarity. Low latency can be understood as a delay less than a preset delay threshold, which can be set based on actual needs.

[0172] Reference Figure 3 , which shows a flowchart of the steps of a video frame encoding method according to an embodiment of the present application, which is applied to a video transmitting end. The method includes: steps 210 to 260.

[0173] Step 210: Obtain a video frame to be encoded;

[0174] In the embodiment of the present application, as described in A12 above, the video transmitter can continuously obtain the video to be encoded and store it in the cache.

[0175] Step 220, determining a target frame in the to-be-encoded video frames; the target frame includes a scene-change type video frame;

[0176] In the embodiment of the present application, the processing may be performed frame by frame in chronological order as in the aforementioned A13. Alternatively, after caching a plurality of video frames to be encoded, it may be determined synchronously whether each frame is the target frame.

[0177] Optionally, in an embodiment of the present application, step 220 includes:

[0178] Sub-step 2201: determining a target frame in the to-be-encoded video frames according to the preset fluency parameter.

[0179] In an embodiment of the present application, a smoothness parameter ws may be pre-set, and then subsequent processing may be performed under the limitation of the smoothness parameter to maximize clarity while reducing the frame rate under the provisions of the smoothness parameter.

[0180] In actual applications, if the video sending end sets a fluency parameter level and the user selects a certain level of fluency parameter, the embodiment of the present application obtains the fluency parameter of the corresponding level.

[0181] If the user is not provided with a fluency parameter setting interface, the default parameters of the application can be used.

[0182] Optionally, in an embodiment of the present application, before step 210, the following steps may be included: steps 201-202;

[0183] Step 201, displaying a fluency parameter input control;

[0184] Step 202: Acquire a fluency parameter in response to an input operation on the fluency parameter input control.

[0185] like Figure 2 The video transmitter may display a video display interface 110, which may include a fluency parameter input control. The user may operate the fluency parameter input control to input the desired fluency parameter based on the current network environment or device performance. The video transmitter may then obtain the desired fluency parameter through the fluency parameter input control.

[0186] Optionally, in an embodiment of the present application, the fluency parameter input control is a sliding control, and the input operation is a sliding operation. Figure 2 In FIG, the fluency parameter input control is a slide bar 111 , and a slide button 112 is included in the slide bar 111 . The user can operate the slide button 112 to adjust the fluency parameter. Figure 2 Slide right to reduce the smoothness, and the lowest value is 0. Figure 2 Slide left to increase the smoothness, up to 100. The lower the smoothness parameter, the higher the video quality, and the higher the smoothness parameter, the lower the video quality.

[0187] The embodiment of the present application proposes the concept of a fluency parameter, which can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the fluency parameter input interface. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0188] Optionally, in an embodiment of the present application, the fluency parameter input control is an input box, and the input operation is inputting in the input box.

[0189] Optionally, before step 210, the following steps may be included: steps 203-204;

[0190] Step 203: Obtain network environment information;

[0191] Step 204: Determine the fluency parameter according to the network environment information.

[0192] In an embodiment of the present application, the network environment information of the video sending end can be automatically identified, and then the fluency parameter can be determined based on the correspondence between the network bandwidth size and the fluency parameter. . Among them, the network environment information can be understood as the relevant parameters that affect the ability of the device to send and receive data in the network. The better the network environment information, the better the ability of the device to send and receive data. Conversely, the worse the network environment information, the worse the ability of the device to send and receive data. The network environment information can be at least one of the network bandwidth size, network stability, network jitter, network packet loss rate, etc. In this way, the fluency parameter can be automatically set according to the network environment, reducing the tediousness of user operations.

[0193] In practical applications, a mathematical model between the above-mentioned network environment information and the fluency parameter can be pre-set or trained. The input of the model is at least one item of the above-mentioned network environment information, and the output is the fluency parameter. In this way, a fluency parameter can be output for various network conditions.

[0194] Parameters such as the video network bandwidth, network stability, network jitter, and network packet loss rate can be detected by sending ping packets, the specific method of which is not limited in this embodiment of the application. Alternatively, a test plug-in can be pre-set and called to test the above network environment information, the specific detection method of which is not limited in this embodiment of the application.

[0195] Optionally, in an embodiment of the present application, the sub-step 2201 includes: sub-steps B11-B12

[0196] Sub-step B11, obtaining the single-frame complexity of the video frame to be encoded and the average complexity of the video frame to be encoded;

[0197] Sub-step B12: determining a target frame in the to-be-encoded video frames according to the single-frame complexity, the average complexity, and the smoothness parameter.

[0198] In an embodiment of the present application, the single-frame complexity may be the aforementioned SAD between the to-be-encoded video frame and the reference frame, and the sum of the absolute values of the differences between the target frame and the reference frame may be used as the single-frame complexity.

[0199] In the embodiment of the present application, for the video to be encoded, the aforementioned (1) and (2) can be performed to determine the type of GOP and each frame in each GOP. For an IDR frame, its reference frame is empty, and its complexity is the sum of its own pixel values. For an I frame or a P frame, its reference frame can be the frame before the frame. For a B frame, its reference frame can be the frame before the frame and the frame after the frame.

[0200] Then, the average complexity FrameCmplx for the target frame can be obtained. In practical applications, since frame-by-frame processing is possible, the average complexity FrameCmplxMean can be calculated using the aforementioned formula (1), without counting the received video frames, thus avoiding counter overflow due to excessive video frames.

[0201] Of course, the complexity of all the to-be-encoded video frames in the cache may also be accumulated, and then the accumulated value may be divided by the number of to-be-encoded videos in the cache. The embodiment of the present application does not limit the method of counting the average complexity.

[0202] After obtaining the single-frame complexity and average complexity of the video frame to be encoded, it can be combined with the aforementioned smoothness parameters to determine whether the video frame to be encoded is a target frame of the scene switching type, so that the judgment of the target frame can be more consistent with the smoothness parameters, and the bit rate of the subsequently encoded video can be more in line with the requirements of the smoothness parameters.

[0203] Optionally, the sub-step B12 includes:

[0204] Sub-step B121 , determining a target frame in the to-be-encoded video frames according to a comparison result between a first product of the single-frame complexity and the smoothness parameter and a second product of the average complexity and a preset coefficient.

[0205] In practical applications, the result of the aforementioned FrameCmplx*ws may be compared with the result of FrameCmplxMean*preset coefficient c to determine the target frame in the video frame to be encoded.

[0206] Here, c can be set according to actual test results, such as 3, and the embodiment of the present application does not limit it.

[0207] Optionally, sub-step B121 includes: sub-step B1211-sub-step B122,

[0208] Sub-step B1211: if the first product is greater than the second product, determining the to-be-encoded video frame as a target frame;

[0209] Sub-step B1212: When the first product is less than or equal to the second product, determine that the video frame to be encoded is a normal video frame.

[0210] For example, if FrameCmplx*ws>FrameCmplxMean*c, the frame is considered to be a scene switching type video frame.

[0211] If FrameCmplx*ws≤FrameCmplxMean*c, the frame is considered to be a normal video frame.

[0212] Of course, in the embodiment of the present application, other methods can also be used to determine the scene change type of video frame, and the embodiment of the present application is not limited thereto. For example, if FrameCmplx / FrameCmplxMean>a / ws, then the frame is considered to be a scene change type video frame. If FrameCmplx / FrameCmplxMean≤a / ws, then the frame is considered to be a normal video frame.

[0213] It should be noted that, for the convenience of calculation, the embodiment of the present application can also calculate the single-frame complexity scaling factor of the target frame, such as FrameCmplx / FrameCmplxMean.

[0214] For the target frame, the process proceeds to step 230. For the normal video frame, the quantization parameter is determined according to the coding model, and then the process proceeds to step 250. The coding model is, for example:

[0215] QP=(a*FrameCmplx / Bits)+b,

[0216] Among them, a and b are parameters trained through video frame samples; FrameCmplx is the single frame complexity of the frame, and Bits is the current bit rate of the frame for a normal video frame.

[0217] It should be noted that the current bit rate may be calculated according to the size of the frame as described in the above embodiment, or may be calculated using the bit rate calculation rule of the encoder.

[0218] Step 230, reducing the bit rate of the target frame;

[0219] This step is as described in A14 above, and the bit rate reduction of the target frame can be calculated to obtain the reduction amount and the remaining bit rate.

[0220] Optionally, the step 230 may include: sub-steps 231-232,

[0221] Sub-step 231, determining the frame type of the target frame; wherein different frame types correspond to different bit rate redistribution models;

[0222] Sub-step 232 : determining a rate reduction amount for the target frame according to the pre-rate reduction of the target frame, the single-frame complexity, the average complexity, and a rate redistribution model corresponding to the frame type.

[0223] In the embodiment of the present application, since there are multiple frame types, such as the aforementioned IDR frames, I frames, B frames, and P frames, different frame types have different bit rate reduction standards. If the same bit rate reduction method is used, the video quality of the frame after the reduction may be seriously degraded, thereby affecting the overall video quality. And since a GOP group may contain IDR frames, P frames, or I frames, P frames, or IDR frames, B frames, P frames, or I frames, B frames, P frames. Therefore, in the embodiment of the present application, different bit rate redistribution models are pre-trained for different types of video frames to calculate the bit rate reduction amount for that type of video frame, ensuring that after the reduction, the video of the frame can be as clear as possible while reducing the frame rate. For example, for IDR frames, an IDR frame bit rate redistribution model is trained; for I frames, an I frame bit rate redistribution model is trained; for B frames, a B frame bit rate redistribution model is trained; and for P frames, a P frame bit rate redistribution model is trained.

[0224] Therefore, for each target frame, the type of the target frame may be determined first, and then a corresponding rate redistribution model may be obtained based on the type.

[0225] In addition, according to the above method, the pre-reduction bit rate of the target frame, the single frame complexity, and the average complexity can be obtained, and the three can be used as inputs of the bit rate reallocation module to output the bit rate reduction amount.

[0226] In practical applications, for each frame type, the rate redistribution model can be trained based on the pre-rate reduction and complexity scaling factor of that frame type. Then, during use, the complexity scaling factor FrameCmplx / FrameCmplxMean and the pre-rate reduction are calculated as described above. The complexity scaling factor and pre-rate reduction are then input into the rate redistribution model to output the rate reduction.

[0227] It should be noted that when training the rate redistribution model, the smoothness parameter ws can also be used as a feature to train the rate redistribution model. Then, during use, the pre-reduction rate of the target frame, the single frame complexity, the average complexity, and the smoothness parameter ws can be used as model inputs to output the rate reduction amount, so as to obtain a rate reduction amount that better matches ws.

[0228] Optionally, in an embodiment of the present application, after step 230, the following steps may be included: steps 231-232;

[0229] Step 231: when the remaining bit rate of the reduced video frame is greater than a frame skipping threshold, skipping a target frame corresponding to the reduced video frame;

[0230] Step 232 , when the remaining bit rate of the reduced video frame is less than or equal to the frame skipping threshold, performs a step of adjusting the quantization parameter for the target frame according to the preset smoothness parameter and the reduced remaining bit rate.

[0231] The frame skipping threshold may correspond to the smoothness parameter ws. For example, a linear function may be used to associate the smoothness and the frame skipping threshold. When the user determines ws, the corresponding frame skipping threshold may be determined based on the corresponding relationship.

[0232] In an embodiment of the present application, the target frame can be further judged based on the bit rate reduction. If the bit rate of the frame still does not meet the smoothness requirement after the frame is reduced, the frame can be skipped without subsequent processing, thereby reducing the frame rate.

[0233] It should be noted that the embodiments of the present application may also adopt other frame skipping mechanisms, which are not limited by the embodiments of the present application.

[0234] Step 240: adjusting the quantization parameter for the target frame according to the preset fluency parameter and the reduced residual bit rate;

[0235] This step refers to the description of the aforementioned step A16 and will not be described in detail here.

[0236] Optionally, in an embodiment of the present application, step 240 may include: sub-steps 241-244;

[0237] Sub-step 241, determining a quantization offset parameter according to the smoothness parameter ws;

[0238] Sub-step 242, adjusting the range value of the quantization parameter according to the quantization offset parameter;

[0239] In the embodiment of the present application, a quantization offset parameter clip_qp_offset can be calculated based on ws, and then the maximum value iMaxFrameQp and the minimum value iMinFrameQp of the quantization parameter of the qp clamping mechanism are modified based on the clip_qp_offset. The qp clamping mechanism is to make the calculated qp between the maximum and minimum values, and its calculation formula can be as follows:

[0240] iMinFrameQp i =iMinFrameQp i-1 +clip_qp_offset;

[0241] iMaxFrameQp i =iMaxFrameQp i-1 +clip_qp_offset;

[0242] Among them, iMinFrameQp i The minimum qp value of the current i-th frame, iMinFrameQp i-1 is the minimum qp value of the i-1th frame.

[0243] iMaxFrameQp i The maximum value of qp for the current i-th frame, iMaxFrameQp i-1 is the maximum qp value of the i-1th frame, and clip_qp_offset is the quantization offset parameter of the i-th frame.

[0244] Optionally, in an embodiment of the present application, sub-step 241 may include: sub-steps 2411-2412;

[0245] Sub-step 2411, obtaining a complexity scaling factor, where the complexity scaling factor is a ratio of a single frame complexity of a target frame to an average complexity of the video frames to be encoded;

[0246] Sub-step 2412: input the smoothness parameter and the complexity scale factor into a preset offset parameter determination model to determine the quantization offset parameter.

[0247] In an embodiment of the present application, the aforementioned complexity scaling factor FrameCmplx / FrameCmplxMean may be obtained in the aforementioned process. Of course, the complexity scaling factor may also be obtained again in this step.

[0248] Then, the smoothness parameter and the complexity scale factor are input into a preset offset parameter determination model to determine the quantization offset parameter clip_qp_offset.

[0249] It can be understood that in the embodiment of the present application, the complexity scale factor samples of the sample frames can be extracted in advance based on the sample frames of the video frames under different ws, and then the offset parameter determination model can be trained with ws and complexity scale factor samples.

[0250] It should be noted that the quantization offset parameter may also be obtained in other ways, such as directly using ws as the quantization offset parameter, or multiplying the quantization offset parameter by a preset coefficient, which is not limited in the embodiments of the present application.

[0251] Sub-step 243, determining a first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate;

[0252] In an embodiment of the present application, for a target frame with a reduced bit rate, the first quantization parameter can be calculated based on the single-frame complexity of the target frame and the remaining bit rate after the reduction. The first quantization parameter can be understood as the QP before clamping. The QP may cause the image bit rate to be too large and not meet the smoothness requirements of WS.

[0253] Optionally, in an embodiment of the present application, sub-step 243 may include:

[0254] Sub-step 2431 : Input the single-frame complexity and the residual bit rate into a preset code control model to determine the first quantization parameter.

[0255] It should be noted that in the embodiment of the present application, the code control model can be pre-trained based on the single frame complexity and the residual bit rate of the sample frame. Then, during use, the single frame complexity and the residual bit rate are input into the code control model to obtain the first quantization parameter.

[0256] The code control model is as follows:

[0257] QP=(a*FrameCmplx / Bits)+b,

[0258] Where a and b are parameters trained using video frame samples; FrameCmplx is the single-frame complexity of the frame; Bits is the bitrate after the target frame is reduced; and Bits is the normally calculated bitrate for non-target frames.

[0259] Of course, the code control model of the embodiment of the present application is not limited to this model.

[0260] Sub-step 244: adjusting the first quantization parameter according to the range value to obtain a final quantization parameter.

[0261] After the first quantization parameter is calculated, the first quantization parameter is clamped through a clamping mechanism so that the final QP is between the adjusted maximum value iMaxFrameQp and the minimum value iMinFrameQp.

[0262] In practical applications, if the first quantization parameter is less than the minimum QP value, the minimum QP value can be used as the final QP. If the first quantization parameter is greater than the maximum QP value, the maximum QP value can be used as the final QP. When the first quantization parameter is greater than or equal to the minimum QP value and less than or equal to the maximum QP value, the first quantization parameter is the final QP. For example, if the maximum QP value of the QP clamping mechanism is 1000 and the minimum value is 500, if the calculated QP = 499, then the QP is adjusted to 500 and the video frame is encoded with 500; if the calculated QP = 1010, then the QP is adjusted to 1000 and the video frame is encoded with 1000.

[0263] In the embodiment of the present application, since a quantization offset parameter can be calculated according to the smoothness parameter, and then the maximum and minimum values of the quantization parameter QP of the QP clamping mechanism are adjusted based on the quantization offset parameter, when the subsequently calculated QP is clamped, a QP that better matches the WS can be obtained, so that the video frames encoded using the QP can be as clear as possible under the requirements of the smoothness parameter.

[0264] Step 250: Encode the reduced video frame according to the quantization parameter;

[0265] This step refers to the aforementioned A17 and will not be described in detail here.

[0266] Step 260 : Allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute step 120 .

[0267] In practical applications, if processing is performed frame by frame, to allocate a more reasonable bitrate, the reduced bitrate can be evenly distributed to all subsequent frames to be encoded. Of course, the complexity of subsequent frames can also be considered during this allocation, with weighted allocation based on the proportion of each frame's complexity to the total complexity. Higher complexity frames are allocated a higher bitrate, while lower complexity frames are allocated a lower bitrate. The total complexity is the sum of the complexities of all subsequent frames to be encoded.

[0268] It should be noted that the video frame samples used in the training of the above-mentioned models can be the same video frame samples. The same video frame samples are used to extract the parameters required by each model respectively, so that the above-mentioned models can be coordinated more accurately and the results obtained can also be more accurate.

[0269] It can be understood that the above models can all be understood as mathematical models. Mathematical models are scientific or engineering models constructed using mathematical logic and mathematical language. A mathematical model is a mathematical structure that uses mathematical language to describe, in a generalized or approximate manner, the characteristics or quantitative dependencies of a system. This mathematical structure is the pure relational structure of a system depicted with the aid of mathematical symbols. A mathematical model can be one or a group of algebraic equations, differential equations, difference equations, integral equations, or statistical equations, or any combination thereof, that quantitatively or qualitatively describe the interrelationships or causal relationships between variables in the system. In addition to mathematical models described using equations, there are also models described using other mathematical tools, such as algebra, geometry, topology, and mathematical logic. Mathematical models describe the behavior and characteristics of a system rather than its actual structure.

[0270] In an embodiment of the present application, for a video frame to be encoded, it is first determined whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of processing the video frame, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0271] Again, since the quantization parameters of the target frame are adjusted based on the smoothness parameters and the remaining bit rate after reduction, when encoding the target frame, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0272] In addition, embodiments of the present application can be applied to a video transmitter, where the video frames are processed as described above at the video source to make the transmitted video frames as smooth as possible. This allows for a low-definition video source, such as video conferencing, to achieve a low-definition video freeze rate while ensuring clarity. Low latency can be understood as a delay less than a preset delay threshold, which can be set based on actual needs.

[0273] Reference Figure 4, which shows a flowchart of the steps of a video frame encoding method according to an embodiment of the present application, which is applied to a video sending end. The method includes: steps 310 to 360.

[0274] Step 310: Obtain a video frame to be encoded;

[0275] This step refers to the aforementioned step 210 and will not be described in detail here.

[0276] Step 320: determining a target frame in the to-be-encoded video frames; the target frame includes a scene-change type video frame;

[0277] This step refers to the aforementioned step 220 and will not be described in detail here.

[0278] Step 330: reducing the bit rate of the target frame;

[0279] This step refers to the aforementioned step 230 and will not be described in detail here.

[0280] Step 340: Allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and perform the step of determining the target frame among the to-be-encoded video frames.

[0281] Step 340 may refer to the description of the aforementioned step 260 and will not be described in detail here.

[0282] Optionally, after step 330, the following steps may be further included:

[0283] Step 350: Encode the target frame based on the remaining bit rate after the reduction;

[0284] Optionally, in an embodiment of the present application, before step 310, the following steps may be further included:

[0285] Step 301, obtaining fluency parameters;

[0286] This step can refer to the aforementioned steps 201-202, or steps 203-204 to obtain the fluency parameter, which will not be described in detail here.

[0287] Optionally, in an embodiment of the present application, step 350 includes: sub-steps 351-352;

[0288] Sub-step 351, adjusting the quantization parameter for the target frame according to the fluency parameter and the reduced residual bit rate;

[0289] Sub-step 352: Encode the reduced video frame according to the quantization parameter.

[0290] Sub-steps 351-352 can refer to the description of the aforementioned steps 240-250 and will not be described in detail here.

[0291] Of course, in the embodiment of the present application, the qp can also be calculated directly based on the remaining bit rate after the reduction, and then encoding can be performed.

[0292] The execution process of the relevant steps of the embodiment of the present application can refer to the aforementioned Figure 3 The embodiments are not described in detail here.

[0293] In an embodiment of the present application, for a video frame to be encoded, it is first determined whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of processing the video frame, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible. In addition, if the quantization parameter of the target frame is adjusted based on the smoothness parameter and the remaining bit rate after reduction, when the target frame is encoded, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0294] Reference Figure 5 , which shows a flowchart of the steps of a video frame encoding method according to an embodiment of the present application, which is applied to a server. The method includes: steps 410 to 460.

[0295] Step 410: obtaining a first video stream sent by a video transmitter and obtaining a fluency parameter of a video receiver;

[0296] Step 420: Decode the first video stream to obtain a video frame to be encoded;

[0297] Step 430: determining a target frame in the to-be-encoded video frames; the target frame includes a scene-change type video frame;

[0298] Step 440: reducing the bit rate of the target frame;

[0299] Step 450: adjusting the quantization parameter for the target frame according to the preset fluency parameter and the reduced residual bit rate;

[0300] Step 460: Encode the reduced video frame according to the quantization parameter, and send the encoded second video stream to the video receiving end;

[0301] Step 470 : Allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and then repeat step 430 .

[0302] The embodiment of the present application differs from the previous embodiment in that the execution entity is the server. Then, in step 410, the server receives the first encoded video stream sent by the video sending end and receives the fluency parameter provided by the video receiving end. Then, in step 420, there is an additional process of decoding the first video stream, and the decoded video frames are used as the video frames to be encoded. Then, in step 470, there is an additional step of sending the encoded video stream directly to the video receiving end. The other execution processes are similar to the principles of the previous embodiment and will not be described in detail here.

[0303] In the embodiments of the present application, first, the concept of a fluency parameter is proposed, and the fluency parameter can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the fluency parameter input interface. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0304] Secondly, for a video frame to be encoded, first determine whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of video frame processing, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0305] Again, since the quantization parameters of the target frame are adjusted based on the smoothness parameters and the remaining bit rate after reduction, when encoding the target frame, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0306] In addition, the embodiments of the present application can be applied to the server side. At the server side, the video frames are processed as described above so that the sent video frames can be as smooth as possible, thereby enabling the video receiving end to reduce the frame rate while ensuring the video clarity as much as possible while meeting its fluency requirements.

[0307] It should be noted that Figure 4 The embodiment can also be referred to Figure 5 The embodiment is applied to the server and will not be described in detail here.

[0308] Reference Figure 6 , which shows a flowchart of the steps of a method for processing video conference data according to an embodiment of the present application, which is applied to a video sending end. The method includes: steps 510 to 516.

[0309] Step 510: Obtain a video frame to be encoded in the video conference data;

[0310] In the embodiment of the present application, if a user wants to conduct a video conference with other users, he can use a video client A through Figure 1 The architecture can be used to conduct a video conference with one or more other users' video clients B. During the video conference, for the video data captured by the camera of the video client A, the video data of the shared desktop, or the video data of the shared window, the embodiment of the present application can obtain the video frames in the above video data as the video frames to be encoded, and then after executing the subsequent steps, send the encoded video data to the server. The server then sends the encoded video data to the video clients of the other participants in the video conference. The specific method of sending the encoded video data to the video clients of the other participants in the video conference is not limited by the embodiment of the present application.

[0311] In the embodiment of the present application, as described in A12 above, the video transmitter can continuously obtain the video to be encoded from the video conference data and store it in the cache. Step 512: determine the target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0312] In the embodiment of the present application, the processing may be performed frame by frame in chronological order as in the aforementioned A13. Alternatively, after caching a plurality of video frames to be encoded, it may be determined synchronously whether each frame is the target frame.

[0313] If it is a target frame, the process proceeds to step 514. If it is not a target frame, the QP can be directly calculated according to the formula of the aforementioned coding model, and then the frame is encoded according to the QP.

[0314] Optionally, determining the target frame in the to-be-encoded video frames includes:

[0315] Sub-step 5121 , determining a target frame in the to-be-encoded video frames according to a preset fluency parameter.

[0316] In an embodiment of the present application, a smoothness parameter ws may be pre-set, and then subsequent processing may be performed under the limitation of the smoothness parameter to maximize clarity while reducing the frame rate under the provisions of the smoothness parameter.

[0317] In actual applications, if the video sending end sets a fluency parameter level and the user selects a certain level of fluency parameter, the embodiment of the present application obtains the fluency parameter of the corresponding level.

[0318] If the user is not provided with a fluency parameter setting interface, the default parameters of the application can be used.

[0319] Optionally, in an embodiment of the present application, before step 510, the following steps may be further included:

[0320] Step D10: Obtain fluency parameters.

[0321] This step can refer to the aforementioned steps 201-202, or steps 203-204 to obtain the fluency parameter, which will not be described in detail here.

[0322] In an embodiment of the present application, the fluency parameter set by the video sending end can be obtained.

[0323] Optionally, step D10 may include: sub-step D101 - sub-step D102.

[0324] Sub-step D101, displaying a fluency parameter input control;

[0325] Sub-step D101 : acquiring a fluency parameter in response to an input operation on the fluency parameter input control.

[0326] like Figure 2 The video transmitter may display a video display interface 110, which may include a fluency parameter input control. The user may operate the fluency parameter input control to input the desired fluency parameter based on the current network environment or device performance. The video transmitter may then obtain the desired fluency parameter through the fluency parameter input control.

[0327] Optionally, in an embodiment of the present application, the fluency parameter input control is a sliding control, and the input operation is a sliding operation. Figure 2 In FIG, the fluency parameter input control is a slide bar 111 , and a slide button 112 is included in the slide bar 111 . The user can operate the slide button 112 to adjust the fluency parameter. Figure 2Slide right to reduce the smoothness, and the lowest value is 0. Figure 2 Slide left to increase the smoothness, up to 100. The lower the smoothness parameter, the higher the video quality, and the higher the smoothness parameter, the lower the video quality.

[0328] The embodiment of the present application proposes the concept of a fluency parameter, which can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the fluency parameter input interface. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0329] Optionally, in an embodiment of the present application, the fluency parameter input control is an input box, and the input operation is inputting in the input box.

[0330] Optionally, in this embodiment of the present application, as in the process of steps 203 and 204 above, network environment information may be obtained before step 510, and then the fluency parameter may be determined based on the network environment information. For the specific principles, refer to the description of steps 203 and 204 above. In this embodiment of the present application, the principles of step 512 are similar to those of step 220 above. Step 512 can refer to the description of step 220 in the above embodiment and will not be described in detail here.

[0331] Step 514: reducing the bit rate of the target frame;

[0332] This step is as described in A14 above, and can calculate the bit rate reduction of the target frame to obtain the reduction amount and the remaining bit rate.

[0333] In the embodiment of the present application, the principle of step 514 is similar to that of the aforementioned step 230. Step 514 can refer to the description of the aforementioned step 230 and will not be described in detail here.

[0334] Step 516 : Allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute step 512 .

[0335] In practical applications, if processing is performed frame by frame, to allocate a more reasonable bitrate, the reduced bitrate can be evenly distributed to all subsequent frames to be encoded. Of course, the complexity of subsequent frames can also be considered during this allocation, with weighted allocation based on the proportion of each frame's complexity to the total complexity. Higher complexity frames are allocated a higher bitrate, while lower complexity frames are allocated a lower bitrate. The total complexity is the sum of the complexities of all subsequent frames to be encoded.

[0336] In the embodiment of the present application, step 516 can refer to the description of the aforementioned step 260 and will not be described in detail here.

[0337] Optionally, after step 514, the following steps may be further included:

[0338] Step 518: Encode the target frame based on the remaining bit rate after the reduction.

[0339] Optionally, in one embodiment of the present application, step 518 includes: sub-steps 5181-5182;

[0340] Sub-step 5181: adjusting the quantization parameter for the target frame according to the fluency parameter and the reduced residual bit rate;

[0341] This step refers to the description of the aforementioned step A16 and will not be described in detail here.

[0342] Optionally, the sub-step 5181 may include: sub-steps K11 to K14

[0343] Sub-step K11, determining a quantization offset parameter according to the fluency parameter;

[0344] Sub-step K12, adjusting the range value of the quantization parameter according to the quantization offset parameter;

[0345] In the embodiment of the present application, a quantization offset parameter clip_qp_offset can be calculated based on ws, and then the maximum value iMaxFrameQp and the minimum value iMinFrameQp of the quantization parameter of the qp clamping mechanism are modified based on the clip_qp_offset. The qp clamping mechanism is to make the calculated qp between the maximum and minimum values, and its calculation formula can be as follows:

[0346] iMinFrameQp i =iMinFrameQp i-1 +clip_qp_offset;

[0347] iMaxFrameQp i =iMaxFrameQp i-1 +clip_qp_offset;

[0348] Among them, iMinFrameQp i The minimum qp value of the current i-th frame, iMinFrameQp i-1 is the minimum qp value of the i-1th frame.

[0349] iMaxFrameQp i The maximum value of qp for the current i-th frame, iMaxFrameQp i-1is the maximum qp value of the i-1th frame, and clip_qp_offset is the quantization offset parameter of the i-th frame.

[0350] Optionally, in an embodiment of the present application, sub-step K11 may include: sub-steps K111-K112;

[0351] Sub-step K111, obtaining a complexity scaling factor, where the complexity scaling factor is a ratio of a single frame complexity of a target frame to an average complexity of the video frames to be encoded;

[0352] In sub-step K112, the fluency parameter and the complexity scaling factor are input into a preset offset parameter determination model to determine the quantization offset parameter.

[0353] In an embodiment of the present application, the aforementioned complexity scaling factor FrameCmplx / FrameCmplxMean may be obtained in the aforementioned process. Of course, the complexity scaling factor may also be obtained again in this step.

[0354] Then, the smoothness parameter and the complexity scale factor are input into a preset offset parameter determination model to determine the quantization offset parameter clip_qp_offset.

[0355] It can be understood that in the embodiment of the present application, the complexity scale factor samples of the sample frames can be extracted in advance based on the sample frames of the video frames under different ws, and then the offset parameter determination model can be trained with ws and complexity scale factor samples.

[0356] It should be noted that the quantization offset parameter may also be obtained in other ways, such as directly using ws as the quantization offset parameter, or multiplying the quantization offset parameter by a preset coefficient, which is not limited in the embodiments of the present application.

[0357] Sub-step K13, determining a first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate;

[0358] In an embodiment of the present application, for a target frame with a reduced bit rate, the first quantization parameter can be calculated based on the single-frame complexity of the target frame and the remaining bit rate after the reduction. The first quantization parameter can be understood as the QP before clamping. The QP may cause the image bit rate to be too large and not meet the smoothness requirements of WS.

[0359] Optionally, in an embodiment of the present application, sub-step K13 may include:

[0360] Sub-step K131 , inputting the single-frame complexity and the residual bit rate into a preset code control model to determine the first quantization parameter.

[0361] It should be noted that in the embodiment of the present application, the code control model can be pre-trained based on the single frame complexity and the residual bit rate of the sample frame. Then, during use, the single frame complexity and the residual bit rate are input into the code control model to obtain the first quantization parameter.

[0362] The code control model is as follows:

[0363] QP=(a*FrameCmplx / Bits)+b,

[0364] Where a and b are parameters trained using video frame samples; FrameCmplx is the single-frame complexity of the frame; Bits is the bitrate after the target frame is reduced; and Bits is the normally calculated bitrate for non-target frames.

[0365] Of course, the code control model of the embodiment of the present application is not limited to this model.

[0366] Sub-step K14: adjusting the first quantization parameter according to the range value to obtain a final quantization parameter.

[0367] After the first quantization parameter is calculated, the first quantization parameter is clamped through a clamping mechanism so that the final QP is between the adjusted maximum value iMaxFrameQp and the minimum value iMinFrameQp.

[0368] In practical applications, if the first quantization parameter is less than the minimum QP value, the minimum QP value can be used as the final QP. If the first quantization parameter is greater than the maximum QP value, the maximum QP value can be used as the final QP. When the first quantization parameter is greater than or equal to the minimum QP value and less than or equal to the maximum QP value, the first quantization parameter is the final QP. For example, if the maximum QP value of the QP clamping mechanism is 1000 and the minimum value is 500, if the calculated QP = 499, then the QP is adjusted to 500 and the video frame is encoded with 500; if the calculated QP = 1010, then the QP is adjusted to 1000 and the video frame is encoded with 1000.

[0369] In the embodiment of the present application, since a quantization offset parameter can be calculated according to the smoothness parameter, and then the maximum and minimum values of the quantization parameter QP of the QP clamping mechanism are adjusted based on the quantization offset parameter, when the subsequently calculated QP is clamped, a QP that better matches the WS can be obtained, so that the video frames encoded using the QP can be as clear as possible under the requirements of the smoothness parameter.

[0370] Sub-step 5182, encoding the reduced video frame according to the quantization parameter.

[0371] This step refers to the aforementioned A17 and will not be described in detail here.

[0372] In the embodiment of the present application, sub-steps 5181-5182 can refer to the description of the aforementioned steps 240-250 and will not be described in detail here.

[0373] Of course, in the embodiment of the present application, the qp can also be calculated directly based on the remaining bit rate after the reduction, and then encoding can be performed.

[0374] It should be noted that if the video conference ends, the above steps may no longer be performed.

[0375] The following is an example to illustrate the above cycle process:

[0376] For example, in step 510 , video frames A1 , A2 , A3 , A4 . . . A10 of video conference frames are obtained during the video conference, and these frames are used as video frames to be encoded.

[0377] During the first loop, for example, frame A1 is determined as the target frame in step 512. Then, in step 514, the bitrate m of frame A1 is reduced. Then, in step 516, m is allocated to one or more of the remaining video frames A2, A3, A4, ..., A10 according to the aforementioned rules. The next loop then begins for video frames A2, A3, A4, ..., A10. Of course, if new video frames are added, they are also added to the next loop. After frame A1's bitrate is reduced, it can be encoded using the aforementioned method and output.

[0378] For video frames A2, A3, A4, ..., A10, if it is confirmed in step 512 that frame A2 is not the target frame, the target frame can be encoded in the aforementioned manner and then output. Video frames A3, A4, ..., A10 enter the next cycle.

[0379] As in the above-mentioned loop processing process, the acquired video frames to be encoded are continuously processed until the video conference ends.

[0380] In an embodiment of the present application, for a video frame to be encoded generated by a video transmitter in a video conference, it can first be determined whether it is a scene-switch type video frame. If the video frame to be encoded is a scene-switch type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded. The above process is then performed on the next video to be encoded. In this way, during the processing of the video frame, the bit rate of a video frame that is itself a scene-switch type, or a video frame that becomes a scene-switch type after being assigned a new bit rate, will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene-switch type video frame is reduced and the bit rate of the normal video frame is increased, which can minimize the overall bit rate change. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not significantly reduce the evaluation of video quality, and can also avoid the problem of high peak bandwidth demand caused by excessive frame rate of scene-switch type video frames. While ensuring that the video quality remains basically unchanged, it can reduce the peak bandwidth and reduce the frame rate, thereby providing a better user experience. In addition, if the quantization parameter of the target frame is adjusted based on the smoothness parameter and the remaining bit rate after reduction, when the target frame is encoded, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0381] Reference Figure 7 , which shows a flowchart of the steps of a method for processing live video data according to an embodiment of the present application, which is applied to a video sending end. The method includes: steps 520 to 526.

[0382] Step 520: Obtain the video frame to be encoded in the live video data;

[0383] In the embodiment of the present application, if user 1 wants to broadcast live and other users want to watch the live broadcast of user A, user 1 can use a video client A to initiate the live broadcast, and the live broadcast data is transmitted through the following methods: Figure 1 The server of the architecture sends the video data to the video client B of one or more other users who are watching the live broadcast. Then, during the live video broadcast, for the video data collected by the camera of the video client A, the video data of the shared desktop, or the video data of the shared window, the embodiment of the present application can obtain the video frames in the above video data as the video frames to be encoded, and then after executing the subsequent steps, send the encoded video data to the server. The server then sends the encoded video data to the video client of the viewer who is watching the live video. The specific method of sending the encoded video data to the video client of the viewer is not limited by the embodiment of the present application.

[0384] Step 522, determining a target frame in the video frames to be encoded; the target frame includes a scene switching type video frame;

[0385] Step 524, reducing the bit rate of the target frame;

[0386] Step 526: Allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and perform the step of determining the target frame among the to-be-encoded video frames.

[0387] Optionally, after step 524, the following steps may be further included:

[0388] Step 528: Encode the target frame based on the remaining bit rate after the reduction;

[0389] Optionally, in an embodiment of the present application, before step 520, the following steps may be further included:

[0390] Step E10, obtaining a fluency parameter;

[0391] This step can refer to the aforementioned steps 201-202, or steps 203-204 to obtain the fluency parameter, which will not be described in detail here.

[0392] Optionally, in an embodiment of the present application, step 528 includes: sub-steps 5281-5282;

[0393] Sub-step 5281: adjusting the quantization parameter for the target frame according to the fluency parameter and the reduced residual bit rate;

[0394] Sub-step 5282: Encode the reduced video frame according to the quantization parameter.

[0395] Sub-steps 5281-5282 can refer to the description of the aforementioned steps 240-250 and will not be described in detail here.

[0396] Of course, in the embodiment of the present application, the qp can also be calculated directly based on the remaining bit rate after the reduction, and then encoding can be performed.

[0397] It should be noted that if the live video broadcast ends, the above steps may no longer be performed.

[0398] In an embodiment of the present application, for a video frame to be encoded generated during a live video broadcast at a video transmitter, it can first be determined whether it is a scene-switch type video frame. If the video frame to be encoded is a scene-switch type video frame, the bit rate of the video frame is reduced, and the reduced bit rate is allocated to other videos to be encoded. The above process is then performed on the next video to be encoded. In this way, during the video frame processing, a video frame that is itself a scene-switch type video frame, or a video frame that becomes a scene-switch type video frame after being assigned a new bit rate, will have its bit rate reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene-switch type video frame is reduced and the bit rate of the normal video frame is increased, thereby minimizing the overall bit rate change. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application does not significantly reduce the video quality evaluation, and can also avoid the problem of high peak bandwidth demand caused by excessive frame rate of scene-switch type video frames. While ensuring that the video quality remains basically unchanged, the peak bandwidth and the frame rate can be reduced, thereby providing a better user experience. In addition, if the quantization parameter of the target frame is adjusted based on the smoothness parameter and the remaining bit rate after reduction, when the target frame is encoded, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0399] Reference Figure 8 , which shows a structural block diagram of an embodiment of a video frame encoding device according to an embodiment of the present application, applied to a video transmitter. The device includes:

[0400] A first video frame acquisition module 610 is used to acquire a video frame to be encoded;

[0401] A first frame detection module 612 is configured to determine a target frame in the video frame to be encoded; the target frame includes a scene-change type video frame;

[0402] A first bit rate reduction module 613, configured to reduce the bit rate of the target frame;

[0403] A first quantization parameter adjustment module 614 is configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and the reduced residual bit rate;

[0404] a first encoding module 615, configured to encode the reduced video frame according to the quantization parameter;

[0405] The first bit rate reallocation module 616 is configured to allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and enter the first frame detection module.

[0406] Optionally, the first frame detection module includes:

[0407] The frame detection submodule is used to determine the target frame in the video frames to be encoded according to the preset fluency parameter.

[0408] Optionally, the frame detection submodule includes:

[0409] A complexity acquisition submodule, configured to acquire the single frame complexity of the to-be-encoded video frame and the average complexity of the to-be-encoded video frame;

[0410] The first frame detection submodule is configured to determine a target frame in the to-be-encoded video frames according to the single-frame complexity, the average complexity, and the smoothness parameter.

[0411] Optionally, the first frame detection submodule includes:

[0412] The complexity comparison submodule is used to determine the target frame in the video frames to be encoded based on a comparison result between a first product of the single frame complexity and the smoothness parameter and a second product of the average complexity and a preset coefficient.

[0413] Optionally, the complexity comparison submodule is specifically used to:

[0414] When the first product is greater than the second product, determining the to-be-encoded video frame as a target frame;

[0415] When the first product is less than or equal to the second product, it is determined that the to-be-encoded video frame is a normal video frame.

[0416] Optionally, the first bit rate reduction module includes:

[0417] a frame type determination submodule, configured to determine the frame type of the target frame; wherein different frame types correspond to different bit rate redistribution models;

[0418] The bit rate reduction amount determination module is used to determine the bit rate reduction amount of the target frame according to the pre-reduction bit rate of the target frame, the single frame complexity, the average complexity and the bit rate redistribution model corresponding to the frame type.

[0419] Optionally, the first quantization parameter adjustment module includes:

[0420] An offset parameter determination submodule, configured to determine a quantization offset parameter according to the fluency parameter;

[0421] a range value determining module, configured to adjust the range value of the quantization parameter according to the quantization offset parameter;

[0422] A first quantization parameter determination module, configured to determine a first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate;

[0423] The quantization parameter clamping module is used to adjust the first quantization parameter according to the range value to obtain a final quantization parameter.

[0424] Optionally, the offset parameter determination submodule is specifically configured to:

[0425] Obtaining a complexity scaling factor, where the complexity scaling factor is a ratio of a single frame complexity of a target frame to an average complexity of the to-be-encoded video frames;

[0426] The fluency parameter and the complexity scale factor are input into a preset offset parameter determination model to determine the quantization offset parameter.

[0427] Optionally, the first quantization parameter determination module is specifically configured to:

[0428] The single frame complexity and the residual bit rate are input into a preset code control model to determine the first quantization parameter.

[0429] Optionally, it also includes:

[0430] The frame skipping judgment module is used to skip the target frame corresponding to the cut video frame when the remaining bit rate of the cut video frame is greater than the frame skipping threshold; and execute the first quantization parameter adjustment module when the remaining bit rate of the cut video frame is less than or equal to the frame skipping threshold.

[0431] Optionally, it also includes:

[0432] Fluency control display module, used to display fluency parameter input controls;

[0433] The fluency parameter acquisition module is configured to acquire a fluency parameter in response to an input operation on the fluency parameter input control.

[0434] Optionally, the fluency parameter input control is a sliding control, and the input operation is a sliding operation; or the fluency parameter input control is an input box, and the input operation is inputting in the input box.

[0435] Optionally, it also includes:

[0436] Network environment acquisition module, used to obtain network environment information;

[0437] The fluency determination module is configured to determine the fluency parameter according to the network environment information.

[0438] In the embodiments of the present application, first, the concept of a fluency parameter is proposed, and the fluency parameter can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the fluency parameter input interface. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0439] Secondly, for a video frame to be encoded, first determine whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of video frame processing, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0440] Again, since the quantization parameters of the target frame are adjusted based on the smoothness parameters and the remaining bit rate after reduction, when encoding the target frame, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0441] In addition, embodiments of the present application can be applied to a video transmitter, where the video frames are processed as described above at the video source to make the transmitted video frames as smooth as possible. This allows for a low-definition video source, such as video conferencing, to achieve a low-definition video freeze rate while ensuring clarity. Low latency can be understood as a delay less than a preset delay threshold, which can be set based on actual needs.

[0442] Reference Figure 9 , which shows a structural block diagram of an embodiment of a bit rate control device according to an embodiment of the present application, applied to a video transmitter. The device includes:

[0443] The second video frame acquisition module 621 is used to acquire the video frame to be encoded;

[0444] A second frame detection module 622 is configured to determine a target frame in the to-be-encoded video frame; the target frame includes a scene-change type video frame;

[0445] A second bit rate reduction module 623, configured to reduce the bit rate of the target frame;

[0446] The second bit rate reallocation module 624 is configured to allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and execute the second frame detection module.

[0447] Optionally, it also includes:

[0448] A second encoding module, configured to encode the target frame based on the bit rate remaining after the reduction;

[0449] Optionally, it also includes:

[0450] A first fluency acquisition module, configured to acquire fluency parameters;

[0451] The second encoding module 625 is further configured to adjust a quantization parameter for the target frame according to the smoothness parameter and the reduced residual bit rate; and encode the reduced video frame according to the quantization parameter.

[0452] The other functional modules of the embodiment of the present application are similar to those of the aforementioned embodiment and will not be described in detail here.

[0453] In an embodiment of the present application, for a video frame to be encoded, it is first determined whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of processing the video frame, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0454] Reference Figure 10 , which shows a structural block diagram of an embodiment of a video frame encoding device according to an embodiment of the present application, applied to a server. The device includes:

[0455] The third acquisition module 631 is used to acquire the first video stream sent by the video sending end and acquire the fluency parameter of the video receiving end;

[0456] A decoding module 632 is configured to decode the first video stream to obtain a video frame to be encoded;

[0457] The third frame detection module 633 is configured to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0458] a third bit rate reduction module 634, configured to reduce the bit rate of the target frame;

[0459] A third quantization parameter adjustment module 635 is configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and the reduced residual bit rate;

[0460] a third encoding module 636, configured to encode the reduced video frame according to the quantization parameter, and send the encoded second video stream to a video receiving end;

[0461] The third bit rate reallocation module 637 is configured to allocate the reduced bit rate to at least one to-be-encoded video frame except the target frame, and execute the third frame detection module.

[0462] It should be noted that the other functional modules of the embodiments of the present application can refer to the description of the aforementioned embodiments and will not be described in detail here.

[0463] In the embodiments of the present application, first, the concept of a fluency parameter is proposed, and the fluency parameter can be provided to the user as an interface for operation, such as the aforementioned fluency parameter control and the fluency parameter input interface. The user can use the fluency parameter control to achieve infinite sliding between clear mode and smooth mode.

[0464] Secondly, for a video frame to be encoded, first determine whether it is a scene switching type video frame. If the video frame to be encoded is a scene switching type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded, and then the above process is performed on the next video to be encoded. In this way, since in the process of video frame processing, the video frame itself is a scene switching type video frame, or a video frame that becomes a scene switching type after being assigned a new bit rate, the bit rate of the above video frame will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene switching type video frame is reduced and the bit rate of the normal video frame is increased, which can ensure that the overall bit rate change is as small as possible. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not reduce the evaluation of video quality much, and can make the freeze rate low while ensuring the clarity as much as possible.

[0465] Again, since the quantization parameters of the target frame are adjusted based on the smoothness parameters and the remaining bit rate after reduction, when encoding the target frame, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0466] In addition, the embodiments of the present application can be applied to the server side. At the server side, the video frames are processed as described above so that the sent video frames can be as smooth as possible, thereby enabling the video receiving end to reduce the frame rate while ensuring the video clarity as much as possible while meeting its fluency requirements.

[0467] Reference Figure 11 , which shows a structural block diagram of an embodiment of a video conferencing data processing device according to an embodiment of the present application, which is applied to a video transmitting end. The device includes:

[0468] The fourth acquisition module 641 is used to obtain the video frame to be encoded in the video conference data;

[0469] The fourth frame detection module 642 is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame;

[0470] A fourth bit rate reduction module 643, configured to reduce the bit rate of the target frame;

[0471] The fourth bit rate reallocation module 644 is configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fourth frame detection module.

[0472] Optionally, it also includes:

[0473] a fourth encoding module, configured to encode the target frame based on the bit rate remaining after the subtraction;

[0474] Optionally, it also includes:

[0475] a fourth encoding module, configured to encode the target frame based on the bit rate remaining after the reduction;

[0476] Optionally, the fourth encoding module includes:

[0477] a fourth quantization parameter adjustment submodule, configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and the reduced residual bit rate;

[0478] The fourth encoding submodule is configured to encode the reduced video frame according to the quantization parameter.

[0479] Optionally, the fourth quantization parameter adjustment submodule is also used to determine a quantization offset parameter based on the smoothness parameter; adjust the range value of the quantization parameter based on the quantization offset parameter; determine the first quantization parameter based on the single-frame complexity of the target frame and the residual bit rate after reduction; adjust the first quantization parameter based on the range value to obtain the final quantization parameter.

[0480] Optionally, it also includes:

[0481] A third fluency acquisition module is used to obtain fluency parameters;

[0482] Optionally, it also includes:

[0483] The fourth control display module is used to display the fluency parameter input control;

[0484] The fourth fluency acquisition module is configured to acquire a fluency parameter in response to an input operation on the fluency parameter input control.

[0485] Optionally, the fluency parameter input control is a sliding control, and the input operation is a sliding operation; or the fluency parameter input control is an input box, and the input operation is inputting in the input box.

[0486] Optionally, the fourth frame detection module 642 is further configured to determine a target frame in the to-be-encoded video frames according to a preset smoothness parameter.

[0487] Optionally, the fourth frame detection module includes:

[0488] The fourth frame detection submodule is configured to determine a target frame in the to-be-encoded video frames according to a preset fluency parameter.

[0489] The fourth frame detection submodule is specifically used to: obtain the single frame complexity of the video frame to be encoded and the average complexity of the video frame to be encoded; and determine the target frame in the video frame to be encoded based on the single frame complexity, the average complexity and the smoothness parameter.

[0490] Optionally, the fourth bit rate reduction module 643 includes:

[0491] a fourth frame type determination submodule, configured to determine the frame type of the target frame; wherein different frame types correspond to different bit rate redistribution models;

[0492] The fourth rate reduction amount determination module is used to determine the rate reduction amount of the target frame according to the pre-rate reduction rate of the target frame, the single frame complexity, the average complexity and the rate redistribution model corresponding to the frame type.

[0493] Optionally, it also includes:

[0494] The fourth frame skipping judgment module is used to skip the target frame corresponding to the cut video frame when the remaining bit rate of the cut video frame is greater than the frame skipping threshold; and execute the first quantization parameter adjustment module when the remaining bit rate of the cut video frame is less than or equal to the frame skipping threshold.

[0495] It should be noted that the other functional modules of the embodiments of the present application can refer to the description of the aforementioned embodiments and will not be described in detail here.

[0496] In an embodiment of the present application, for a video frame to be encoded generated by a video transmitter in a video conference, it can first be determined whether it is a scene-switch type video frame. If the video frame to be encoded is a scene-switch type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded. The above process is then performed on the next video to be encoded. In this way, during the processing of the video frame, the bit rate of a video frame that is itself a scene-switch type, or a video frame that becomes a scene-switch type after being assigned a new bit rate, will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene-switch type video frame is reduced and the bit rate of the normal video frame is increased, which can minimize the overall bit rate change. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not significantly reduce the evaluation of video quality, and can also avoid the problem of high peak bandwidth demand caused by excessive frame rate of scene-switch type video frames. While ensuring that the video quality remains basically unchanged, it can reduce the peak bandwidth and reduce the frame rate, thereby providing a better user experience. In addition, if the quantization parameter of the target frame is adjusted based on the smoothness parameter and the remaining bit rate after reduction, when the target frame is encoded, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0497] Reference Figure 12 , which shows a structural block diagram of an embodiment of a device for processing live video data according to an embodiment of the present application, which is applied to a video transmitter. The device includes:

[0498] The fifth acquisition module 651 is used to obtain the video frame to be encoded in the live video data;

[0499] A fifth frame detection module 652 is configured to determine a target frame in the to-be-encoded video frames; the target frame includes a scene-change type video frame;

[0500] a fifth bit rate reduction module 653, configured to reduce the bit rate of the target frame;

[0501] The fifth bit rate reallocation module 654 is configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fifth frame detection module.

[0502] Optionally, it also includes:

[0503] a fifth encoding module, configured to encode the target frame based on the bit rate remaining after the reduction;

[0504] Optionally, it also includes:

[0505] A third fluency acquisition module is used to obtain fluency parameters;

[0506] The fifth encoding module is further configured to adjust a quantization parameter for the target frame according to the smoothness parameter and the reduced residual bit rate; and encode the reduced video frame according to the quantization parameter.

[0507] It should be noted that the other functional modules of the embodiments of the present application refer to the description of the aforementioned embodiments and will not be described in detail here.

[0508] In an embodiment of the present application, for a video frame to be encoded generated by a video transmitter in a video conference, it can first be determined whether it is a scene-switch type video frame. If the video frame to be encoded is a scene-switch type video frame, the bit rate of the video frame is reduced, and then the reduced bit rate is allocated to other videos to be encoded. The above process is then performed on the next video to be encoded. In this way, during the processing of the video frame, the bit rate of a video frame that is itself a scene-switch type, or a video frame that becomes a scene-switch type after being assigned a new bit rate, will be reduced and allocated to other frames, such as normal video frames in the video to be encoded. In this way, the bit rate of the scene-switch type video frame is reduced and the bit rate of the normal video frame is increased, which can minimize the overall bit rate change. Under the current evaluation mechanism of evaluating video quality by testing a frame sequence, this method of the embodiment of the present application will not significantly reduce the evaluation of video quality, and can also avoid the problem of high peak bandwidth demand caused by excessive frame rate of scene-switch type video frames. While ensuring that the video quality remains basically unchanged, it can reduce the peak bandwidth and reduce the frame rate, thereby providing a better user experience. In addition, if the quantization parameter of the target frame is adjusted based on the smoothness parameter and the remaining bit rate after reduction, when the target frame is encoded, the encoded video frame can be made as clear as possible while reducing the bit rate of the target frame, which can further achieve the effect of reducing the stuttering rate while ensuring clarity.

[0509] It should be noted that the execution principle of the device part of the embodiment of the present application is similar to the execution principle of the method embodiment. Its execution process refers to the description of the method embodiment and will not be described in detail here.

[0510] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0511] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In the present application, the electronic device includes various types of devices such as terminal devices and server (cluster).

[0512] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and server (clusters). Figure 11 An exemplary device 700 that can be used to implement various embodiments described in the embodiments of this application is schematically shown.

[0513] For one embodiment, Figure 11 An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0514] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 700 can serve as a terminal device, a server (cluster), or other device described in the embodiments of the present application.

[0515] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage 708) having instructions 714 and one or more processors 702 configured in conjunction with the one or more computer-readable media to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0516] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0517] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0518] The memory 706 can be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0519] For one embodiment, the control module 704 may include one or more input / output controllers to provide interfaces to the NVM / storage device 708 and the input / output device(s) 710 .

[0520] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0521] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessible over a network via input / output device(s) 710.

[0522] (One or more) input / output devices 710 may provide an interface for apparatus 700 to communicate with any other appropriate devices. Input / output devices 710 may include communication components, audio components, sensor components, etc. Network interface 712 may provide an interface for apparatus 700 to communicate via one or more networks. Apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0523] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0524] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0525] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in the memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0526] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0527] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0528] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0529] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0530] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0531] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0532] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0533] The above is a detailed introduction to the video frame encoding method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video frame encoding method, characterized in that: The method comprises: Get the video frame to be encoded; Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; reducing the bit rate of the target frame; Adjusting a quantization parameter for the target frame according to a preset fluency parameter and a residual bit rate after reduction; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; encoding the reduced video frame according to the quantization parameter; Allocating the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and performing the step of determining the target frame among the to-be-encoded video frames; The reducing the bit rate of the target frame includes: Obtaining a normal bit rate of the target frame; Using a pre-trained rate reallocation model, calculate the rate reduction amount for the target frame; different frame types of the target frame correspond to different rate reallocation models; The normal bit rate of the target frame is subtracted from the bit rate reduction amount of the target frame to obtain the residual bit rate after the reduction; wherein the normal bit rate of the target frame is calculated according to the size of the video frame to be encoded or predicted according to the bit rate allocation model in the encoder.

2. The method according to claim 1, characterized in that The determining of the target frame in the to-be-encoded video frames includes: According to the preset smoothness parameter, a target frame in the to-be-encoded video frames is determined.

3. The method according to claim 2, wherein the step of determining the target frame in the to-be-encoded video frames according to the preset smoothness parameter comprises: Obtaining a single frame complexity of the to-be-encoded video frame and an average complexity of the to-be-encoded video frame; A target frame in the to-be-encoded video frames is determined according to the single-frame complexity, the average complexity, and the smoothness parameter.

4. The method according to claim 3, wherein the other group is to determine the target frame in the to-be-encoded video frames according to the single-frame complexity, the average complexity, and the smoothness parameter, comprising: A target frame in the to-be-encoded video frames is determined according to a comparison result between a first product of the single-frame complexity and the smoothness parameter and a second product of the average complexity and a preset coefficient.

5. The method according to claim 4, wherein the determining the target frame among the to-be-encoded video frames based on a comparison result between a first product of the single-frame complexity and the smoothness parameter and a second product of the average complexity and a preset coefficient comprises: When the first product is greater than the second product, determining the to-be-encoded video frame as a target frame; When the first product is less than or equal to the second product, it is determined that the to-be-encoded video frame is a normal video frame.

6. The method according to claim 3, characterized in that The reducing the bit rate of the target frame includes: Determining a frame type of the target frame; wherein different frame types correspond to different bit rate redistribution models; The bit rate reduction amount of the target frame is determined according to the pre-reduction bit rate of the target frame, the single frame complexity, the average complexity, and a bit rate redistribution model corresponding to the frame type.

7. The method according to claim 1, characterized in that The adjusting the quantization parameter for the target frame according to the preset fluency parameter and the reduced residual bit rate includes: Determining a quantization offset parameter according to the fluency parameter; Adjusting the range of the quantization parameter according to the quantization offset parameter; Determining a first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate; The first quantization parameter is adjusted according to the range value to obtain a final quantization parameter.

8. The method according to claim 7, characterized in that The determining of the first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate includes: The single frame complexity and the residual bit rate are input into a preset code control model to determine the first quantization parameter.

9. The method according to claim 1, characterized in that Also includes: When the remaining bit rate of the reduced video frame is greater than a frame skipping threshold, skipping a target frame corresponding to the reduced video frame; When the remaining bit rate of the reduced video frame is less than or equal to the frame skipping threshold, a step of adjusting the quantization parameter for the target frame according to a preset smoothness parameter and the reduced remaining bit rate is performed.

10. The method according to claim 1, characterized in that Also includes: Display the input control of the fluency parameter; In response to an input operation on the fluency parameter input control, a fluency parameter is obtained.

11. The method according to claim 10, characterized in that The fluency parameter input control is a sliding control, and the input operation is a sliding operation; Alternatively, the fluency parameter input control is an input box, and the input operation is inputting in the input box.

12. The method according to claim 1, characterized in that Also includes: Obtain network environment information; The fluency parameter is determined according to the network environment information.

13. A rate control method, characterized in that: The method comprises: Get the video frame to be encoded; Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; reducing the bit rate of the target frame; Allocating the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and performing the step of determining the target frame among the to-be-encoded video frames; The reducing the bit rate of the target frame includes: Obtaining a normal bit rate of the target frame; Using a pre-trained rate reallocation model, calculate the rate reduction amount for the target frame; different frame types of the target frame correspond to different rate reallocation models; Subtracting the rate reduction amount of the target frame from the normal rate of the target frame to obtain a residual rate after the reduction; wherein the normal rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on a rate allocation model in the encoder; The method further comprises: Get the fluency parameters; encoding the target frame based on the remaining bit rate after the reduction; The encoding of the target frame based on the remaining bit rate after the reduction includes: Adjusting a quantization parameter for the target frame according to the fluency parameter and the reduced residual bit rate; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; The reduced video frame is encoded according to the quantization parameter.

14. A video frame encoding method, characterized in that: The method comprises: Obtaining a first video stream sent by a video transmitter and obtaining a fluency parameter of a video receiver; Decoding the first video stream to obtain a video frame to be encoded; Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; reducing the bit rate of the target frame; Adjusting a quantization parameter for the target frame according to a preset fluency parameter and a residual bit rate after reduction; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; encoding the reduced video frame according to the quantization parameter, and sending the encoded second video stream to a video receiving end; Allocating the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and performing the step of determining the target frame among the to-be-encoded video frames; The reducing the bit rate of the target frame includes: Obtaining a normal bit rate of the target frame; Using a pre-trained rate reallocation model, calculate the rate reduction amount for the target frame; different frame types of the target frame correspond to different rate reallocation models; The normal bit rate of the target frame is subtracted from the bit rate reduction amount of the target frame to obtain the residual bit rate after the reduction; wherein the normal bit rate of the target frame is calculated according to the size of the video frame to be encoded or predicted according to the bit rate allocation model in the encoder.

15. A method for processing video conference data, characterized in that: The method comprises: Obtaining the video frame to be encoded in the video conference data; Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; reducing the bit rate of the target frame; Allocating the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and performing the step of determining the target frame among the to-be-encoded video frames; The reducing the bit rate of the target frame includes: Obtaining a normal bit rate of the target frame; Calculating a rate reduction for the target frame using a pre-trained rate reallocation model; subtracting the rate reduction amount of the target frame from the normal rate of the target frame to obtain a residual rate after the reduction; wherein the normal rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on a rate allocation model in the encoder; different frame types of the target frame correspond to different rate redistribution models; The method further comprises: Adjusting the quantization parameter for the target frame according to a preset fluency parameter and the remaining bit rate after the reduction; The reduced video frame is encoded according to the quantization parameter; the smoothness parameter is input by performing an input operation on a smoothness parameter input control; the smoothness parameter input control is a sliding control, and the input operation is a sliding operation.

16. The method according to claim 15, characterized in that The adjusting the quantization parameter for the target frame according to the preset fluency parameter and the reduced residual bit rate includes: Determining a quantization offset parameter according to the fluency parameter; Adjusting the range of the quantization parameter according to the quantization offset parameter; Determining a first quantization parameter according to the single-frame complexity of the target frame and the reduced residual bit rate; The first quantization parameter is adjusted according to the range value to obtain a final quantization parameter.

17. The method according to claim 15, characterized in that The determining of the target frame in the to-be-encoded video frames includes: According to a preset fluency parameter, a target frame in the to-be-encoded video frames is determined.

18. The method according to any one of claims 15 to 17, characterized in that: Also includes: Display the input control of the fluency parameter; In response to an input operation on the fluency parameter input control, a fluency parameter is obtained.

19. A method for processing live video data, characterized in that: The method comprises: Obtain the video frame to be encoded in the live video data; Determining a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; reducing the bit rate of the target frame; Allocating the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and performing the step of determining the target frame among the to-be-encoded video frames; The reducing the bit rate of the target frame includes: Obtaining a normal bit rate of the target frame; Calculating a rate reduction for the target frame using a pre-trained rate reallocation model; subtracting the rate reduction amount of the target frame from the normal rate of the target frame to obtain a residual rate after the reduction; wherein the normal rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on a rate allocation model in the encoder; different frame types of the target frame correspond to different rate redistribution models; The method further comprises: Adjusting the quantization parameter for the target frame according to a preset fluency parameter and the remaining bit rate after the reduction; The reduced video frame is encoded according to the quantization parameter; the smoothness parameter is input by performing an input operation on a smoothness parameter input control; the smoothness parameter input control is a sliding control, and the input operation is a sliding operation.

20. A video frame encoding device, characterized in that: The device comprises: A first video frame acquisition module, used to acquire a video frame to be encoded; A first frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame; A first bit rate reduction module, configured to reduce the bit rate of the target frame; a first quantization parameter adjustment module, configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and a reduced residual bit rate; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; a first encoding module, configured to encode the reduced video frame according to the quantization parameter; A first bit rate redistribution module is configured to distribute the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and enter the first frame detection module; The first bit rate reduction module is specifically used to obtain the normal bit rate of the target frame; use a pre-trained bit rate redistribution model to calculate the bit rate reduction amount of the target frame; subtract the bit rate reduction amount of the target frame from the normal bit rate of the target frame to obtain the residual bit rate after the reduction; wherein, the normal bit rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on the bit rate allocation model in the encoder; different frame types of the target frame correspond to different bit rate redistribution models.

21. The device according to claim 20, characterized in that The first frame detection module includes: The frame detection submodule is used to determine the target frame in the video frames to be encoded according to the preset fluency parameter.

22. The apparatus according to claim 21, wherein the frame detection submodule comprises: A complexity acquisition submodule, configured to acquire the single frame complexity of the to-be-encoded video frame and the average complexity of the to-be-encoded video frame; The first frame detection submodule is configured to determine a target frame in the to-be-encoded video frames according to the single-frame complexity, the average complexity, and the smoothness parameter.

23. The device according to claim 22, characterized in that The first bit rate reduction module includes: a frame type determination submodule, configured to determine the frame type of the target frame; wherein different frame types correspond to different bit rate redistribution models; The bit rate reduction amount determination module is used to determine the bit rate reduction amount of the target frame according to the pre-reduction bit rate of the target frame, the single frame complexity, the average complexity and the bit rate redistribution model corresponding to the frame type.

24. The device according to claim 20, characterized in that Also includes: a frame skipping judgment module, configured to skip a target frame corresponding to the reduced video frame if the remaining bit rate of the reduced video frame is greater than a frame skipping threshold; In a case where the remaining bit rate of the reduced video frames is less than or equal to the frame skipping threshold, a first quantization parameter adjustment module is executed.

25. The device according to claim 20, characterized in that Also includes: Fluency control display module, used to display fluency parameter input controls; The fluency parameter acquisition module is configured to acquire a fluency parameter in response to an input operation on the fluency parameter input control.

26. The device according to claim 20, characterized in that The fluency parameter input control is a sliding control, and the input operation is a sliding operation; Alternatively, the fluency parameter input control is an input box, and the input operation is inputting in the input box.

27. A bit rate control device, characterized in that: The device comprises: A second video frame acquisition module, used to acquire the video frame to be encoded; A second frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame; A second bit rate reduction module, configured to reduce the bit rate of the target frame; a second bit rate redistribution module, configured to distribute the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute a second frame detection module; The second rate reduction module is specifically configured to obtain a normal bit rate of the target frame; calculate a rate reduction amount for the target frame using a pre-trained rate redistribution model; and subtract the rate reduction amount from the normal bit rate of the target frame to obtain a residual bit rate after the reduction. The normal bit rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on a rate allocation model in the encoder; and different frame types of the target frame correspond to different rate redistribution models. The device is further used to obtain a fluency parameter; encoding the target frame based on the remaining bit rate after the reduction; The encoding of the target frame based on the remaining bit rate after the reduction includes: Adjusting a quantization parameter for the target frame according to the fluency parameter and the reduced residual bit rate; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; The reduced video frame is encoded according to the quantization parameter.

28. A video frame encoding device, characterized in that: The device comprises: A third acquisition module is used to acquire the first video stream sent by the video sending end and acquire the fluency parameter of the video receiving end; A decoding module, configured to decode the first video stream to obtain a video frame to be encoded; A third frame detection module is used to determine a target frame in the video frame to be encoded; the target frame includes a scene switching type video frame; a third bit rate reduction module, configured to reduce the bit rate of the target frame; a third quantization parameter adjustment module, configured to adjust the quantization parameter for the target frame according to a preset fluency parameter and the reduced residual bit rate; the fluency parameter is input by performing an input operation on a fluency parameter input control; the fluency parameter input control is a sliding control, and the input operation is a sliding operation; a third encoding module, configured to encode the reduced video frame according to the quantization parameter, and send the encoded second video stream to a video receiving end; a third bit rate redistribution module, configured to distribute the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute a third frame detection module; The third rate reduction module is specifically used to obtain the normal rate of the target frame; use a pre-trained rate redistribution model to calculate the rate reduction amount of the target frame; subtract the rate reduction amount of the target frame from the normal rate of the target frame to obtain the residual rate after the reduction; wherein the normal rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on the rate allocation model in the encoder; different frame types of the target frame correspond to different rate redistribution models.

29. A device for processing video conference data, characterized in that: The device comprises: A fourth acquisition module is used to obtain the video frame to be encoded in the video conference data; a fourth frame detection module, configured to determine a target frame in the to-be-encoded video frame; the target frame comprises a scene-change type video frame; a fourth bit rate reduction module, configured to reduce the bit rate of the target frame; a fourth bit rate reallocation module, configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fourth frame detection module; The fourth rate reduction module is specifically configured to obtain a normal bit rate of the target frame; calculate a rate reduction amount for the target frame using a pre-trained rate redistribution model; and subtract the rate reduction amount from the normal bit rate of the target frame to obtain a residual bit rate after the reduction. The normal bit rate of the target frame is calculated based on the size of the video frame to be encoded or predicted based on a rate allocation model in the encoder; and different frame types of the target frame correspond to different rate redistribution models. The device is further configured to adjust a quantization parameter for the target frame according to a preset fluency parameter and a reduced residual bit rate; The reduced video frame is encoded according to the quantization parameter; the smoothness parameter is input by performing an input operation on a smoothness parameter input control; the smoothness parameter input control is a sliding control, and the input operation is a sliding operation.

30. A device for processing live video data, characterized in that: The device comprises: A fifth acquisition module is used to acquire the video frame to be encoded in the live video data; a fifth frame detection module, configured to determine a target frame in the to-be-encoded video frames; the target frame comprises a scene-change type video frame; a fifth bit rate reduction module, configured to reduce the bit rate of the target frame; a fifth bit rate reallocation module, configured to allocate the reduced bit rate to at least one to-be-encoded video frame other than the target frame, and execute the fifth frame detection module; The fifth rate reduction module is specifically configured to obtain a normal bit rate of the target frame; calculate a rate reduction amount for the target frame using a pre-trained rate redistribution model; and subtract the rate reduction amount from the normal bit rate of the target frame to obtain a residual bit rate after the reduction. The normal bit rate of the target frame is calculated based on the size of the to-be-encoded video frame or predicted based on a rate allocation model in an encoder; and different frame types of the target frame correspond to different rate redistribution models. The device is further configured to adjust a quantization parameter for the target frame according to a preset fluency parameter and a residual bit rate after reduction; The reduced video frame is encoded according to the quantization parameter; the smoothness parameter is input by performing an input operation on a smoothness parameter input control; the smoothness parameter input control is a sliding control, and the input operation is a sliding operation.

31. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to perform the method according to any one of claims 1 to 19.

32. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method according to any one of claims 1 to 19.

Images