Video encoding method and apparatus, and device and storage medium

Abstract

The embodiments of the present disclosure provide a video encoding method and apparatus, and a device and a storage medium. The video encoding method comprises: in response to current network latency being less than or equal to preset latency, on the basis of a current reference frame queue and frame reception acknowledgment information that is fed back by a receiving end, determining a received reference frame in the current reference frame queue; determining the current encoding complexity between a current video frame to be encoded and the received reference frame; on the basis of the current encoding complexity and a target complexity threshold, determining a target encoded frame type corresponding to the current video frame; and on the basis of the target encoded frame type, performing encoding processing on the current video frame, and on the basis of the current video frame, updating the current reference frame queue. The technical solution in the embodiments of the present disclosure enables dynamic determination of an encoded frame type, so as to realize dynamic video encoding, thereby reducing end-to-end latency and stuttering, and also improving the video clarity.

Description

Video encoding method, device, apparatus and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202410411587.9, filed on April 7, 2024, the disclosure of which is incorporated herein in its entirety as part of the present application. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to a video encoding method, device, apparatus and storage medium. BACKGROUND

[0003] With the rapid development of computer technology, it is often necessary to transmit video in real time to realize real-time communication (RTC) of video. At present, in the real-time communication scene, the collected frame sequence is usually encoded according to a fixed encoding structure (such as IPPPPP). However, this fixed encoding method often leads to large amount of encoded frame data and fixed blur, thereby increasing the end-to-end delay and stuttering, and also reducing the video clarity. SUMMARY

[0004] The present disclosure provides a video encoding method, device, apparatus and storage medium to dynamically determine the encoding frame type, thereby realizing dynamic video encoding, reducing the end-to-end delay and stuttering, and also improving the video clarity.

[0005] In a first aspect, the embodiments of the present disclosure provide a video encoding method, comprising:

[0006] In response to the current network delay being less than or equal to a preset delay, determining a received reference frame in a current reference frame queue based on the current reference frame queue and frame reception confirmation information fed back by a receiving end;

[0007] Determining a current encoding complexity between a current video frame to be encoded and the received reference frame;

[0008] Determining a target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold;

[0009] Encoding and processing the current video frame based on the target encoding frame type, and updating the current reference frame queue based on the current video frame.

[0010] In a second aspect, the embodiments of the present disclosure also provide a video encoding device, comprising:

[0011] a received reference frame determination module configured to determine a received reference frame in the current reference frame queue based on the current reference frame queue and frame reception acknowledgement information fed back by the receiving end, in response to the current network delay being less than or equal to the preset delay;

[0012] a coding complexity determination module configured to determine a current coding complexity between the current video frame to be coded and the received reference frame;

[0013] a coding frame type determination module configured to determine a target coding frame type corresponding to the current video frame based on the current coding complexity and a target complexity threshold;

[0014] a coding processing module configured to perform coding processing on the current video frame based on the target coding frame type, and update a current reference frame queue based on the current video frame.

[0015] In a third aspect, an electronic device is provided, and the electronic device includes:

[0016] one or more processors;

[0017] a storage device configured to store one or more programs,

[0018] when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the video coding method according to any of the embodiments of the present disclosure.

[0019] In a fourth aspect, a storage medium containing computer executable instructions is provided, and the computer executable instructions are used to perform the video coding method according to any of the embodiments of the present disclosure when executed by a computer processor. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent as various embodiments of the present disclosure are described in conjunction with the following detailed description, taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0021] FIG. 1 is a flow diagram of a video coding method according to an embodiment of the present disclosure;

[0022] FIG. 2 is an example diagram of a reference frame queue according to an embodiment of the present disclosure;

[0023] FIG. 3 is a flow diagram of another video coding method according to an embodiment of the present disclosure;

[0024] FIG. 4 is a structural diagram of a video coding apparatus according to an embodiment of the present disclosure; and

[0025] FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes, and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related definitions are given below in the description of the embodiments.

[0029] It should be noted that the concepts of "first", "second", and the like mentioned in the present disclosure are merely used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are merely for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0032] FIG. 1 is a flow schematic diagram of a video encoding method provided by an embodiment of the present disclosure, which is applicable to the case of encoding a video in a real-time communication scenario, which can include a live broadcast, a remote video conference, or a cloud game service scenario. The method can be executed by a video encoding device, which can be implemented in the form of software and / or hardware, and the device is integrated in a sending end. Optionally, the sending end can be implemented by an electronic device, which can be a mobile terminal, a PC end, or a server, etc.

[0033] As shown in FIG. 1, the video encoding method specifically comprises the following steps:

[0034] S110, in response to the current network delay being less than or equal to the preset delay, determining a received reference frame in the current reference frame queue based on the current reference frame queue and the frame receiving acknowledgement information fed back by the receiving end.

[0035] The sending end can be a terminal for collecting and generating real-time video and encoding and sending the real-time video. The receiving end can be a terminal corresponding to the sending end, for receiving, decoding and playing the real-time video. The number of receiving ends can be one or more. The current network delay can refer to the delay generated by the network transmission data between the sending end and the receiving end at the current time. For example, the current network delay can refer to the round-trip delay for transmitting data at the current time. The current network delay can dynamically change over time. The preset delay can be the maximum allowed delay of the receiving end feedback information received by the sending end, which is set in advance based on the business requirements. The frame receiving acknowledgement information fed back by the receiving end can refer to the frame ACK (Acknowledge character) acknowledgement message fed back by the receiving end when receiving a frame. The receiving end feeds back the information in frame granularity. The current reference frame queue can refer to the current encoded video frame queue referenced by the encoder in the sending end when encoding. The current reference frame queue is dynamically updated with the encoding of the video frame. The current reference frame queue can include at least one encoded video frame. The maximum length of the current reference frame queue can be set to 16 at most. The reference frames in the current reference frame queue can be divided into long-term reference frames and short-term reference frames. For example, FIG. 2 is an example diagram of a reference frame queue according to an embodiment of the present disclosure, as shown in FIG. 2, the maximum length of the current reference frame queue is 9, which contains 8 long-term reference frames and 1 short-term reference frame. The long-term reference frame can be dynamically updated based on the content of the video frame. The reference frame refers to the video frame encoded by the encoder, which is used as a reference for subsequent video frame encoding. The received reference frame refers to the reference frame that has been received by the receiving end. The number of received reference frames can be one or more.

[0036] Specifically, the sending end can detect whether the current network delay is less than or equal to the preset delay in real time or periodically, so as to determine whether the encoding frame type can be dynamically determined based on the receiving end feedback information, and then ensure the real-time communication effect. In the case of detecting that the current network delay is less than or equal to the preset delay, it indicates that the delay of the receiving end feedback information is within the allowable range, at this time, whether each current reference frame in the current reference frame queue exists frame receiving confirmation information of the receiving end feedback can be detected, and the received reference frame in the current reference frame queue is determined based on the detection result. For example, each reference frame in the current reference frame queue which exists frame receiving confirmation information is determined as the received reference frame in the current reference frame queue. The sending end can determine all the received reference frames in the current reference frame queue in real time based on the frame receiving confirmation information of the receiving end frame-by-frame feedback, so as to ensure the encoding effect.

[0037] It should be noted that due to the network delay between the sending end and the receiving end, after the sending end sends the encoded video frame, the receiving end often does not immediately receive the sent video frame. If there is a packet loss in the transmission process of the encoded video frame, the receiving end cannot receive the video frame, and thus cannot feed back the receiving confirmation information of the frame. At this time, the video frame can be retransmitted until the receiving confirmation information of the frame is fed back when the video frame is received. At this time, the network delay is large.

[0038] S120, determining the current encoding complexity between the current video frame to be encoded and the received reference frame.

[0039] The current video frame can refer to the current captured video frame to be encoded in the real-time communication scenario. The current video frame refers to the video frame currently generated in the real-time communication video. The current encoding complexity can be represented by the content similarity between the current video frame and the received reference frame. The higher the content similarity of the two frames, the lower the current encoding complexity. The current encoding complexity can also be used to reflect the amount of encoded frame data after the received reference frame is used as the reference frame of the current video frame for encoding. The lower the current encoding complexity, the smaller the amount of encoded frame data, and the smaller the encoding code rate.

[0040] Specifically, the current encoding complexity between the current video frame and each received reference frame can be determined by comparing the content difference between the current video frame and each received reference frame. For example, for each received reference frame, the pixel difference value corresponding to the same pixel in the current video frame and the received reference frame can be determined based on the pixel value in the current video frame and the pixel value in the received reference frame, and the current encoding complexity between the current video frame and the received reference frame can be determined based on the pixel difference value. For example, the pixel difference values corresponding to all pixels can be averaged to obtain the pixel difference average as the current encoding complexity between the current video frame and the received reference frame.

[0041] S130, determining a target coding frame type corresponding to the current video frame based on the current coding complexity and a target complexity threshold.

[0042] The target coding frame type refers to a coding frame type matched with the current video frame. In a real-time communication scenario, the target coding frame type can include an instant decoding refresh frame (i.e., an IDR frame) or a forward prediction coding frame (i.e., a P frame). The instant decoding refresh frame is encoded by referring to blocks within the current frame and does not need to refer to frames. The forward prediction coding frame is encoded by referring to previous frames. The target complexity threshold can refer to a maximum coding complexity allowed to be coded as a forward prediction coding frame. The target complexity threshold can be pre-set based on service requirements or dynamically determined based on a video source scene corresponding to the current video frame to improve the accuracy of coding type determination.

[0043] Specifically, the current coding complexity between the current video frame and each received reference frame is compared with the target complexity threshold, and the target coding frame type matched with the current video frame is determined based on the comparison result, so as to determine the coding frame type most suitable for the current video frame, and realize dynamic determination of the coding frame type without the need to code according to a fixed coding structure.

[0044] For example, step S130 can include: if there is no target received reference frame with a current coding complexity less than or equal to the target complexity threshold, determining that the target coding frame type corresponding to the current video frame is an instant decoding refresh frame; if there is at least one target received reference frame with a current coding complexity less than or equal to the target complexity threshold, determining that the target coding frame type corresponding to the current video frame is a forward prediction coding frame, and determining a target reference frame corresponding to the current video frame from the at least one target received reference frame.

[0045] Specifically, the current encoding complexity between the current video frame and each received reference frame is compared with the target complexity threshold, and it is determined whether there is a target received reference frame with the current encoding complexity less than or equal to the target complexity threshold. If there is no target received reference frame, it indicates that the amount of data of the current encoding as a P frame is large, and other frames also need to be relied on for decoding, at this time, it can be determined that the target encoding frame type corresponding to the current video frame is an IDR frame, so as to ensure that the current video frame can be quickly decoded. If there is at least one target received reference frame, it indicates that the amount of data of the current encoding as a P frame is small, within an acceptable range, at this time, it can be determined that the target encoding frame type corresponding to the current video frame is a P frame, so as to reduce the amount of encoding frame data, improve the data transmission efficiency, and determine the target reference frame corresponding to the current video frame from all the target received reference frames, so as to ensure that the current video frame can be normally decoded. For example, the last target received reference frame in the current reference frame queue can be determined as the target reference frame corresponding to the current video frame; or, the target reference frame corresponding to the current video frame can also be determined from the at least one target received reference frame based on the current encoding complexity corresponding to each target received reference frame.

[0046] It should be noted that when encoding according to a fixed encoding structure, an IDR frame type is inserted at a default insertion time, and the five video frames after the IDR frame are directly encoded as P frames according to the fixed encoding structure, such as IPPPPPP. This default insertion of the IDR frame type at the insertion time has a siphon effect, is prone to fixed blur, and also encodes as an IDR frame when the encoding complexity is very small, which further increases the video playing delay and stuttering. In the existing encoding mode, the P frame is always encoded by default by referring to the previous frame, and when the scene content changes greatly, the amount of data of the encoded P frame is particularly large, thereby increasing the end-to-end delay. Moreover, if the receiving end does not receive the video frame referred to by the P frame when decoding the P frame, the P frame cannot be decoded, and thus cannot be played, resulting in video stuttering and reducing the video viewing experience. The embodiment can ensure the fast decoding of the current video frame by taking one target received reference frame as the target reference frame corresponding to the current video frame, thereby reducing the end-to-end stuttering and improving the video viewing experience.

[0047] For example, determining the target reference frame corresponding to the current video frame from the at least one target received reference frame can include: if there are at least two target received reference frames, comparing the current encoding complexity corresponding to each target received reference frame, and determining the target received reference frame with the smallest current encoding complexity as the target reference frame corresponding to the current video frame.

[0048] Specifically, if there is only one target received reference frame, the target received reference frame is determined as the target reference frame corresponding to the current video frame. If there are at least two target received reference frames, the target received reference frame with the minimum current encoding complexity can be determined as the target reference frame corresponding to the current video frame, so that the data amount of the encoded current video frame is minimized, the data transmission efficiency is further improved, and the delay and stuttering are further reduced.

[0049] In S140, the current video frame is encoded based on the target encoding frame type, and the current reference frame queue is updated based on the current video frame.

[0050] Specifically, the sender encodes the current video frame into an encoded frame of the target encoding frame type through the encoder, and sends the encoded frame corresponding to the current video frame to the receiver. After receiving the encoded frame, the receiver feeds back the receiving confirmation information corresponding to the current video frame to the sender, to inform the sender that the current video frame has been received and can be normally decoded. After encoding the current video frame, the sender adds the current video frame to the current reference frame queue to dynamically update the current reference frame queue. For example, if the current length of the current reference frame queue is less than the preset length (i.e., the maximum allowed length), the current video frame can be directly added to the current reference frame queue. If the current length of the current reference frame queue is equal to the preset length (i.e., the maximum allowed length), one long-term reference frame or short-term reference frame in the current reference frame queue is deleted, and the current video frame is added to the queue after deletion as a new reference frame. For another example, it can be determined whether the current video frame belongs to a long-term reference frame or a short-term reference frame. If it belongs to a long-term reference frame, the current video frame can replace one long-term reference frame in the current reference frame queue for storage. If it belongs to a short-term reference frame, the current video frame can replace one short-term reference frame in the current reference frame queue for storage.

[0051] It should be noted that each video frame collected by the sender in real time is encoded according to the operations of the above steps S110-S140, so that the dynamic encoding of real-time video is realized, and the end-to-end video is clear, and the stuttering is in the optimal balance state.

[0052] For example, in S140, the current video frame is encoded based on the target encoding frame type, which can include: if the target encoding frame type is an instant decoding refresh frame, the current video frame is encoded into an instant decoding refresh frame, and the current reference frame queue is emptied; if the target encoding frame type is a forward prediction encoding frame, the current video frame is encoded into a forward prediction encoding frame based on the target reference frame.

[0053] Specifically, when the target coding frame type is an instant decoding refresh frame, the current video frame can be coded as an IDR frame, and all reference frames in the current reference frame queue are deleted, so that the current reference frame queue is an empty queue, in order to re-add reference frames. When the target coding frame type is a forward prediction coding frame, the current video frame is coded as a P frame based on the target reference frame received by the receiving end, so as to ensure the normal decoding and playing of the P frame.

[0054] The technical scheme of the embodiment of the present disclosure determines the received reference frame in the current reference frame queue based on the current reference frame queue and the frame receiving confirmation information fed back by the receiving end, and dynamically determines the target coding frame type matched with the current video frame based on the current coding complexity between the current video frame to be coded and the received reference frame and the target complexity threshold, and codes the current video frame based on the target coding frame type, so as to avoid the large amount of coding frame data and the fixed blur caused by coding the fixed type, thereby realizing the dynamic coding of the video, and further reducing the end-to-end delay and the stall, and improving the video definition.

[0055] FIG. 3 is a flowchart of another video coding method provided by the embodiment of the present disclosure, and the embodiment of the present disclosure optimizes the step of "determining the current coding complexity between the current video frame to be coded and the received reference frame" based on the above disclosed embodiments. The explanations of the same or corresponding terms in the above disclosed embodiments are not repeated here.

[0056] As shown in FIG. 3, the video coding method specifically includes the following steps:

[0057] S310, in response to the current network delay being less than or equal to the preset delay, determining the received reference frame in the current reference frame queue based on the current reference frame queue and the frame receiving confirmation information fed back by the receiving end.

[0058] S320, if the resolution of the current video frame to be coded is greater than or equal to the preset resolution, performing downsampling processing on the current video frame and the received reference frame based on the preset downsampling multiple.

[0059] The preset resolution can be a preset minimum resolution that needs to be downsampled. For example, the preset resolution is 1080P. The preset downsampling multiple can be a preset downsampling multiple, such as 2 times.

[0060] Specifically, whether to perform down-sampling on the current video frame is determined by detecting whether the resolution of the current video frame to be encoded is greater than or equal to a preset resolution. If the resolution of the current video frame is greater than or equal to the preset resolution, the current video frame and the received reference frame are subjected to down-sampling processing with the same multiple based on a preset down-sampling multiple. For example, if the preset down-sampling multiple is 2, the current video frame and the received reference frame with 1 / 2 original resolution are obtained after down-sampling processing. If the resolution of the current video frame is less than the preset resolution, the current encoding complexity between the current video frame and the received reference frame is directly determined based on the current video frame and the received reference frame.

[0061] In S330, the current encoding complexity between the current video frame and the received reference frame is determined based on the down-sampled current video frame and the received reference frame.

[0062] Specifically, by using the current video frame and the received reference frame with lower resolution, the current encoding complexity between the current video frame and the received reference frame can be determined more quickly, thereby reducing the calculation amount and improving the encoding complexity determination efficiency.

[0063] For example, S330 can include determining a pixel difference value corresponding to the same pixel based on a pixel value in the down-sampled current video frame and a pixel value in the down-sampled received reference frame, and determining the current encoding complexity between the current video frame and the received reference frame based on the pixel difference value.

[0064] Specifically, for each received reference frame, a pixel difference value corresponding to the same pixel can be determined based on a pixel value in the down-sampled current video frame and a pixel value in the down-sampled received reference frame, and the pixel difference values corresponding to all pixels can be averaged to obtain a pixel difference average value as the current encoding complexity between the current video frame and the received reference frame.

[0065] In S340, a target encoding frame type corresponding to the current video frame is determined based on the current encoding complexity and a target complexity threshold.

[0066] In S350, the current video frame is subjected to encoding processing based on the target encoding frame type, and the current reference frame queue is updated based on the current video frame.

[0067] The technical scheme of the embodiments of the present disclosure can reduce the calculation amount and improve the encoding efficiency by performing down-sampling processing on the current video frame and the received reference frame when the resolution of the current video frame to be encoded is greater than or equal to a preset resolution, and determining the current encoding complexity between the current video frame and the received reference frame based on the down-sampled current video frame and the received reference frame.

[0068] On the basis of each of the technical solutions above, before step S340, the method can further include: obtaining a target video source scene corresponding to the current video frame; and determining a target complexity threshold corresponding to the target video source scene, wherein the complexity threshold is positively correlated with a scene motion degree of the video source scene.

[0069] The target video source scene can refer to a service scene in which a video source generating a real-time video is located. The service scene can be divided based on a scene motion degree in the service scene. For example, the target video source scene can include a first video source scene or a second video source scene. The first video source scene can refer to a service scene in which a scene motion degree is less than or equal to a preset motion degree, such as a static scene or a small motion scene. The second video source scene can refer to a service scene in which a scene motion degree is greater than the preset motion degree, such as a medium motion scene or a high motion scene. Different video source scenes can correspond to different target complexity thresholds, and the greater the scene motion degree of the video source scene, the greater the target complexity threshold.

[0070] Specifically, before determining the target coding frame type corresponding to the current video frame, a target video source scene corresponding to the current video frame is obtained, such as inputting the current video frame and historical video frames in a recent period of time into a pre-trained scene classification network model for scene prediction, and based on the output of the scene classification network model, the target video source scene corresponding to the current video frame can be obtained. For example, the scene classification network model can be a binary classification model, that is, based on the input information, it is predicted whether the target video source scene corresponding to the current video frame is a first video source scene or a second video source scene. By using the scene classification network model, the target video source scene corresponding to the current video frame can be accurately determined. The target complexity threshold value matched with the target video source scene can be determined based on the scene motion degree of the target video source scene, wherein the greater the scene motion degree of the target video source scene, the greater the target complexity threshold value. Alternatively, the complexity threshold value corresponding to each video source scene can also be determined based on the scene motion degree of each video source scene, and in actual application, the target complexity threshold value corresponding to the target video source scene is determined based on the complexity threshold value corresponding to each video source scene and the target video source scene. For example, a first complexity threshold value corresponding to the first video source scene and a second complexity threshold value corresponding to the second video source scene are pre-set, and the first complexity threshold value is less than the second complexity threshold value. If the target video source scene is the first video source scene, the target complexity threshold value is determined as the first complexity threshold value, and if the target video source scene is the second video source scene, the target complexity threshold value is determined as the second complexity threshold value. By determining different target complexity threshold values based on different target video source scenes, the accuracy of the coding frame determination can be further ensured, and the situation that P frames are always coded and IDR frames are rarely coded due to the target complexity threshold value being set too large when the scene motion degree is small can be avoided. At the same time, the situation that IDR frames are always coded and P frames are rarely coded due to the target complexity threshold value being set too small when the scene motion degree is large can also be avoided.

[0071] On the basis of the above technical solutions, the method can further include: in response to the current network delay being greater than a preset delay, determining the target coding frame type corresponding to the current video frame to be coded based on the current network state.

[0072] The current network state can be used to represent whether there is a packet loss in the current network. For example, the current network state can include a normal state and a weak network state. The normal state means that there is no packet loss in the current network. The weak network state means that there is packet loss in the current network.

[0073] Specifically, in the case that the current network delay is greater than the preset delay, it indicates that the receiving end cannot feed back information to the sending end within the allowable delay range, so as to ensure the real-time communication effect, the encoding frame type cannot be determined based on the receiving end feedback information, at this time, the target encoding frame type matched with the current video frame can be determined based on the current network state. For example, if the current network state is a weak network state, it is determined that the target encoding frame type corresponding to the current video frame is an instant decoding refresh frame, so as to encode the current video frame as an IDR frame, so as to ensure the normal decoding of the current video frame as much as possible under the weak network state. If the current network state is a normal state, it indicates that there is no packet loss in the current network, only the delay is large, at this time, the target encoding frame type corresponding to the current video frame can be determined as a forward prediction encoding frame, and the previous video frame of the current video frame is taken as the target reference frame corresponding to the current video frame, so as to perform P frame encoding on the current video frame based on the previous video frame, so as to ensure the video clarity as much as possible, and achieve the optimal current frame encoding data amount and quality.

[0074] FIG. 4 is a structural schematic diagram of a video encoding device provided by an embodiment of the present disclosure, as shown in FIG. 4, the device specifically comprises: a received reference frame determination module 410, an encoding complexity determination module 420, an encoding frame type determination module 430, and an encoding processing module 440.

[0075] The received reference frame determination module 410 is configured to, in response to the current network delay being less than or equal to the preset delay, determine a received reference frame in the current reference frame queue based on the current reference frame queue and the frame receiving confirmation information fed back by the receiving end. The encoding complexity determination module 420 is configured to determine a current encoding complexity between a current video frame to be encoded and the received reference frame. The encoding frame type determination module 430 is configured to determine a target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold. The encoding processing module 440 is configured to perform encoding processing on the current video frame based on the target encoding frame type, and update the current reference frame queue based on the current video frame.

[0076] The technical scheme provided by the embodiment of the present disclosure, in response to the current network delay being less than or equal to the preset delay, determines a received reference frame in the current reference frame queue based on the current reference frame queue and the frame receiving confirmation information fed back by the receiving end, and dynamically determines a target encoding frame type matched with the current video frame based on the current encoding complexity between the current video frame to be encoded and the received reference frame and the target complexity threshold, and encodes the current video frame based on the target encoding frame type, so as to avoid the cases of large encoding frame data amount and fixed blur caused by encoding fixed type, thereby realizing video dynamic encoding, and further reducing the end-to-end delay and lag, and improving the video clarity.

[0077] On the basis of the technical solutions above, the target coding frame type comprises an instant decoding refresh frame or a forward prediction coding frame; and the coding frame type determination module 430 comprises:

[0078] The first determination unit is configured to determine that the target coding frame type corresponding to the current video frame is an instant decoding refresh frame if there is no target received reference frame with a current coding complexity less than or equal to a target complexity threshold.

[0079] The second determination unit is configured to determine that the target coding frame type corresponding to the current video frame is a forward prediction coding frame if there is at least one target received reference frame with a current coding complexity less than or equal to a target complexity threshold, and determine the target reference frame corresponding to the current video frame from the at least one target received reference frame.

[0080] On the basis of the technical solutions above, the second determination unit is specifically configured to compare the current coding complexity of each target received reference frame if there are at least two target received reference frames, and determine the target received reference frame with the minimum current coding complexity as the target reference frame corresponding to the current video frame.

[0081] On the basis of the technical solutions above, the coding processing module 440 is specifically configured to:

[0082] If the target coding frame type is an instant decoding refresh frame, the current video frame is coded as an instant decoding refresh frame, and the current reference frame queue is emptied.

[0083] If the target coding frame type is a forward prediction coding frame, the current video frame is coded as a forward prediction coding frame based on the target reference frame.

[0084] On the basis of the technical solutions above, the coding complexity determination module 420 comprises:

[0085] The down-sampling processing unit is configured to perform down-sampling processing on the current video frame and the received reference frame based on a preset down-sampling multiple if the resolution of the current video frame to be coded is greater than or equal to a preset resolution.

[0086] The coding complexity determination unit is configured to determine the current coding complexity between the current video frame and the received reference frame based on the down-sampled current video frame and received reference frame.

[0087] On the basis of the technical solutions above, the coding complexity determination unit is specifically configured to:

[0088] determine a pixel difference value corresponding to the same pixel based on the pixel value in the down-sampled current video frame and the pixel value in the down-sampled received reference frame; and determine a current encoding complexity between the current video frame and the received reference frame based on the pixel difference value.

[0089] On the basis of the above technical solutions, the device further comprises:

[0090] a target complexity threshold value determination module, configured to, before determining a target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold value, acquire a target video source scene corresponding to the current video frame; and determine a target complexity threshold value corresponding to the target video source scene, wherein the complexity threshold value is positively correlated with a scene motion degree of the video source scene.

[0091] On the basis of the above technical solutions, the encoding frame type determination module 430 is further configured to, in response to the current network delay being greater than a preset delay, determine a target encoding frame type corresponding to the current video frame to be encoded based on a current network state.

[0092] The video encoding device provided by the embodiments of the present disclosure can execute the video encoding method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of executing the method.

[0093] It should be noted that each unit and module included in the above device is only divided according to the function logic, and is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific name of each functional unit is only for convenient mutual distinction, and does not limit the protection scope of the embodiments of the present disclosure.

[0094] FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. Referring to FIG. 5, a structural schematic diagram of an electronic device (such as a terminal device or a server in FIG. 5) 500 suitable for implementing the embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 5 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0095] As shown in FIG. 5, the electronic device 500 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage device 508. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0096] Generally, the following devices can be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 can allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 5 shows the electronic device 500 with various devices, it should be understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0097] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 509, or installed from the storage devices 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0098] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0099] The electronic device provided by the embodiments of the present disclosure and the video encoding method provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above-mentioned embodiments, and the present embodiments have the same beneficial effects as the above-mentioned embodiments.

[0100] The embodiments of the present disclosure provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the video encoding method provided by the above-mentioned embodiments.

[0101] Note that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer readable program code is contained. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF, etc., or any suitable combination thereof.

[0102] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0103] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and can be accessed via the electronic device.

[0104] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: in response to the current network delay being less than or equal to a preset delay, determine a received reference frame in the current reference frame queue based on a current reference frame queue and frame receiving acknowledgement information fed back by a receiving end; determine a current encoding complexity between a current video frame to be encoded and the received reference frame; determine a target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold; and encode process the current video frame based on the target encoding frame type, and update the current reference frame queue based on the current video frame.

[0105] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0106] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specific logical functions specified for the block. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block of the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or by combinations of dedicated hardware and computer instructions.

[0107] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the names of the units do not constitute a limitation on the units themselves. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0108] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0109] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0110] According to one or more embodiments of the present disclosure, example one provides a video encoding method, comprising:

[0111] In response to the current network delay being less than or equal to a preset delay, determining a received reference frame in the current reference frame queue based on the current reference frame queue and frame receiving acknowledgement information fed back by the receiving end;

[0112] Determining a current encoding complexity between a current video frame to be encoded and the received reference frame;

[0113] Determining a target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold;

[0114] Encoding and processing the current video frame based on the target encoding frame type, and updating the current reference frame queue based on the current video frame.

[0115] According to one or more embodiments of the present disclosure, example two provides a video encoding method, further comprising:

[0116] Optionally, the target encoding frame type includes an instant decoding refresh frame or a forward prediction encoding frame.

[0117] The determining of the target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold includes:

[0118] If there is no target received reference frame with the current encoding complexity less than or equal to the target complexity threshold, the target encoding frame type corresponding to the current video frame is determined as the instant decoding refresh frame.

[0119] If there is at least one target received reference frame with the current encoding complexity less than or equal to the target complexity threshold, the target encoding frame type corresponding to the current video frame is determined as the forward prediction encoding frame, and a target reference frame corresponding to the current video frame is determined from the at least one target received reference frame.

[0120] According to one or more embodiments of the present disclosure, Example Three provides a video encoding method, further comprising:

[0121] Optionally, the determining of the target reference frame corresponding to the current video frame from the at least one target received reference frame includes:

[0122] If there are at least two target received reference frames, the current encoding complexity of each target received reference frame is compared, and the target received reference frame with the minimum current encoding complexity is determined as the target reference frame corresponding to the current video frame.

[0123] According to one or more embodiments of the present disclosure, Example Four provides a video encoding method, further comprising:

[0124] Optionally, the encoding processing of the current video frame based on the target encoding frame type includes:

[0125] If the target encoding frame type is the instant decoding refresh frame, the current video frame is encoded as the instant decoding refresh frame, and a current reference frame queue is emptied.

[0126] If the target encoding frame type is the forward prediction encoding frame, the current video frame is encoded as the forward prediction encoding frame based on the target reference frame.

[0127] According to one or more embodiments of the present disclosure, Example Five provides a video encoding method, further comprising:

[0128] Optionally, the determining of the current encoding complexity between the current video frame to be encoded and the received reference frame includes:

[0129] if a resolution of a current video frame to be encoded is greater than or equal to a preset resolution, performing downsampling processing on the current video frame and the received reference frame based on a preset downsampling multiple;

[0130] based on the downsampled current video frame and the received reference frame, determining a current encoding complexity between the current video frame and the received reference frame.

[0131] According to one or more embodiments of the present disclosure, example six provides a video encoding method, further comprising:

[0132] Optionally, the determining the current encoding complexity between the current video frame and the received reference frame based on the downsampled current video frame and the received reference frame comprises:

[0133] based on pixel values in the downsampled current video frame and pixel values in the downsampled received reference frame, determining pixel difference values corresponding to the same pixels;

[0134] based on the pixel difference values, determining the current encoding complexity between the current video frame and the received reference frame.

[0135] According to one or more embodiments of the present disclosure, example seven provides a video encoding method, further comprising:

[0136] Optionally, before determining the target encoding frame type corresponding to the current video frame based on the current encoding complexity and a target complexity threshold, the method further comprises:

[0137] obtaining a target video source scene corresponding to the current video frame;

[0138] determining a target complexity threshold corresponding to the target video source scene, wherein the complexity threshold is positively correlated with a scene motion degree of the video source scene.

[0139] According to one or more embodiments of the present disclosure, example eight provides a video encoding method, further comprising:

[0140] Optionally, the method further comprises:

[0141] in response to a current network delay being greater than a preset delay, determining a target encoding frame type corresponding to a current video frame to be encoded based on a current network state.

[0142] According to one or more embodiments of the present disclosure, example nine provides a video encoding apparatus, comprising:

[0143] a received reference frame determination module configured to determine a received reference frame in the current reference frame queue based on the current reference frame queue and frame receiving acknowledgement information fed back by the receiving end, in response to the current network delay being less than or equal to the preset delay;

[0144] a coding complexity determination module configured to determine a current coding complexity between the current video frame to be coded and the received reference frame;

[0145] a coding frame type determination module configured to determine a target coding frame type corresponding to the current video frame based on the current coding complexity and a target complexity threshold;

[0146] a coding processing module configured to perform coding processing on the current video frame based on the target coding frame type, and update a current reference frame queue based on the current video frame.

[0147] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology employed. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0148] In addition, although each operation is depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments. The scope of the present disclosure is defined by the appended claims.

[0149] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A video encoding method, comprising: In response to a current network delay being less than or equal to a preset delay, determining a received reference frame in the current reference frame queue based on the current reference frame queue and frame reception confirmation information fed back by the receiving end; determining a current coding complexity between a current video frame to be encoded and the received reference frame; Determining a target coding frame type corresponding to the current video frame based on the current coding complexity and a target complexity threshold; The current video frame is encoded based on the target encoding frame type, and the current reference frame queue is updated based on the current video frame.

2. The video encoding method according to claim 1, wherein: The target coding frame type includes an instant decoding refresh frame or a forward prediction coding frame; The determining, based on the current coding complexity and the target complexity threshold, a target coding frame type corresponding to the current video frame includes: If there is no target received reference frame whose current coding complexity is less than or equal to the target complexity threshold, determining that the target coding frame type corresponding to the current video frame is an instant decoding refresh frame; If there is at least one target received reference frame whose current coding complexity is less than or equal to the target complexity threshold, the target coding frame type corresponding to the current video frame is determined to be a forward prediction coding frame, and the target reference frame corresponding to the current video frame is determined from at least one target received reference frame.

3. The video encoding method according to claim 2, wherein: The determining a target reference frame corresponding to the current video frame from at least one target received reference frame includes: If there are at least two target received reference frames, the current encoding complexities corresponding to each target received reference frame are compared, and the target received reference frame with the smallest current encoding complexity is determined as the target reference frame corresponding to the current video frame.

4. The video encoding method according to claim 2 or 3, wherein: The encoding process of the current video frame based on the target encoding frame type includes: If the target coding frame type is an instant decoding refresh frame, encoding the current video frame as an instant decoding refresh frame and clearing the current reference frame queue; If the target coding frame type is a forward prediction coding frame, the current video frame is encoded as a forward prediction coding frame based on the target reference frame.

5. The video encoding method according to any one of claims 1 to 4, wherein: The determining of a current coding complexity between the current video frame to be encoded and the received reference frame comprises: If the resolution of the current video frame to be encoded is greater than or equal to the preset resolution, downsampling the current video frame and the received reference frame based on a preset downsampling multiple; Based on the downsampled current video frame and the received reference frame, a current coding complexity between the current video frame and the received reference frame is determined. The video encoding method according to claim 5 , wherein: The determining, based on the downsampled current video frame and the received reference frame, a current coding complexity between the current video frame and the received reference frame, includes: Determining a pixel difference value corresponding to the same pixel based on a pixel value in the downsampled current video frame and a pixel value in the downsampled received reference frame; Based on the pixel difference values, a current coding complexity between the current video frame and the received reference frame is determined.

7. The video encoding method according to any one of claims 1 to 6, wherein: Before determining the target coding frame type corresponding to the current video frame based on the current coding complexity and the target complexity threshold, the video coding method further includes: Obtaining a target video source scene corresponding to the current video frame; A target complexity threshold corresponding to the target video source scene is determined, wherein the complexity threshold is positively correlated with the scene motion degree of the video source scene.

8. The video encoding method according to any one of claims 1 to 7, further comprising: In response to the current network delay being greater than the preset delay, a target coding frame type corresponding to the current video frame to be encoded is determined based on the current network state.

9. A video encoding apparatus, comprising: a received reference frame determining module, configured to, in response to a current network delay being less than or equal to a preset delay, determine a received reference frame in the current reference frame queue based on the current reference frame queue and frame reception confirmation information fed back by the receiving end; a coding complexity determination module configured to determine a current coding complexity between a current video frame to be encoded and the received reference frame; a coding frame type determination module, configured to determine a target coding frame type corresponding to the current video frame based on the current coding complexity and a target complexity threshold; The encoding processing module is configured to perform encoding processing on the current video frame based on the target encoding frame type, and update the current reference frame queue based on the current video frame.

10. An electronic device comprising: one or more processors; a storage device configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the video encoding method according to any one of claims 1 to 8.

11. A storage medium containing computer-executable instructions, wherein: When the computer executable instructions are executed by a computer processor, the computer executable instructions are used to perform the video encoding method according to any one of claims 1 to 8.

Images