A video data processing method and device, computer equipment and a storage medium

Abstract

Embodiments of the present application provide a video data processing method and device, computer equipment and a storage medium, which are applied to cloud technology, vehicle-mounted scene, audio and video coding and decoding scene, and the method comprises the following steps: obtaining a target video frame from video data, and obtaining an associated coding unit associated with a to-be-coded unit in the target video frame; the coding order of the associated coding unit is earlier than that of the to-be-coded unit, and the associated coding unit is adjacent to the to-be-coded unit; determining a set of available reference frames corresponding to the to-be-coded unit according to the associated coding unit in the video data; if there is a parent coding unit to which the to-be-coded unit belongs in the target video frame, determining a set of candidate reference frames corresponding to the to-be-coded unit according to the parent coding unit and the set of available reference frames; the set of candidate reference frames is used to traverse target reference frames for the to-be-coded unit. By using the present application, the coding effect and coding efficiency of the target video frame can be considered simultaneously.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a video data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] In a data transmission scenario (for example, a live streaming scenario), video data to be transmitted needs to be encoded to obtain a video bitstream corresponding to the video data, so as to improve transmission efficiency. It can be understood that, in the process of encoding the video data, a to-be-encoded unit of a target video frame to be encoded needs to be obtained from the video data, and then the to-be-encoded unit is subjected to inter-frame prediction or intra-frame prediction. In the inter-frame prediction process, an inter-frame prediction mode needs to use a reference frame selection algorithm to determine a reference frame used for encoding the to-be-encoded unit in the video data.

[0003] A current reference frame selection algorithm can obtain video frames that are in front of the target video frame in the encoding order. One reference frame selection algorithm can determine distances between the video frames and the target video frame, and encoding qualities of the video frames themselves, and then superimposes the distances and the encoding qualities, sorts the superimposed result from large to small, and selects a video frame corresponding to a maximum value in the sorted result as a target reference frame corresponding to the to-be-encoded unit in the target video frame. However, the reference frame selection algorithm only considers the distance between the target reference frame and the target video frame, and the encoding quality of the target reference frame, and does not consider the content similarity between the target reference frame and the target video frame. When the image content changes dramatically from the target reference frame to the target video frame, the content in the target reference frame and the content in the target video frame are quite different. Based on the target reference frame with the large content difference, the encoding of the target video frame is obviously reduced. Another reference frame selection algorithm can traverse the video frames, so as to implement encoding of each possible reference frame combination to find the best reference frame. However, if the number of video frames in front of the target video frame in the encoding order is relatively large, the reference frame selection algorithm needs to consume a large amount of time in the process of traversing the video frames, thereby reducing the encoding efficiency of the target video frame. Therefore, the current reference frame selection algorithm cannot balance the encoding effect and the encoding efficiency. SUMMARY

[0004] The embodiments of the present application provide a video data processing method, device, computer equipment and storage medium, which can balance the encoding effect and the encoding efficiency of a target video frame.

[0005] The embodiments of the present application provide a video data processing method, device, computer equipment and storage medium, which can balance the encoding effect and the encoding efficiency of a target video frame.

[0006] The target video frame is acquired from video data, and an associated coding unit associated with a to-be-coded unit in the target video frame is acquired; the coding order of the associated coding unit is earlier than the coding order of the to-be-coded unit, and the associated coding unit is adjacent to the to-be-coded unit.

[0007] A set of available reference frames corresponding to the to-be-coded unit is determined according to the associated coding unit in the video data.

[0008] If a parent coding unit to which the to-be-coded unit belongs exists in the target video frame, a set of candidate reference frames corresponding to the to-be-coded unit is determined according to the parent coding unit and the set of available reference frames; the set of candidate reference frames is used to traverse a target reference frame for the to-be-coded unit; and the target reference frame is used for encoding processing of the to-be-coded unit.

[0009] Embodiments of the present application provide a video data processing device, comprising:

[0010] The coding unit acquisition module is configured to acquire a target video frame from video data, and acquire an associated coding unit associated with a to-be-coded unit in the target video frame; the coding order of the associated coding unit is earlier than the coding order of the to-be-coded unit, and the associated coding unit is adjacent to the to-be-coded unit.

[0011] The available set determination module is configured to determine a set of available reference frames corresponding to the to-be-coded unit according to the associated coding unit in the video data.

[0012] The candidate set determination module is configured to, if a parent coding unit to which the to-be-coded unit belongs exists in the target video frame, determine a set of candidate reference frames corresponding to the to-be-coded unit according to the parent coding unit and the set of available reference frames; the set of candidate reference frames is used to traverse a target reference frame for the to-be-coded unit; and the target reference frame is used for encoding processing of the to-be-coded unit.

[0013] The number of the associated coding units is S, and S is a positive integer.

[0014] The available set determination module comprises:

[0015] The first type acquisition unit is configured to acquire a prediction coding type of the S associated coding units.

[0016] The first determination unit is configured to, if there is an associated coding unit with an intra prediction coding type in the S associated coding units, acquire a full reference frame set constructed for the to-be-coded unit in the video data, and determine the full reference frame set as the set of available reference frames corresponding to the to-be-coded unit.

[0017] The second determination unit is configured to, if the prediction coding types of the S associated coding units are all inter prediction, determine the set of available reference frames corresponding to the to-be-coded unit according to the S associated coding units in the video data.

[0018] wherein the full reference frame set comprises a forward full reference frame set and a backward full reference frame set;

[0019] The first determining unit is specifically configured to acquire, in the video data, a coded video frame earlier than the target video frame in coding order.

[0020] The first determining unit is specifically configured to add the coded video frame to the forward full reference frame set if the coded video frame is earlier than the target video frame in playback order.

[0021] The first determining unit is specifically configured to add the coded video frame to the backward full reference frame set if the coded video frame is later than the target video frame in playback order.

[0022] The second determining unit is specifically configured to acquire a coding unit quantity threshold value associated with the video encoder; the coding unit quantity threshold value is greater than or equal to S.

[0023] The second determining unit is specifically configured to acquire, in the video data, a full reference frame set constructed for the to-be-coded unit if S is less than the coding unit quantity threshold value, and determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit.

[0024] The second determining unit is specifically configured to determine, in the video data, the available reference frame set corresponding to the to-be-coded unit according to the reference frames used by the S associated coding units if S is equal to the coding unit quantity threshold value.

[0025] The number of the associated coding units is S, and S is a positive integer.

[0026] The available set determining module comprises:

[0027] The full set determining unit is configured to acquire, in the video data, a full reference frame set constructed for the to-be-coded unit if the prediction coding types of the S associated coding units are all inter prediction and S is equal to a coding unit quantity threshold value associated with the video encoder; the full reference frame set comprises a forward full reference frame set and a backward full reference frame set.

[0028] The reference frame determining unit is configured to acquire, in the forward full reference frame set, a forward reference frame closest to the position of the target video frame, and acquire, in the backward full reference frame set, a backward reference frame closest to the position of the target video frame.

[0029] The reference frame merging unit is configured to merge the reference frames used by the S associated coding units, the forward reference frame and the backward reference frame to obtain the available reference frame set corresponding to the to-be-coded unit.

[0030] The available reference frame set includes a forward available reference frame set and a backward available reference frame set.

[0031] The reference frame merging unit is specifically configured to determine a union of reference frames used by the S associated coding units as the associated reference frame set.

[0032] The reference frame merging unit is specifically configured to, if the associated reference frame set does not include a forward reference frame and a backward reference frame, determine a reference frame earlier in the playing order than the target video frame and the forward reference frame in the associated reference frame set as the forward available reference frame set corresponding to the to-be-encoded unit.

[0033] The reference frame merging unit is specifically configured to determine a reference frame later in the playing order than the target video frame and the backward reference frame in the associated reference frame set as the backward available reference frame set corresponding to the to-be-encoded unit.

[0034] The candidate set determination module includes:

[0035] The second type acquisition unit is configured to acquire a prediction coding type of the parent coding unit.

[0036] The third determination unit is configured to, if the prediction coding type of the parent coding unit is intra prediction, determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit.

[0037] The fourth determination unit is configured to, if the prediction coding type of the parent coding unit is inter prediction, determine the candidate reference frame set corresponding to the to-be-encoded unit according to the reference frame used by the parent coding unit and the available reference frame set.

[0038] The fourth determination unit is specifically configured to acquire an inter coding mode of the parent coding unit.

[0039] The fourth determination unit is specifically configured to, if the inter coding mode is not inter skip mode, determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit.

[0040] The fourth determination unit is specifically configured to, if the inter coding mode is inter skip mode, match the reference frame used by the parent coding unit and the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-encoded unit.

[0041] The candidate set determination module includes:

[0042] The reference frame matching unit is configured to, if the prediction coding type of the parent coding unit is inter prediction and the inter coding mode of the parent coding unit is inter skip mode, match the reference frame used by the parent coding unit and the available reference frame set.

[0043] The fifth determining unit is configured to, if there is an intersection between the reference frames used by the parent coding unit and the reference frames in the set of available reference frames, determine the intersection between the reference frames used by the parent coding unit and the reference frames in the set of available reference frames as the set of candidate reference frames corresponding to the to-be-coded unit.

[0044] The sixth determining unit is configured to, if there is no intersection between the reference frames used by the parent coding unit and the reference frames in the set of available reference frames, determine the set of available reference frames as the set of candidate reference frames corresponding to the to-be-coded unit.

[0045] The set of available reference frames includes a set of forward available reference frames and a set of backward available reference frames; and the set of candidate reference frames includes a set of forward candidate reference frames and a set of backward candidate reference frames.

[0046] The fifth determining unit is specifically configured to determine, as the set of forward candidate reference frames corresponding to the to-be-coded unit, the reference frames in the set of forward available reference frames and the reference frames used by the parent coding unit that are earlier than the target video frame in the playing order.

[0047] The fifth determining unit is specifically configured to determine, as the set of backward candidate reference frames corresponding to the to-be-coded unit, the reference frames in the set of backward available reference frames and the reference frames used by the parent coding unit that are later than the target video frame in the playing order.

[0048] The apparatus further includes:

[0049] The set delivering module is configured to, if the parent coding unit to which the to-be-coded unit belongs does not exist in the target video frame, determine the set of available reference frames as the set of candidate reference frames corresponding to the to-be-coded unit.

[0050] An embodiment of the present application provides a computer device, including a processor and a memory.

[0051] The processor is connected with the memory, where the memory is configured to store a computer program, and the computer program is executed by the processor to enable the computer device to perform the method provided in the embodiments of the present application.

[0052] An embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is adapted to be loaded and executed by a processor, so as to enable a computer device having the processor to perform the method provided in the embodiments of the present application.

[0053] An embodiment of the present application provides a computer program product, which includes a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to enable the computer device to perform the method provided in the embodiments of the present application.

[0054] It can be seen that the fast reference frame selection algorithm provided in the embodiments of the present application can fully consider the prediction correlation of the video content in the target video frame (i.e., the reference frames of the coding units in the same frame are probably the same), on the one hand, the associated coding units adjacent to the to-be-encoded unit are obtained, the reference frames of the to-be-encoded unit are limited by using the associated coding units, and the available reference frame set is obtained, on the other hand, the parent coding unit to which the to-be-encoded unit belongs is obtained, and the child coding unit is limited by using the parent coding unit, and the candidate reference frame set is obtained. It can be understood that by using the fast reference frame selection algorithm provided in the embodiments of the present application, the candidate reference frame set that fuses the reference frames used by the associated coding units and the reference frames used by the parent coding unit can be selected from all video frames. Because the reference frames in the candidate reference frame set are determined by the associated coding units and the parent coding unit, the reference frames in the candidate reference frame set have a high content similarity with the target video frame. In this way, the embodiments of the present application can not need to traverse all the encoded video frames, but traverse the video frames in the candidate reference frame set with fewer frame numbers, which not only reduces the traversal time consumption, but also can obtain the target reference frame with the best encoding effect in the traversal result when traversing the candidate reference frame set to which the reference frame with high content similarity belongs, so that the encoding effect and the encoding efficiency of the target video frame can be considered at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0056] Figure 1 is a structural schematic diagram of a network architecture provided by the embodiments of the present application;

[0057] Figure 2 is a scene schematic diagram of data interaction provided by the embodiments of the present application;

[0058] Figure 3 is a flow schematic diagram of a video data processing method provided by the embodiments of the present application;

[0059] Figure 4 is a scene schematic diagram of obtaining associated coding units provided by the embodiments of the present application;

[0060] Figure 5 is a scene schematic diagram of obtaining forward reference frames and backward reference frames provided by the embodiments of the present application;

[0061] Figure 6 is a flowchart of a fast reference frame selection provided by an embodiment of the present application;

[0062] Figure 7 is a flowchart of a video data processing method provided by an embodiment of the present application;

[0063] Figure 8 is a flowchart of a video data processing method provided by an embodiment of the present application;

[0064] Figure 9 is a structural diagram of a video data processing apparatus provided by an embodiment of the present application;

[0065] Figure 10 is a structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0067] Specifically, please refer to Figure 1 , Figure 1 is a structural diagram of a network architecture provided by an embodiment of the present application. As shown in Figure 1 , the network architecture can include a server 2000 and a terminal device cluster. The terminal device cluster can specifically include one or more terminal devices, and the number of terminal devices in the terminal device cluster will not be limited here. As shown in Figure 1 , the plurality of terminal devices can specifically include a terminal device 3000a, a terminal device 3000b, a terminal device 3000c, …, and a terminal device 3000n. The terminal device 3000a, the terminal device 3000b, the terminal device 3000c, …, and the terminal device 3000n can be directly or indirectly network-connected with the server 2000 through wired or wireless communication, so that each terminal device can perform data interaction with the server 2000 through the network connection.

[0068] Each terminal device in the terminal device cluster can include a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device, a smart home appliance (for example, a smart television), a wearable device, a vehicle-mounted terminal, an aircraft, and other smart terminals with data processing functions. It should be understood that Figure 1Each terminal device in the terminal device cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the server 2000. This application client can include social clients, multimedia clients (e.g., video clients), entertainment clients (e.g., game clients), educational clients, live streaming clients, and other application clients with video encoding capabilities. This application client can be a standalone client or an embedded sub-client integrated into another client; no limitation is made here.

[0069] Among them, server 2000 can be the server corresponding to the application client. Server 2000 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0070] For ease of understanding, the embodiments of this application may be described in detail below. Figure 1 From the multiple terminal devices shown, one terminal device is selected as the target terminal device. For example, in the embodiments of this application, a terminal device can be selected as the target terminal device. Figure 1 The terminal device 3000c shown serves as the target terminal device, which can integrate an application client with video encoding capabilities. In this case, the target terminal device can interact with the server 2000 through the application client.

[0071] It should be understood that the video data processing method provided in this application embodiment can be executed by a computer device with video encoding capabilities. This computer device can utilize cloud technology to encode and transmit multimedia data (e.g., video data). Specifically, the video data processing method provided in this application embodiment can be executed by a server 2000 (i.e., the aforementioned computer device can be a server 2000), by a target terminal device (i.e., the aforementioned computer device can be a target terminal device), or by both the server 2000 and the target terminal device. In other words, the server 2000 can encode the video data using the video data processing method provided in this application embodiment, and then send the encoded video stream to the target terminal device, which can then decode and play the video stream. Alternatively, the target terminal device can also encode the video data using the video data processing method provided in this application embodiment, and then send the encoded video stream to the server 2000. Optionally, the target terminal device can also send the encoded video stream to other terminal devices in the terminal device cluster.

[0072] Among them, the cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or local area network to realize data calculation, storage, processing and sharing. The cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portals. With the high development and application of the Internet industry, every item may have its own identification mark in the future, which needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system support to be realized through cloud computing.

[0073] It can be understood that the above network framework can be applied to video call scenarios, video transmission scenarios, cloud conference scenarios, live broadcast scenarios, cloud game scenarios, etc. Here, specific business scenarios will not be enumerated one by one. Among them, cloud gaming (Cloud gaming) can also be called gaming on demand, which is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In the cloud gaming scenario, the game does not run on the player's game terminal, but on the cloud server, and the cloud server renders the game scene into a video and audio stream and transmits it to the player's game terminal. The player's game terminal does not need to have strong graphics processing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input instructions and send them to the cloud server.

[0074] Among them, cloud conference is a kind of efficient, convenient and low-cost conference form based on cloud computing technology. Users only need to use the Internet interface and simple operation to quickly and efficiently share voice, data files and video with teams and customers around the world. The complex technology of data transmission and processing in the conference is operated by cloud conference service providers. At present, domestic cloud conference mainly focuses on SaaS (Software as a Service) mode as the main service content, including telephone, network, video and other service forms. Video conference based on cloud computing is called cloud conference. In the era of cloud conference, data transmission, processing and storage are all handled by computer resources of video conference manufacturers. Users no longer need to purchase expensive hardware and install cumbersome software. They only need to open a browser and log in to the corresponding interface to conduct efficient remote conferences. Cloud conference system supports multi-server dynamic cluster deployment and provides multiple high-performance servers, greatly improving the stability, security and availability of the conference. In recent years, video conference has been widely used in transportation, transportation, finance, operators, education, enterprises, Internet of Vehicles and other fields because it can greatly improve communication efficiency, continuously reduce communication costs and upgrade internal management. There is no doubt that video conference using cloud computing has stronger attraction in convenience, speed and ease of use, and will certainly stimulate a new high tide of video conference application.

[0075] It should be understood that a computer device with video coding function (for example, a target terminal device) can code process video data through a video encoder to obtain a video code stream corresponding to the video data, thereby improving the transmission efficiency of the video data. For example, the video encoder can be a HEVC (High Efficiency Video Coding) video encoder, a VVC (Versatile Video Coding) video encoder, etc. The VCC video encoder is also called H.266 video encoder, and the Versatile Video Coding standard specifies the decoding process and syntax of the H.266 video encoder decoding and the encoding process and syntax of the encoding.

[0076] Among them, the H.266 video encoder is an encoding standard, and its code rate is about 50% of the previous generation standard HEVC under the same subjective quality, which is of great help to the current massive video service data, because the same quality of video stream needs less storage space and less bandwidth. However, the encoding complexity of the H.266 video encoder is also increased by several times, because the new standard introduces more complex encoding tools to obtain higher video compression ratio. High encoding complexity means that encoding requires more computing resources and longer time, and high encoding complexity will directly reduce the user's service experience for live streaming and other low-latency services. Therefore, it is very meaningful to try to retain the low code rate capability of the video encoder as much as possible, and to reduce the encoding complexity as much as possible.

[0077] For ease of understanding, the video frame to be encoded in the video data in the embodiments of the present application can be referred to as a target video frame, and the basic coding unit to be encoded in the target video frame can be referred to as a to-be-encoded unit. Among them, the to-be-encoded unit can be a coding unit (Coding Unit, CU) to be encoded, and the CU is a basic coding unit in the H.266 video encoder.

[0078] It can be understood that the target video frame can have different video frame types (i.e., frame types), and the reference frame selected when encoding the to-be-encoded unit in the target video frame is different when the frame type of the target video frame is different. The frame type of the target video frame here can include a first type, a second type and a third type. Among them, the frame type of the intra picture (I frame) can be referred to as the first type, the frame type of the bi-directional interpolated prediction frame (B frame) can be referred to as the second type, and the frame type of the predictive-frame (P frame) can be referred to as the third type.

[0079] It can be understood that the video data in the embodiments of the present application can be any video data that needs to be encoded in a service scenario. For example, the video data can be directly collected by the image collector (for example, a camera) in the terminal device, the video data can be real-time recorded by the image collector in the terminal device during live streaming / video call, the video data can be downloaded by the terminal device on the network, and the video data can be obtained by the terminal device from the server during the game / meeting.

[0080] For ease of understanding, further, please refer to Figure 2 , Figure 2This is a schematic diagram illustrating a data interaction scenario provided in an embodiment of this application. For example... Figure 2 The server 20a shown can be the one described above. Figure 1 The server 2000 in the corresponding embodiment, such as Figure 2 The terminal device 20b shown can be the one described above. Figure 1 The target terminal device in the corresponding embodiment. For ease of understanding, this application embodiment uses terminal device 20b as the sending end for sending video data and server 20a as the receiving end for receiving video data as an example for explanation.

[0081] It should be understood that terminal device 20b can acquire video data (e.g., video data 21a). Video data 21a may include one or more video frames; this embodiment does not limit the number of video frames in video data 21a. Furthermore, terminal device 20b needs to encode video data 21a using a video encoder (e.g., an H.266 video encoder) to generate a video stream associated with video data 21a.

[0082] like Figure 2 As shown, when the terminal device 20b encodes the video data 21a, it can obtain the target video frame (e.g., video frame 21b) to be encoded from the video data 21a, then obtain the unit to be encoded from the video frame 21b, and then obtain the associated encoding units related to the unit to be encoded from the video data 21a. The encoding order of the associated encoding units is earlier than the encoding order of the unit to be encoded (i.e., all associated encoding units have been encoded before the unit to be encoded begins encoding), and the associated encoding units are adjacent to the unit to be encoded. It should be understood that this application embodiment does not limit the number of associated encoding units; this application embodiment uses the existence of associated encoding units related to the unit to be encoded as an example for explanation.

[0083] Furthermore, terminal device 20b can encode the units to be encoded based on the encoding strategy of the video encoder to obtain the compressed bitstream corresponding to the units to be encoded. It should be understood that when terminal device 20b completes the encoding process of each unit to be encoded in the target video frame, it can obtain the compressed bitstream corresponding to each unit to be encoded separately. Then, when completing the encoding process of each video frame in video data 21a, it can encapsulate the compressed bitstream corresponding to each unit to be encoded into a video bitstream associated with video data 21a, thereby completing the encoding process of video data 21a.

[0084] The encoding strategy of the video encoder can include an intra prediction mode (i.e., intra prediction encoding) and an inter prediction mode (i.e., inter prediction encoding), which can be collectively referred to as an encoding prediction technique. Intra prediction (i.e., intra encoding) indicates that the encoding of a current frame does not refer to information of other frames, and inter prediction (i.e., inter encoding) indicates that the current frame is predicted using information of adjacent frames. When performing inter prediction on a to-be-encoded unit in a target video frame, the video encoder can select one frame as a reference frame from a forward reference frame list or a backward reference frame list (i.e., single-direction prediction), or can select one or two frames as reference frames from the two reference frame lists (i.e., double-direction prediction). Alternatively, the single-direction prediction can also be that one frame is selected as a reference frame from the forward reference frame list, and one frame cannot be selected as a reference frame from the backward reference frame list. The second type of video frame (i.e., B frame) can use single-direction prediction or double-direction prediction for inter prediction, and the third type of video frame (i.e., P frame) can use single-direction prediction for inter prediction.

[0085] It should be understood that the embodiments of the present application can be applied to the selection of reference frames in the inter prediction mode, as shown in Figure 2 The terminal device 20b can determine, according to the associated encoding unit, a set of available reference frames corresponding to the to-be-encoded unit in the video data 21a. The set of available reference frames can include one or more video frames, and the number of reference frames in the set of available reference frames is not limited by the embodiments of the present application. The reference frames in the set of available reference frames are all encoded video frames. Alternatively, if there is no associated encoding unit related to the to-be-encoded unit, the terminal device 20b can determine, as the set of available reference frames corresponding to the to-be-encoded unit, a video frame in the video data 21a that is earlier in the encoding order than the target video frame.

[0086] Further, as shown in Figure 2 The terminal device 20b can determine whether there is a parent encoding unit to which the to-be-encoded unit belongs in the video frame 21b. If there is a parent encoding unit to which the to-be-encoded unit belongs in the target video frame, the terminal device 20b can determine, according to the parent encoding unit and the set of available reference frames, a set of candidate reference frames corresponding to the to-be-encoded unit. The set of candidate reference frames can include one or more video frames, and the number of reference frames in the set of candidate reference frames is not limited by the embodiments of the present application. The reference frames in the set of candidate reference frames are all encoded video frames. Alternatively, if there is no parent encoding unit to which the to-be-encoded unit belongs, the terminal device 20b can determine, as the set of candidate reference frames corresponding to the to-be-encoded unit, the set of available reference frames.

[0087] Further, as shown in Figure 2As shown, after determining the candidate reference frame set, the terminal device 20b can traverse the target reference frame in the candidate reference frame set, encode the to-be-encoded unit based on the target reference frame, obtain the compressed code stream corresponding to the to-be-encoded unit, and further obtain the video code stream associated with the video data 21a. At this time, the terminal device 20b can send the video code stream associated with the video data 21a to the server 20a, so that the server 20a can decode the video code stream through the video decoder when receiving the video code stream to obtain the video data 21a.

[0088] It can be understood that the compressed code stream corresponding to the to-be-encoded unit can include but is not limited to a motion vector, a reference frame index, a reference frame list, etc. The server 20a can use the information in the compressed code stream to generate the inter prediction pixel value, that is, to restore the to-be-encoded unit. The reference frame index can indicate the index of a specific reference frame in the reference frame list. Through the reference frame index, the specific reference frame used for encoding the to-be-encoded unit can be located in the reference frame list. Here, the reference frame list can be a full reference frame list or a candidate reference frame list, which is not limited in the present application.

[0089] It can be seen that in the present application, when the to-be-encoded unit in the target video frame needs to be encoded, the associated encoding unit associated with the to-be-encoded unit and the parent encoding unit to which the to-be-encoded unit belongs can be obtained from the target video frame. According to the associated encoding unit and the parent encoding unit, the candidate reference frame set corresponding to the to-be-encoded unit is determined. It can be understood that the reference frames in the candidate reference frame set are the reference frames associated with the associated encoding unit and the parent encoding unit. Considering the correlation between the to-be-encoded unit and the video content in the parent encoding unit and the correlation between the to-be-encoded unit and the video content in the associated encoding unit, it can be known that the reference frames in the candidate reference frame set have high content similarity with the target video frame to which the to-be-encoded unit belongs. Therefore, when the to-be-encoded unit in the target video frame is encoded based on the candidate reference frame set, the candidate reference frame set can be traversed without traversing all the video frames that have been encoded. This not only ensures the encoding effect of the target video frame, but also simplifies the selection of reference frames, effectively reduces the proportion of reference frame selection in the overall encoding process complexity, thereby reducing the computational complexity of the inter-frame encoding process of the video encoder, and further reducing the encoding time (i.e., improving the encoding efficiency), the computational resource and the bandwidth resource.

[0090] The specific implementation of the computer device with video encoding function in determining the candidate reference frame set in the video data can be referred to the following Figures 3-8 corresponding embodiments.

[0091] Further, please refer to Figure 3 ,Figure 3 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. The method can be executed by a server, a terminal device, or both. The server can be one of the aforementioned... Figure 2 The server 20a in the corresponding embodiment can be the terminal device described above. Figure 2 The corresponding embodiment is terminal device 20b. For ease of understanding, this application embodiment is described using the method executed by the terminal device as an example. The video data processing method may include the following steps S101-S103:

[0092] Step S101: Obtain the target video frame from the video data, and obtain the associated coding unit associated with the unit to be encoded in the target video frame;

[0093] Specifically, the terminal device can obtain video frames to be encoded from video data and identify the obtained video frames as target video frames. Further, the terminal device can perform image block partitioning processing on the target video frames using a video encoder to obtain one or more image blocks (i.e., coding blocks) of the target video frames, and then obtain the unit to be encoded from one or more image blocks. The purpose of image block partitioning is to more accurately process predictions, using smaller image blocks for slightly moving parts and larger image blocks for static backgrounds. In this embodiment, the coding unit CU can be referred to as an image block. Further, the terminal device can, based on the position of the unit to be encoded within one or more image blocks, obtain image blocks that satisfy association conditions from one or more image blocks, and identify the obtained image blocks as associated coding units. The encoding order of the associated coding units is earlier than the encoding order of the unit to be encoded, and the associated coding units are adjacent to the unit to be encoded. Therefore, the association condition refers to the condition that the encoding order of the obtained image blocks is earlier than the encoding order of the unit to be encoded, and the obtained image blocks are adjacent to the unit to be encoded.

[0094] For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a scenario for obtaining associated coding units provided in an embodiment of this application. For example... Figure 4The position diagram 44a shown can include a to-be-encoded unit and associated encoding units. In the position diagram 44a, E can represent the to-be-encoded unit, and A, B, C, and D can represent the associated encoding units, which can also be referred to as neighboring CUs. A can represent a CU on the left side, B can represent a CU on the top left side, C can represent a CU on the top side, and D can represent a CU on the top right side. In addition, the to-be-encoded unit is drawn large and the associated encoding units are drawn small in the position diagram 44a, which is merely illustrative. In practice, the associated encoding units are not necessarily smaller than the to-be-encoded unit. It should be understood that the present application does not limit the sizes of the to-be-encoded unit and the associated encoding units.

[0095] It can be understood that, according to the encoding standard, the associated encoding unit A, the associated encoding unit B, the associated encoding unit C, and the associated encoding unit D have all been encoded before the to-be-encoded unit E starts to be encoded. It should be understood that the present application does not limit the sizes of the to-be-encoded unit E, the associated encoding unit A, the associated encoding unit B, the associated encoding unit C, and the associated encoding unit D. The sizes of the to-be-encoded unit E, the associated encoding unit A, the associated encoding unit B, the associated encoding unit C, and the associated encoding unit D are determined by the partitioning manner of the video encoder, for example, the partitioning manner can be a quadtree, a binary tree, a ternary tree, and the like, and the present application does not limit the partitioning manner.

[0096] Optionally, the present application can also add a limit to the size of the to-be-encoded unit (i.e., add a size limit). For example, when the number of pixels of the to-be-encoded unit exceeds a pixel threshold (for example, 512), the fast strategy provided by the present application is executed, and the fast strategy is step S101-step S103 in the present application.

[0097] As Figure 4 The image block partitioning diagram 44b shown can be an image block partitioning diagram of a target video frame to be encoded obtained from video data. The image block partitioning diagram 44b can be a diagram obtained by image block partitioning processing of an H.266 video encoder. The present application does not limit the diagram obtained by the image block partitioning processing, which is taken as the image block partitioning diagram 44b for illustration. The image block partitioning diagram obtained by other video encoders can refer to the description of the image block partitioning diagram 44b. Similarly, the present application does not limit the specific partitioning form in the image block partitioning diagram 44b, and the specific partitioning form in the image block partitioning diagram 44b meets the partitioning manner specified by the video encoder.

[0098] In the image block division diagram 44b, the target video frame can be divided into (image block 40a, image block 40b, image block 40c, image block 40d, image block 40e, image block 40f, image block 40g, image block 40h, image block 40i, image block 40j), (image block 41a), (image block 42a, image block 42b, image block 42c, image block 42d, image block 42e), (image block 43a, image block 43b, image block 43c, image block 43d, image block 43e, image block 43f, image block 43g, image block 43h, image block 43i, image block 43j).

[0099] In the image block division diagram 44b, the target video frame can be divided into (image block 40a, image block 40b, image block 40c, image block 40d, image block 40e, image block 40f, image block 40g, image block 40h, image block 40i, image block 40j), (image block 41a), (image block 42a, image block 42b, image block 42c, image block 42d, image block 42e), (image block 43a, image block 43b, image block 43c, image block 43d, image block 43e, image block 43f, image block 43g, image block 43h, image block 43i, image block 43j).

[0100] For the convenience of understanding, as shown in the image block 42e in the image block division diagram 44b can be used as the to-be-encoded unit E, at this time, the image block 42d can be used as the associated encoding unit A, the image block 40g can be used as the associated encoding unit B, the image block 40j can be used as the associated encoding unit C, and the image block 41a can be used as the associated encoding unit D. Figure 4 As shown in the image block 42d in the image block division diagram 44b can be used as the to-be-encoded unit E, at this time, (image block 42a, image block 42b, image block 42c) can be used as the associated encoding unit A, the image block 40g can be used as the associated encoding unit B, the image block 40j can be used as the associated encoding unit C, and the image block 41a can be used as the associated encoding unit D. Figure 4 As shown in the image block 42d in the image block division diagram 44b can be used as the to-be-encoded unit E, at this time, (image block 42a, image block 42b, image block 42c) can be used as the associated encoding unit A, the image block 40g can be used as the associated encoding unit B, the image block 40j can be used as the associated encoding unit C, and the image block 41a can be used as the associated encoding unit D.

[0101] It should be understood that the terminal device can obtain the encoding strategy of the video encoder (for example, the H.266 video encoder), and encode the to-be-encoded unit based on the encoding strategy of the video encoder. Among them, the encoding modes associated with the encoding strategy can include an inter-prediction mode and an intra-prediction mode, so that the terminal device can determine the reference video frame associated with the to-be-encoded unit based on the frame type of the target video frame when performing inter-prediction processing on the to-be-encoded unit. Among them, different video compression standards can correspond to different reference video frames. Among them, if the frame type of the target video frame is a B frame (i.e., the second type) or a P frame (i.e., the third type), the terminal device can perform the following steps S102-S103; optionally, if the frame type of the target video frame is an I frame (i.e., the first type), the terminal device does not need to perform the following steps S102-S103.

[0102] Among them, the H.266 video encoder encodes in a block division manner, and when encoding, an image block is divided into a plurality of CUs. The CU can be nested and divided, and a CU can be further divided into a plurality of CUs as a new image block until the minimum CU size limit is reached. Therefore, the CU is the basic unit of encoding prediction.

[0103] Among them, the number of associated encoding units is S, and S here can be a non-negative integer. It can be understood that if S is a positive integer, the terminal device can perform the following steps S102-S103. Optionally, if S is equal to 0, the terminal device can obtain a full reference frame set constructed for the to-be-encoded unit in the video data, and determine the full reference frame set as the available reference frame set corresponding to the to-be-encoded unit.

[0104] Step S102, determining the available reference frame set corresponding to the to-be-encoded unit in the video data according to the associated encoding units;

[0105] Specifically, if the prediction encoding types of the S associated encoding units are all inter-prediction, and S is equal to the encoding unit number threshold associated with the video encoder, the terminal device can obtain a full reference frame set constructed for the to-be-encoded unit in the video data. Among them, the full reference frame set includes a forward full reference frame set and a backward full reference frame set. Further, the terminal device can obtain the forward reference frame closest to the position of the target video frame in the forward full reference frame set, and obtain the backward reference frame closest to the position of the target video frame in the backward full reference frame set. Further, the terminal device can merge the reference frames used by the S associated encoding units, the forward reference frame and the backward reference frame to obtain the available reference frame set corresponding to the to-be-encoded unit.

[0106] It can be understood that when inter prediction is performed, the video encoder can construct a reference frame list for a target video frame, the reference frame list including two parts, one part being a forward reference frame list (i.e., a forward full reference frame set) and the other part being a backward reference frame list (i.e., a backward full reference frame set). The forward reference frame list includes video frames that are both in coding order and in playing order before the current frame (i.e., the target video frame), and the backward reference frame list includes video frames that are in coding order before the current frame (i.e., the target video frame) and in playing order after the current frame (i.e., the target video frame). Wherein, the number of video frames in the reference frame list is not limited in the embodiments of the present application.

[0107] It can be understood that according to the coding rule, the likelihood of a video frame in the full reference frame list being selected as the best reference frame is not the same. Since there is content continuity in video data, the likelihood of the same video frame selecting the same reference frame for a continuous region is very high (i.e., the reference frame region of the same image block region is probably the same). Therefore, the embodiments of the present application propose to limit the reference frames available to the current CU (i.e., the to-be-coded unit) using the reference frames used by the neighboring CUs (i.e., associated coding units). Specifically, before the video encoder performs inter prediction with motion search on the current CU, the video encoder first counts the reference frames used by the neighboring CUs. Only the reference frames used by the neighboring CUs can be selected as the reference frames of the current frame.

[0108] It can be understood that according to the coding rule, the likelihood of a video frame in the full reference frame list being selected as the best reference frame is not the same. Since the likelihood of the content similarity of the video frames being closer in playing order being higher is greater, the frames closer in playing order to the current to-be-coded frame (i.e., the target video frame) are more likely to be selected as reference frames. Therefore, in order to minimize the impact on the coding effect, the two frames closest in playing distance to the current frame (i.e., the target video frame) in the reference frame list (i.e., the forward full reference frame set and the backward full reference frame set) are always available, even if the two frames are not referenced by the neighboring CUs (i.e., the two frames do not belong to the reference frames used by the S associated coding units). Wherein, the video frame closest in playing distance to the target video frame in the forward full reference frame set is the forward reference frame, and the video frame closest in playing distance to the target video frame in the backward full reference frame set is the backward reference frame.

[0109] For example, the forward full reference frame list can include video frame X1, video frame X2 and video frame X3, the backward full reference frame list can include video frame X4, video frame X5 and video frame X6, the encoding order of video frame X1, video frame X2, video frame X3, video frame X4, video frame X5 and video frame X6 is all before the target video frame, the playing order of video frame X1, video frame X2 and video frame X3 is before the target video frame, the playing order of video frame X4, video frame X5 and video frame X6 is after the target video frame, video frame X1, video frame X2, video frame X3, the target video frame, video frame X4, video frame X5 and video frame X6 are arranged in the playing order, at this time, the terminal device can determine video frame X3 as the forward reference frame and video frame X4 as the backward reference frame.

[0110] The available reference frame set includes a forward available reference frame set and a backward available reference frame set. It should be understood that the terminal device merges the reference frames, the forward reference frames and the backward reference frames used by the S associated coding units to obtain the available reference frame set corresponding to the to-be-encoded unit: the terminal device can determine the union of the reference frames used by the S associated coding units as the associated reference frame set. Wherein, the terminal device can determine the reference frames in the associated reference frame set whose playing order is earlier than the target video frame as the forward associated reference frame set, and determine the reference frames in the associated reference frame set whose playing order is later than the target video frame as the backward associated reference frame set, and the forward associated reference frame set and the backward associated reference frame set can be collectively referred to as the associated reference frame set. In other words, the terminal device can add the reference frames in the reference frames used by the S associated coding units whose playing order is earlier than the target video frame to the forward associated reference frame set, and add the reference frames in the reference frames used by the S associated coding units whose playing order is later than the target video frame to the backward associated reference frame set. Further, if the associated reference frame set does not include the forward reference frame and the backward reference frame, the terminal device can determine the associated reference frame set, the forward reference frame and the backward reference frame as the available reference frame set corresponding to the to-be-encoded unit. In other words, if the associated reference frame set does not include the forward reference frame and the backward reference frame, the terminal device can determine the reference frames in the associated reference frame set whose playing order is earlier than the target video frame and the forward reference frame as the forward available reference frame set corresponding to the to-be-encoded unit, and determine the reference frames in the associated reference frame set whose playing order is later than the target video frame and the backward reference frame as the backward available reference frame set corresponding to the to-be-encoded unit.

[0111] For ease of understanding, the embodiments of the present application are described by taking an example that the forward associated reference frame set includes the reference frame earlier than the target video frame in the playing order and the backward associated reference frame set includes the reference frame later than the target video frame in the playing order. Optionally, if the forward associated reference frame set does not include the reference frame (i.e., the associated reference frame set does not include the reference frame earlier than the target video frame in the playing order), the terminal device can determine the forward reference frame as the forward available reference frame set corresponding to the to-be-encoded unit; if the backward associated reference frame set does not include the reference frame (i.e., the associated reference frame set does not include the reference frame later than the target video frame in the playing order), the terminal device can determine the backward reference frame as the backward available reference frame set corresponding to the to-be-encoded unit.

[0112] For example, the S is taken as 3 for example, the S associated encoding units can include the associated encoding unit P1, the associated encoding unit P2 and the associated encoding unit P3, the reference frame used by the associated encoding unit P1 can be the reference frame W1 and the reference frame W2, the reference frame used by the associated encoding unit P2 can be the reference frame W1, the reference frame used by the associated encoding unit P3 can be the reference frame W3. Therefore, the union of the reference frames used by the S associated encoding units can be the reference frame W1, the reference frame W2 and the reference frame W3, the reference frame W1, the reference frame W2 and the reference frame W3 can be used to constitute the associated reference frame set, wherein if the playing order of the reference frame W1 and the reference frame W3 is earlier than that of the target video frame and the playing order of the reference frame W2 is later than that of the target video frame, the reference frame W1 and the reference frame W3 can be used to constitute the forward associated reference frame set, and the reference frame W2 can be used to constitute the backward associated reference frame set.

[0113] Optionally, if the associated reference frame set includes the forward reference frame and the backward reference frame, the associated reference frame set is determined as the available reference frame set corresponding to the to-be-encoded unit. In other words, if the associated reference frame set includes the forward reference frame and the backward reference frame, the reference frame earlier than the target video frame in the playing order in the associated reference frame set is determined as the forward available reference frame set corresponding to the to-be-encoded unit; and the reference frame later than the target video frame in the playing order in the associated reference frame set is determined as the backward available reference frame set corresponding to the to-be-encoded unit.

[0114] Optionally, if the associated reference frame set includes the forward reference frame and does not include the backward reference frame, the associated reference frame set and the backward reference frame are determined as the available reference frame set corresponding to the to-be-encoded unit. In other words, if the associated reference frame set includes the forward reference frame and does not include the backward reference frame, the reference frame earlier than the target video frame in the playing order in the associated reference frame set is determined as the forward available reference frame set corresponding to the to-be-encoded unit; and the reference frame later than the target video frame in the playing order in the associated reference frame set and the backward reference frame are determined as the backward available reference frame set corresponding to the to-be-encoded unit.

[0115] For ease of understanding, the embodiments of the present application are described by taking an example that the backward associated reference frame set includes the reference frame with a later playing order than the target video frame. Optionally, if the backward associated reference frame set does not include the reference frame, the terminal device can determine the backward reference frame as the backward available reference frame set corresponding to the to-be-encoded unit.

[0116] Optionally, if the associated reference frame set does not include the forward reference frame and includes the backward reference frame, the associated reference frame set and the forward reference frame are determined as the available reference frame set corresponding to the to-be-encoded unit. In other words, if the associated reference frame set does not include the forward reference frame and includes the backward reference frame, the reference frame with an earlier playing order than the target video frame in the associated reference frame set and the forward reference frame are determined as the forward available reference frame set corresponding to the to-be-encoded unit; the reference frame with a later playing order than the target video frame in the associated reference frame set is determined as the backward available reference frame set corresponding to the to-be-encoded unit.

[0117] For ease of understanding, the embodiments of the present application are described by taking an example that the forward associated reference frame set includes the reference frame with an earlier playing order than the target video frame. Optionally, if the forward associated reference frame set does not include the reference frame, the terminal device can determine the forward reference frame as the forward available reference frame set corresponding to the to-be-encoded unit.

[0118] Optionally, if there is an associated coding unit with an intra prediction type in the S associated coding units, or S is less than the threshold value of the number of coding units associated with the video encoder, the terminal device can determine the full reference frame set as the available reference frame set corresponding to the to-be-encoded unit.

[0119] It should be understood that the embodiments of the present application do not limit the specific value of the threshold value of the number of coding units. For example, when the video encoder is an H.266 video encoder, the threshold value of the number of coding units can be equal to 4.

[0120] In step S103, if there is a parent coding unit to which the to-be-encoded unit belongs in the target video frame, the candidate reference frame set corresponding to the to-be-encoded unit is determined according to the parent coding unit and the available reference frame set.

[0121] Specifically, the specific process that the terminal device determines the candidate reference frame set corresponding to the to-be-encoded unit according to the parent coding unit and the available reference frame set can be described as follows: if the prediction coding type of the parent coding unit is inter prediction and the inter coding mode of the parent coding unit is inter skip mode, the terminal device can match the reference frame used by the parent coding unit with the available reference frame set. Further, if there is an intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set, the terminal device can determine the intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit. Alternatively, if there is no intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit.

[0122] It can be understood that the inter coding mode of the parent coding unit can be inter skip mode (i.e., SKIP mode), inter merge mode (i.e., Merge mode), AMVP (Advanced Motion Vector Prediction) mode, etc. When the parent coding unit is inter predicted using the SKIP mode, the parent coding unit does not need to transmit residual coefficients and MVD (Motion Vector Difference); when the parent coding unit is inter predicted using the Merge mode, the motion parameters of the parent coding unit can be directly obtained from a neighboring already-coded coding unit (i.e., an associated coding unit associated with the parent coding unit, which means the associated coding unit obtained when the parent coding unit is taken as the to-be-encoded unit); and when the parent coding unit is inter predicted using the AMVP mode, MVD is needed.

[0123] The SKIP mode is a special inter prediction technique that directly multiplexes a block of content of a reference frame. The SKIP mode is limited to improve the probability that the parent CU (i.e., the parent coding unit) and the child CU (i.e., the child coding unit) have the same reference frame. The parent CU and the child CU are relative. The parent coding unit can be regarded as the parent CU of the to-be-encoded unit, and the to-be-encoded unit can be regarded as the child CU of the parent coding unit. Therefore, the child coding unit herein is the to-be-encoded unit.

[0124] The available reference frame set includes a forward available reference frame set and a backward available reference frame set; and the candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set. The specific process in which the terminal device determines the candidate reference frame set corresponding to the to-be-encoded unit can be described as follows: the terminal device can determine, as the forward candidate reference frame set corresponding to the to-be-encoded unit, the reference frames that are earlier than the target video frame in the play order and that are in the reference frames used by the parent coding unit and in the forward available reference frame set; and determine, as the backward candidate reference frame set corresponding to the to-be-encoded unit, the reference frames that are later than the target video frame in the play order and that are in the reference frames used by the parent coding unit and in the backward available reference frame set.

[0125] For ease of understanding, the embodiments of the present application are described by taking the reference frames used by the parent coding unit as including the reference frames earlier than the target video frame in the play order and the reference frames later than the target video frame in the play order. Optionally, if the reference frames used by the parent coding unit do not include the reference frames earlier than the target video frame in the play order, the terminal device can determine, as the forward candidate reference frame set corresponding to the to-be-encoded unit, the forward available reference frame set; and if the reference frames used by the parent coding unit do not include the reference frames later than the target video frame in the play order, the terminal device can determine, as the backward candidate reference frame set corresponding to the to-be-encoded unit, the backward available reference frame set.

[0126] Optionally, if the prediction coding type of the parent coding unit is intra prediction, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit. Optionally, if the prediction coding type of the parent coding unit is inter prediction and the inter coding mode of the parent coding unit is not inter skip mode, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit. Optionally, if the parent coding unit to which the to-be-encoded unit belongs does not exist in the target video frame, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit.

[0127] For ease of understanding, please refer to Figure 4 , the image block 42e can be the to-be-encoded unit, and (the image block 42a, the image block 42e) can be the parent coding unit to which the image block 42e belongs; optionally, the image block 40d can be the to-be-encoded unit, and (the image block 40b, the image block 40d) can be the parent coding unit to which the image block 40d belongs (which means that the image block 40b and the image block 40d can belong to the same coding unit), or (the image block 40c, the image block 40d) can be the parent coding unit to which the image block 40d belongs (which means that the image block 40c and the image block 40d can belong to the same coding unit).

[0128] The candidate reference frame set is used to traverse the target reference frame for the to-be-encoded unit; and the target reference frame is used for encoding processing of the to-be-encoded unit. It should be understood that the terminal device can determine the video frames in the candidate reference frame set as the reference video frames associated with the to-be-encoded unit, and the video encoder does not determine how to specifically select the reference video frames for encoding processing. Different selection will have different encoding effects. In order to obtain the best encoding effect, the video encoder can perform encoding on each possible reference frame combination, which includes a very high complexity of motion search and motion compensation, so as to obtain the reference frame combination with the best encoding effect. In the embodiment of the present application, the encoding effect can be understood as distortion, and the encoding effect can be measured by using rate-distortion cost.

[0129] The candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set. It should be understood that the specific process of the terminal device encoding the target reference frame in the candidate reference frame set can be described as follows: the terminal device can determine the video frame type of the target video frame. The video frame type of the target video frame can be used to guide the video encoder to select the reference frame for encoding processing of the target video frame in the candidate reference frame set. The reference frame obtained by traversing in the candidate reference frame set can be referred to as the target reference frame in the embodiment of the present application. Further, if the video frame type is a single prediction type (i.e., the third type), the terminal device can traverse the target reference frame for encoding processing of the to-be-encoded unit in the forward candidate reference frame set or the backward candidate reference frame set. Alternatively, if the video frame type is a bidirectional prediction type (i.e., the second type), the terminal device can traverse the target reference frame for encoding processing of the to-be-encoded unit in the forward candidate reference frame set, the backward candidate reference frame set, or a bidirectional reference frame set. The bidirectional reference frame set includes the forward candidate reference frame set and the backward candidate reference frame set. In other words, if the video frame type is a bidirectional prediction type, the terminal device can traverse the target reference frame for encoding processing of the to-be-encoded unit in the forward candidate reference frame set or the backward candidate reference frame set; or the terminal device can traverse the target reference frame for encoding processing of the to-be-encoded unit in the forward candidate reference frame set and the backward candidate reference frame set.

[0130] For ease of understanding, please refer to Figure 5 , Figure 5 is a scene diagram provided by an embodiment of the present application for obtaining a forward reference frame and a backward reference frame. As Figure 5The diagram illustrates bidirectional prediction of the unit to be encoded. Video frame 53c can be the target video frame, video frame set 53a can be the forward candidate reference frame set corresponding to the target video frame 53c, and video frame set 53b can be the backward candidate reference frame set corresponding to the target video frame 53c. The forward candidate reference frame set 53a can include multiple video frames, and the backward candidate reference frame set 53a can also include multiple video frames. This embodiment does not limit the number of video frames in the forward and backward candidate reference frame sets 53a. For ease of understanding, this embodiment uses the example where both the forward and backward candidate reference frame sets 53a contain three video frames. The forward candidate reference frame set 53a can include video frames 50a, 50b, and 50c, and the backward candidate reference frame set 53a can include video frames 51a, 51b, and 51c.

[0131] like Figure 5 As shown, the target video frame 53c may include the unit to be encoded 52a, the video frame 50c may include the encoding unit 52b, and the video frame 51a may include the encoding unit 52c. When the video frame 50c and the video frame 51a are determined as reference frames used by the target video frame, the video encoder can perform encoding processing on the unit to be encoded 52a based on the encoding units 52b and 52c. At this time, the unit to be encoded selects one frame as a reference frame from the forward candidate reference frame set 53a and the backward candidate reference frame set 53b respectively and performs motion search. The encoding units 52b and 52c can be referred to as reference blocks.

[0132] Optionally, when the unit to be encoded selects one frame as a reference frame from the forward candidate reference frame set 53a and the backward candidate reference frame set 53b respectively and performs motion search, the video encoder can determine video frame 50a and video frame 51a as reference frames used by the target video frame, the video encoder can also determine video frame 50b as a reference frame used by the target video frame, and the video encoder can also determine video frame 51b as a reference frame used by the target video frame. There are no limitations here.

[0133] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating a fast reference frame selection process provided in an embodiment of this application. For example... Figure 6As shown in FIG. 13, the terminal device can perform step S21 to obtain the to-be-encoded unit in the target video frame through step S21. Further, the terminal device can perform step S22 to obtain the associated encoding units associated with the to-be-encoded unit in the target video frame through step S22. If all the associated encoding units exist (i.e., the number of the associated encoding units is equal to the encoding unit number threshold) and all the associated encoding units are encoded by using inter prediction (i.e., the prediction encoding types of the associated encoding units are all inter prediction), the terminal device can perform step S23. Otherwise, the terminal device skips performing step S23 and step S24.

[0134] As shown in FIG. 13, Figure 6 The terminal device can disable, through step S23, the reference frames in the reference frame list (i.e., the full reference frame list) that are not used by the associated encoding units, i.e., obtain, through step S23, the reference frames used by the associated encoding units. Further, the terminal device can perform step S24 to set, through step S24, the frame farthest from the current frame (i.e., the target video frame) in the forward reference frame list (i.e., the forward full reference frame list) and the backward reference frame list (i.e., the backward full reference frame list) as available. At this time, the terminal device can set the frame farthest from the current frame (i.e., the target video frame) in the forward full reference frame list (i.e., the forward reference frame) and the frame farthest from the current frame (i.e., the target video frame) in the backward full reference frame list (i.e., the backward reference frame) as available, i.e., the terminal device can merge the reference frames used by the associated encoding units, the forward reference frame, and the backward reference frame to obtain the available reference frame set corresponding to the to-be-encoded unit.

[0135] As shown in FIG. 13, Figure 6 The terminal device can perform step S25 to obtain, through step S25, the parent encoding unit to which the to-be-encoded unit belongs in the target video frame. If the parent encoding unit exists, the parent encoding unit uses the inter-frame skip mode (i.e., the inter-frame encoding mode of the parent encoding unit is the inter-frame skip mode), and the reference frame used by the parent encoding unit is available (i.e., the reference frame used by the parent encoding unit is available to the to-be-encoded unit, indicating that the reference frame used by the parent encoding unit is used by the associated encoding unit (or is farthest from the target video frame), and the reference frame used by the parent encoding unit exists in the full reference frame list), the terminal device can perform step S26. Otherwise, the terminal device skips performing step S26.

[0136] As shown in FIG. 13, Figure 6 The terminal device can disable, through step S26, the reference frames in the reference frame list except for the reference frame used by the parent encoding unit, i.e., obtain, through step S26, the intersection between the reference frame used by the parent encoding unit and the reference frames in the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-encoded unit.

[0137] Further, as shown in FIG. 13, Figure 6As shown, the terminal device can perform step S27, which indicates that the fast reference frame selection provided by the embodiments of the present application ends, and the output result of the fast reference frame selection is a candidate reference frame set, which can be used to select a reference frame for the to-be-encoded unit in the target video frame for encoding processing.

[0138] Optionally, the terminal device can further obtain a target video frame from the video data, and obtain a to-be-encoded unit from the target video frame. Further, if there is a parent coding unit to which the to-be-encoded unit belongs in the target video frame, the terminal device can determine a set of available reference frames corresponding to the to-be-encoded unit in the video data according to the parent coding unit and the full reference frame set. The specific process of determining the set of available reference frames according to the parent coding unit and the full reference frame set can be referred to the description of determining the candidate reference frame set according to the parent coding unit and the available reference frame set, which will not be described here. Further, the terminal device can obtain an associated coding unit associated with the to-be-encoded unit, and determine a candidate reference frame set corresponding to the to-be-encoded unit according to the associated coding unit and the available reference frame set. The specific process of determining the candidate reference frame set according to the associated coding unit and the available reference frame set can be referred to the description of determining the available reference frame set according to the associated coding unit and the full reference frame set, which will not be described here.

[0139] It can be seen that the fast reference frame selection algorithm can fully consider the prediction correlation of the video content in the target video frame (i.e., the reference frames of the coding units in the same frame are probably the same), on the one hand, the associated coding units adjacent to the to-be-encoded unit are obtained, the reference frames of the to-be-encoded unit are limited by using the associated coding units, and a set of available reference frames is obtained, on the other hand, the parent coding unit to which the to-be-encoded unit belongs is obtained, and the candidate reference frame set is obtained by limiting the child coding unit by using the parent coding unit. It can be understood that by using the fast reference frame selection algorithm provided in the embodiments of the present application, the candidate reference frame set that fuses the reference frames used by the associated coding units and the reference frames used by the parent coding units can be selected from all video frames. Because the reference frames in the candidate reference frame set are determined by the associated coding units and the parent coding units, the reference frames in the candidate reference frame set have a high content similarity with the target video frame. In this way, the embodiments of the present application can not need to traverse all the encoded video frames, but traverse the video frames in the candidate reference frame set with fewer frame numbers, which not only reduces the traversal time consumption, but also when traversing the candidate reference frame set to which the reference frames with high content similarity belong, the target reference frame with the best encoding effect can be obtained in the traversal result, so that the encoding effect and the encoding efficiency of the target video frame can be considered at the same time (i.e., while ensuring the encoding efficiency of the target video frame, the encoding effect of the target video frame is improved, and while ensuring the encoding effect of the target video frame, the encoding efficiency of the target video frame is improved).

[0140] Further, please refer to Figure 7 , Figure 7 is a flowchart of a video data processing method provided by the embodiments of the present application. The video data processing method can include the following steps S1021-S1023, and steps S1021-S1023 are Figure 3 a specific embodiment of step S102 in the embodiments corresponding to the above.

[0141] Step S1021, obtaining the prediction coding type of S associated coding units;

[0142] The prediction coding type can include an inter prediction type (i.e., inter prediction) and an intra prediction type (i.e., intra prediction). The S associated coding units can have the same prediction coding type or different prediction coding types, which are not limited in this application. For example, assuming that S is equal to 4, the 4 associated coding units can include an associated coding unit P1, an associated coding unit P2, an associated coding unit P3, and an associated coding unit P4. The prediction coding types of the associated coding unit P1, the associated coding unit P2, and the associated coding unit P3 can all be inter prediction, and the prediction coding type of the associated coding unit P4 can be intra prediction. Alternatively, the prediction coding types of the associated coding unit P1, the associated coding unit P2, the associated coding unit P3, and the associated coding unit P4 can all be inter prediction.

[0143] In step S1022, if there is an associated coding unit with an intra prediction type in the S associated coding units, a full reference frame set constructed for the to-be-coded unit is obtained from the video data, and the full reference frame set is determined as the available reference frame set corresponding to the to-be-coded unit.

[0144] Specifically, if there is an associated coding unit with an intra prediction type in the S associated coding units, the terminal device can obtain an encoded video frame earlier than the target video frame in the video data. The number of the encoded video frame is not limited in this application. Further, if the playback order of the encoded video frame is earlier than that of the target video frame, the terminal device can add the encoded video frame to the forward full reference frame set. Alternatively, if the playback order of the encoded video frame is later than that of the target video frame, the terminal device can add the encoded video frame to the backward full reference frame set. In other words, the terminal device can add the encoded video frame earlier than the target video frame to the forward full reference frame set, and add the encoded video frame later than the target video frame to the backward full reference frame set. The full reference frame set includes the forward full reference frame set and the backward full reference frame set, that is, the forward full reference frame set and the backward full reference frame set can be collectively referred to as the full reference frame set. Further, the terminal device can determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit.

[0145] For the convenience of understanding, the embodiments of the present application are described by taking an example that the forward full reference frame set includes the coded video frames with earlier play order than the target video frame and the backward full reference frame set includes the coded video frames with later play order than the target video frame. Optionally, if there is no coded video frame with earlier play order than the target video frame, the terminal device can determine the backward full reference frame set as the available reference frame set corresponding to the to-be-encoded unit, i.e., the forward full reference frame set is empty, in other words, the terminal device can determine the backward full reference frame set and the forward full reference frame set as empty set as the available reference frame set corresponding to the to-be-encoded unit; if there is no coded video frame with later play order than the target video frame, the terminal device can determine the forward full reference frame set as the available reference frame set corresponding to the to-be-encoded unit, i.e., the backward full reference frame set is empty, in other words, the terminal device can determine the forward full reference frame set and the backward full reference frame set as empty set as the available reference frame set corresponding to the to-be-encoded unit.

[0146] In step S1023, if the prediction encoding types of the S associated coding units are all inter prediction, the terminal device determines the available reference frame set corresponding to the to-be-encoded unit in the video data according to the S associated coding units.

[0147] Specifically, if the prediction encoding types of the S associated coding units are all inter prediction, the terminal device can obtain a coding unit quantity threshold value associated with the video encoder. The coding unit quantity threshold value is greater than or equal to S. Further, if S is less than the coding unit quantity threshold value, the terminal device can obtain a full reference frame set constructed for the to-be-encoded unit in the video data, and determines the full reference frame set as the available reference frame set corresponding to the to-be-encoded unit. The specific process of obtaining the full reference frame set constructed for the to-be-encoded unit in the video data by the terminal device can refer to the description of step S1022 above, which will not be described here. If S is less than the coding unit quantity threshold value, it indicates that the to-be-encoded unit is located in the image boundary region of the target video frame. Optionally, if S is equal to the coding unit quantity threshold value, the terminal device can determine the available reference frame set corresponding to the to-be-encoded unit in the video data according to the reference frames used by the S associated coding units.

[0148] It should be understood that the specific process of determining the available reference frame set corresponding to the to-be-encoded unit in the video data by the terminal device according to the reference frames used by the S associated encoding units can be described as follows: the terminal device can obtain a full reference frame set constructed for the to-be-encoded unit in the video data. The full reference frame set includes a forward full reference frame set and a backward full reference frame set. The specific process of obtaining the full reference frame set constructed for the to-be-encoded unit in the video data by the terminal device can refer to the description of step S1022 above, and will not be described here. Further, the terminal device can obtain the forward reference frame closest to the position of the target video frame in the forward full reference frame set, and obtain the backward reference frame closest to the position of the target video frame in the backward full reference frame set. Further, the terminal device can merge the reference frames used by the S associated encoding units, the forward reference frame, and the backward reference frame to obtain the available reference frame set corresponding to the to-be-encoded unit.

[0149] It should be understood that the terminal device can sort the reference frames in the forward full reference frame set according to the playback order to obtain sorted reference frames of the forward full reference frame set, and determine the last reference frame in the sorted reference frames of the forward full reference frame set as the forward reference frame closest to the position of the target video frame. Similarly, the terminal device can sort the reference frames in the backward full reference frame set according to the playback order to obtain sorted reference frames of the backward full reference frame set, and determine the first reference frame in the sorted reference frames of the backward full reference frame set as the backward reference frame closest to the position of the target video frame.

[0150] The specific process of merging the reference frames used by the S associated encoding units, the forward reference frame, and the backward reference frame by the terminal device can refer to the description of step S102 in the corresponding embodiment, and will not be described here. Figure 3 The specific process of merging the reference frames used by the S associated encoding units, the forward reference frame, and the backward reference frame by the terminal device can refer to the description of step S102 in the corresponding embodiment, and will not be described here.

[0151] It can be understood that by determining the full reference frame set as the available reference frame set corresponding to the to-be-encoded unit, the traversal manner of selecting reference frames can still be used for the full reference frame set in the area where the prediction effect is poor (i.e., there is an associated encoding unit with an intra prediction prediction encoding type in the S associated encoding units, or S is less than the encoding unit quantity threshold), so as to minimize the influence on the encoding effect.

[0152] Optionally, the terminal device can acquire a threshold of a number of coding units associated with the video encoder. Further, if S is less than the threshold of the number of coding units, the terminal device can acquire, from the video data, a full reference frame set constructed for the to-be-coded unit, and determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit. Optionally, if S is equal to the threshold of the number of coding units, the terminal device can determine, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the S associated coding units. The specific process of determining, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the S associated coding units can be described as follows: the terminal device can acquire the prediction coding types of the S associated coding units. Further, if there is an associated coding unit with an intra prediction coding type in the S associated coding units, the terminal device can acquire, from the video data, the full reference frame set constructed for the to-be-coded unit, and determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit. Optionally, if the prediction coding types of the S associated coding units are all inter prediction, the terminal device can determine, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the reference frames used by the S associated coding units.

[0153] Optionally, the terminal device can acquire the prediction coding types of the S associated coding units. Further, if there is an associated coding unit with an intra prediction coding type in the S associated coding units, the terminal device can acquire, from the video data, the full reference frame set constructed for the to-be-coded unit, and determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit. Optionally, if the prediction coding types of the S associated coding units are all inter prediction, the terminal device can determine, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the reference frames used by the S associated coding units.

[0154] Optionally, the terminal device can acquire a threshold of a number of coding units associated with the video encoder. Further, if S is less than the threshold of the number of coding units, the terminal device can acquire, from the video data, a full reference frame set constructed for the to-be-coded unit, and determine the full reference frame set as the available reference frame set corresponding to the to-be-coded unit. Optionally, if S is equal to the threshold of the number of coding units, the terminal device can determine, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the reference frames used by the S associated coding units.

[0155] Optionally, if S is less than the threshold of the number of coding units, the terminal device can determine, from the video data, the available reference frame set corresponding to the to-be-coded unit according to the reference frames used by the S associated coding units.

[0156] It can be seen that the embodiment of the present application can obtain the associated coding units adjacent to the to-be-encoded unit in the target video frame, determine the available reference frame set corresponding to the to-be-encoded unit according to the prediction coding type of the associated coding units and the number of the associated coding units, thereby improving the accuracy of the available reference frame set and the accuracy of the candidate reference frame set, and then when encoding processing is performed on the to-be-encoded unit in the target video frame based on the candidate reference frame set, the candidate reference frame set can be traversed without traversing the full reference frame set, so that the encoding effect and the encoding efficiency of the target video frame can be considered at the same time. It can be understood that when the prediction coding types of the associated coding units are all inter prediction and the number of the associated coding units is equal to the coding unit number threshold, the available reference frame set corresponding to the to-be-encoded unit can be determined in the video data according to the reference frames used by the S associated coding units (that is, the full reference frame list of the to-be-encoded unit is reduced according to the reference frame use of the associated coding units), otherwise the full reference frame set obtained in the video data is determined as the available reference frame set corresponding to the to-be-encoded unit.

[0157] Further, please refer to Figure 8 , Figure 8 is a flowchart of a video data processing method provided by an embodiment of the present application. The video data processing method can include the following steps S1031-S1033, and step S1031-S1033 are Figure 3 a specific embodiment of step S103 in the embodiment corresponding to

[0158] Step S1031, obtaining the prediction coding type of the parent coding unit;

[0159] The prediction coding type can include an inter prediction type (i.e., inter prediction) and an intra prediction type (i.e., intra prediction).

[0160] Step S1032, if the prediction coding type of the parent coding unit is intra prediction, determining the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit;

[0161] It can be understood that when the terminal device determines the available reference frame set as the candidate reference frame set corresponding to the to-be-encoded unit, the terminal device can determine the forward available reference frame set as the forward candidate reference frame set corresponding to the to-be-encoded unit, and determine the backward available reference frame set as the backward candidate reference frame set corresponding to the to-be-encoded unit. The forward available reference frame set and the backward available reference frame set can be collectively referred to as the available reference frame set, and the forward candidate reference frame set and the backward candidate reference frame set can be collectively referred to as the candidate reference frame set.

[0162] In step S1033, if the prediction coding type of the parent coding unit is inter prediction, a candidate reference frame set corresponding to the to-be-coded unit is determined according to the reference frame used by the parent coding unit and the available reference frame set.

[0163] Specifically, if the prediction coding type of the parent coding unit is inter prediction, the terminal device can obtain the inter coding mode of the parent coding unit. Further, if the inter coding mode is not inter skip mode, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit. Alternatively, if the inter coding mode is inter skip mode, the terminal device can match the reference frame used by the parent coding unit with the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-coded unit.

[0164] It should be understood that the specific process of matching the reference frame used by the parent coding unit with the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-coded unit can be described as follows: the terminal device can match the reference frame used by the parent coding unit with the available reference frame set. Further, if there is an intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set, the terminal device can determine the intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit. Wherein, determining the intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit means that the sub-CU (i.e. sub coding unit) can only use the reference frame of the parent CU (i.e. parent coding unit). Alternatively, if there is no intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit.

[0165] Alternatively, the terminal device can obtain the inter coding mode of the parent coding unit. Further, if the inter coding mode is not inter skip mode, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit. Alternatively, if the inter coding mode is inter skip mode, the terminal device can match the reference frame used by the parent coding unit with the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-coded unit.

[0166] Alternatively, the terminal device can obtain the prediction coding type of the parent coding unit. Further, if the prediction coding type of the parent coding unit is intra prediction, the terminal device can determine the available reference frame set as the candidate reference frame set corresponding to the to-be-coded unit. Alternatively, if the prediction coding type of the parent coding unit is inter prediction, the terminal device can match the reference frame used by the parent coding unit with the available reference frame set to obtain the candidate reference frame set corresponding to the to-be-coded unit.

[0167] Therefore, the embodiment of the present application can obtain the parent coding unit to which the to-be-coded unit belongs in the target video frame, determine the candidate reference frame set corresponding to the to-be-coded unit according to the prediction coding type of the parent coding unit and the inter-frame coding mode of the parent coding unit, thereby improving the accuracy of the candidate reference frame set, and then when the to-be-coded unit in the target video frame is coded based on the candidate reference frame set, the candidate reference frame set can be traversed without traversing the full reference frame set, so that the coding effect and coding efficiency of the target video frame can be considered at the same time. It can be understood that, when the prediction coding type of the parent coding unit is inter-frame prediction, the inter-frame coding mode of the parent coding unit is inter-frame skip mode, and the reference frame used by the parent coding unit is available for the to-be-coded unit, the intersection between the reference frame used by the parent coding unit and the reference frame in the available reference frame set is determined as the candidate reference frame set corresponding to the to-be-coded unit (that is, the to-be-coded unit reuses the reference frame of the parent coding unit in a specific case, that is, the reference frame of the sub-coding unit is limited by the reference frame of the parent coding unit), otherwise the available reference frame set corresponding to the to-be-coded unit is determined as the candidate reference frame set corresponding to the to-be-coded unit.

[0168] Further, please refer to Figure 9 , Figure 9 is a structural schematic diagram of a video data processing device provided by the embodiment of the present application. The video data processing device 1 can include: an encoding unit obtaining module 11, an available set determining module 12, and a candidate set determining module 13. Further, the data processing device 1 can further include: a set delivery module 14.

[0169] The encoding unit obtaining module 11 is configured to obtain a target video frame from video data, and obtain an associated coding unit associated with a to-be-coded unit in the target video frame. The coding order of the associated coding unit is earlier than the coding order of the to-be-coded unit, and the associated coding unit is adjacent to the to-be-coded unit.

[0170] The available set determining module 12 is configured to determine an available reference frame set corresponding to the to-be-coded unit according to the associated coding unit in the video data.

[0171] The number of the associated coding units is S, and S is a positive integer.

[0172] The available set determining module 12 includes: a first type obtaining unit 121, a first determining unit 122, a second determining unit 123, a full set determining unit 124, a reference frame determining unit 125, and a reference frame merging unit 126.

[0173] The first type obtaining unit 121 is configured to obtain the prediction coding type of the S associated coding units.

[0174] The first determination unit 122 is configured to, if there is an associated coding unit of the S associated coding units whose prediction coding type is intra prediction, acquire a full-reference frame set constructed for the to-be-coded unit in the video data, and determine the full-reference frame set as the available reference frame set corresponding to the to-be-coded unit.

[0175] The full-reference frame set includes a forward full-reference frame set and a backward full-reference frame set.

[0176] The first determination unit 122 is specifically configured to acquire, in the video data, a coded video frame earlier than the target video frame in coding order.

[0177] The first determination unit 122 is specifically configured to, if the coded video frame is earlier than the target video frame in playing order, add the coded video frame to the forward full-reference frame set.

[0178] The first determination unit 122 is specifically configured to, if the coded video frame is later than the target video frame in playing order, add the coded video frame to the backward full-reference frame set.

[0179] The second determination unit 123 is configured to, if the prediction coding types of the S associated coding units are all inter prediction, determine, according to the S associated coding units, the available reference frame set corresponding to the to-be-coded unit in the video data.

[0180] The second determination unit 123 is specifically configured to acquire a coding unit quantity threshold value associated with the video encoder; and the coding unit quantity threshold value is greater than or equal to S.

[0181] The second determination unit 123 is specifically configured to, if S is less than the coding unit quantity threshold value, acquire a full-reference frame set constructed for the to-be-coded unit in the video data, and determine the full-reference frame set as the available reference frame set corresponding to the to-be-coded unit.

[0182] The second determination unit 123 is specifically configured to, if S is equal to the coding unit quantity threshold value, determine, according to the reference frames used by the S associated coding units, the available reference frame set corresponding to the to-be-coded unit in the video data.

[0183] The full-set determination unit 124 is configured to, if the prediction coding types of the S associated coding units are all inter prediction and S is equal to a coding unit quantity threshold value associated with the video encoder, acquire a full-reference frame set constructed for the to-be-coded unit in the video data; and the full-reference frame set includes a forward full-reference frame set and a backward full-reference frame set.

[0184] The reference frame determination unit 125 is configured to acquire a forward reference frame closest to the position of the target video frame from the set of forward full reference frames, and acquire a backward reference frame closest to the position of the target video frame from the set of backward full reference frames.

[0185] The reference frame merging unit 126 is configured to merge the reference frames used by the S associated coding units, the forward reference frame and the backward reference frame, to obtain a set of available reference frames corresponding to the to-be-coded unit.

[0186] The set of available reference frames includes a set of forward available reference frames and a set of backward available reference frames.

[0187] The reference frame merging unit 126 is specifically configured to determine the union of the reference frames used by the S associated coding units as the set of associated reference frames.

[0188] The reference frame merging unit 126 is specifically configured to, if the set of associated reference frames does not include the forward reference frame and the backward reference frame, determine the reference frame earlier than the target video frame in the playback order and the forward reference frame in the set of associated reference frames as the set of forward available reference frames corresponding to the to-be-coded unit.

[0189] The reference frame merging unit 126 is specifically configured to determine the reference frame later than the target video frame in the playback order and the backward reference frame in the set of associated reference frames as the set of backward available reference frames corresponding to the to-be-coded unit.

[0190] The specific implementation of the first type acquisition unit 121, the first determination unit 122, the second determination unit 123, the full set determination unit 124, the reference frame determination unit 125 and the reference frame merging unit 126 can refer to the descriptions of the corresponding embodiments of steps S102 and Figure 3 The descriptions of steps S1021-S1023 in the corresponding embodiments will not be repeated here. Figure 7 The descriptions of steps S1021-S1023 in the corresponding embodiments will not be repeated here.

[0191] The candidate set determination module 13 is configured to, if the parent coding unit to which the to-be-coded unit belongs exists in the target video frame, determine a set of candidate reference frames corresponding to the to-be-coded unit according to the parent coding unit and the set of available reference frames; the set of candidate reference frames is used for traversing target reference frames for the to-be-coded unit; and the target reference frames are used for encoding processing of the to-be-coded unit.

[0192] The candidate set determination module 13 includes a second type acquisition unit 131, a third determination unit 132, a fourth determination unit 133, a reference frame matching unit 134, a fifth determination unit 135 and a sixth determination unit 136.

[0193] The second type acquisition unit 131 is configured to acquire a prediction coding type of the parent coding unit.

[0194] The third determination unit 132 is configured to determine the set of available reference frames as the set of candidate reference frames corresponding to the to-be-encoded unit if the prediction encoding type of the parent coding unit is intra prediction.

[0195] The fourth determination unit 133 is configured to determine the set of candidate reference frames corresponding to the to-be-encoded unit according to the reference frame used by the parent coding unit and the set of available reference frames if the prediction encoding type of the parent coding unit is inter prediction.

[0196] The fourth determination unit 133 is specifically configured to acquire the inter coding mode of the parent coding unit.

[0197] The fourth determination unit 133 is specifically configured to determine the set of available reference frames as the set of candidate reference frames corresponding to the to-be-encoded unit if the inter coding mode is not inter skip mode.

[0198] The fourth determination unit 133 is specifically configured to match the reference frame used by the parent coding unit with the set of available reference frames to obtain the set of candidate reference frames corresponding to the to-be-encoded unit if the inter coding mode is inter skip mode.

[0199] The reference frame matching unit 134 is configured to match the reference frame used by the parent coding unit with the set of available reference frames if the prediction encoding type of the parent coding unit is inter prediction and the inter coding mode of the parent coding unit is inter skip mode.

[0200] The fifth determination unit 135 is configured to determine the intersection between the reference frame used by the parent coding unit and the reference frames in the set of available reference frames as the set of candidate reference frames corresponding to the to-be-encoded unit if there is an intersection between the reference frame used by the parent coding unit and the reference frames in the set of available reference frames.

[0201] The set of available reference frames includes a set of forward available reference frames and a set of backward available reference frames; and the set of candidate reference frames includes a set of forward candidate reference frames and a set of backward candidate reference frames.

[0202] The fifth determination unit 135 is specifically configured to determine the reference frame used by the parent coding unit and the set of forward available reference frames, which have an earlier playing order than the target video frame, as the set of forward candidate reference frames corresponding to the to-be-encoded unit.

[0203] The fifth determination unit 135 is specifically configured to determine the reference frame used by the parent coding unit and the set of backward available reference frames, which have a later playing order than the target video frame, as the set of backward candidate reference frames corresponding to the to-be-encoded unit.

[0204] The sixth determining unit 136 is configured to determine the set of available reference frames as the candidate reference frame set corresponding to the to-be-encoded unit if there is no intersection between the reference frames used by the parent coding unit and the reference frames in the set of available reference frames.

[0205] The specific implementation of the second type obtaining unit 131, the third determining unit 132, the fourth determining unit 133, the reference frame matching unit 134, the fifth determining unit 135, and the sixth determining unit 136 can refer to the descriptions of the corresponding embodiments of the above Figure 3 The descriptions of steps S103 and S1031-S1033 in the corresponding embodiments will not be repeated here. Figure 8 The descriptions of steps S1031-S1033 in the corresponding embodiments will not be repeated here.

[0206] Optionally, the set delivering module 14 is configured to determine the set of available reference frames as the candidate reference frame set corresponding to the to-be-encoded unit if the parent coding unit to which the to-be-encoded unit belongs does not exist in the target video frame.

[0207] The specific implementation of the coding unit obtaining module 11, the available set determining module 12, the candidate set determining module 13, and the set delivering module 14 can refer to the descriptions of the corresponding embodiments of the above Figure 3 The descriptions of steps S101-S103, S1021-S1023, and S1031-S1033 in the corresponding embodiments will not be repeated here. Figure 7 The descriptions of steps S1021-S1023 and S1031-S1033 in the corresponding embodiments will not be repeated here. Figure 8 The descriptions of steps S1031-S1033 in the corresponding embodiments will not be repeated here. In addition, the descriptions of the beneficial effects of using the same method will not be repeated here.

[0208] Further, refer to Figure 10 , Figure 10 is a structural schematic diagram of a computer device provided in the embodiments of the present application. The computer device can be a terminal device or a server. As shown in Figure 10As shown, the computer device 1000 can include a processor 1001, a network interface 1004 and a memory 1005, and further include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is configured to realize the connection and communication between the components. In some embodiments, the user interface 1003 can include a display, a keyboard, and optionally a standard wired interface, a wireless interface. Optionally, the network interface 1004 can include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a non-volatile memory such as at least one disk storage. Optionally, the memory 1005 can be at least one storage device located away from the aforementioned processor 1001. As shown Figure 10 The memory 1005 as a computer readable storage medium can include an operating system, a network communication module, a user interface module and a device control application.

[0209] In the computer device 1000 as shown Figure 10 , the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to realize:

[0210] obtaining a target video frame from the video data, and obtaining an associated coding unit associated with a to-be-coded unit in the target video frame; the coding order of the associated coding unit is earlier than the coding order of the to-be-coded unit, and the associated coding unit is adjacent to the to-be-coded unit;

[0211] determining a set of available reference frames corresponding to the to-be-coded unit according to the associated coding unit in the video data;

[0212] if there is a parent coding unit to which the to-be-coded unit belongs in the target video frame, determining a set of candidate reference frames corresponding to the to-be-coded unit according to the parent coding unit and the set of available reference frames; the set of candidate reference frames is used to traverse target reference frames for the to-be-coded unit; and the target reference frames are used for encoding processing of the to-be-coded unit.

[0213] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the description of the video data processing method in the foregoing Figure 3 、 Figure 7 or Figure 8 corresponding embodiments, and can also execute the description of the video data processing method in the foregoing Figure 9The description of the video data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of using the same method will also not be repeated.

[0214] In addition, it should be noted that the embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium stores the computer program executed by the video data processing device 1 mentioned above. When the processor executes the computer program, the computer device can execute the description of the video data processing method in the corresponding embodiment of Figure 3 、 Figure 7 or Figure 8 The description of the video data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer readable storage medium embodiments involved in the present application, please refer to the description of the method embodiments of the present application.

[0215] In addition, it should be noted that the embodiments of the present application also provide a computer program product, which can include a computer program that can be stored in a computer readable storage medium. The processor of the computer device reads the computer program from the computer readable storage medium, and the processor can execute the computer program to make the computer device execute the description of the video data processing method in the corresponding embodiment of Figure 3 、 Figure 7 or Figure 8 The description of the video data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program product embodiments involved in the present application, please refer to the description of the method embodiments of the present application.

[0216] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0217] The above disclosure is only the preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application, so the equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video data processing method, characterized in that, include: Obtain a target video frame from the video data, and obtain associated coding units that are related to the unit to be encoded in the target video frame; the coding order of the associated coding units is earlier than the coding order of the unit to be encoded, and the associated coding units are adjacent to the unit to be encoded; the number of associated coding units is S, where S is a positive integer; The available reference frame set corresponding to the unit to be encoded is determined in the video data based on the associated coding units. If there is an associated coding unit with intra-frame prediction among the S associated coding units, or if S is less than the threshold for the number of coding units associated with the video encoder, then the available reference frame set is the full reference frame set constructed for the unit to be encoded. If the prediction coding type of the S associated coding units is inter-frame prediction, and S is equal to the threshold for the number of coding units, then the available reference frame set includes the reference frame in the full reference frame set that is closest to the position of the target video frame. If a parent coding unit to which the unit to be encoded belongs exists in the target video frame, then a candidate reference frame set corresponding to the unit to be encoded is determined based on the parent coding unit and the available reference frame set; the candidate reference frame set is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

2. The method according to claim 1, characterized in that, The step of determining the set of available reference frames corresponding to the unit to be encoded in the video data based on the associated coding unit includes: Obtain the predicted coding type of the S associated coding units; If there is an associated coding unit with intra-frame prediction among the S associated coding units, then obtain the full reference frame set constructed for the unit to be coded in the video data, and determine the full reference frame set as the available reference frame set corresponding to the unit to be coded. If the prediction coding type of all S associated coding units is inter-frame prediction, then based on the S associated coding units, the set of available reference frames corresponding to the unit to be coded is determined in the video data.

3. The method according to claim 2, characterized in that, The full reference frame set includes a forward full reference frame set and a backward full reference frame set; The step of obtaining the full set of reference frames constructed for the unit to be encoded from the video data includes: Obtain encoded video frames from the video data whose encoding order is earlier than that of the target video frame; If the playback order of the encoded video frame is earlier than that of the target video frame, then the encoded video frame is added to the forward full reference frame set. If the playback order of the encoded video frame is later than that of the target video frame, then the encoded video frame is added to the backward full reference frame set.

4. The method according to claim 2, characterized in that, The step of determining the set of available reference frames corresponding to the unit to be encoded in the video data based on the S associated coding units includes: Obtain a threshold for the number of coding units associated with the video encoder; the threshold for the number of coding units is greater than or equal to S; If S is less than the threshold for the number of coding units, then a full set of reference frames constructed for the coding unit is obtained from the video data, and the full set of reference frames is determined as the available set of reference frames corresponding to the coding unit. If S is equal to the threshold number of coding units, then based on the reference frames used by the S associated coding units, the set of available reference frames corresponding to the unit to be encoded is determined in the video data.

5. The method according to claim 1, characterized in that, The number of the associated coding units is S, where S is a positive integer; The step of determining the set of available reference frames corresponding to the unit to be encoded in the video data based on the associated coding unit includes: If the predicted coding type of the S associated coding units is inter-frame prediction, and the S is equal to the threshold of the number of coding units associated with the video encoder, then a full reference frame set constructed for the unit to be coded is obtained from the video data; the full reference frame set includes a forward full reference frame set and a backward full reference frame set. Obtain the forward reference frame closest to the target video frame from the forward full reference frame set, and obtain the backward reference frame closest to the target video frame from the backward full reference frame set; The reference frames used by the S associated coding units, the forward reference frames, and the backward reference frames are merged to obtain the set of available reference frames corresponding to the coding unit.

6. The method according to claim 5, characterized in that, The set of available reference frames includes a forward available reference frame set and a backward available reference frame set; The step of merging the reference frames used by the S associated coding units, the forward reference frames, and the backward reference frames to obtain the available reference frame set corresponding to the coding unit includes: The union of the reference frames used by the S associated coding units is determined as the associated reference frame set; If the associated reference frame set does not include the forward reference frame and the backward reference frame, then the reference frames in the associated reference frame set whose playback order is earlier than the target video frame and the forward reference frame are determined as the forward available reference frame set corresponding to the unit to be encoded. The reference frames in the associated reference frame set whose playback order is later than the target video frame and the backward reference frames are determined as the backward available reference frame set corresponding to the unit to be encoded.

7. The method according to claim 1, characterized in that, The step of determining the candidate reference frame set corresponding to the unit to be encoded based on the parent coding unit and the available reference frame set includes: Obtain the predicted coding type of the parent coding unit; If the prediction coding type of the parent coding unit is intra-frame prediction, then the available reference frame set is determined as the candidate reference frame set corresponding to the coding unit. If the prediction coding type of the parent coding unit is inter-frame prediction, then the candidate reference frame set corresponding to the unit to be coded is determined based on the reference frame used by the parent coding unit and the available reference frame set.

8. The method according to claim 7, characterized in that, The step of determining the candidate reference frame set corresponding to the unit to be encoded based on the reference frame used by the parent coding unit and the available reference frame set includes: Obtain the inter-frame coding mode of the parent coding unit; If the inter-frame coding mode is not the inter-frame skip mode, then the available reference frame set is determined as the candidate reference frame set corresponding to the unit to be coded; If the inter-frame coding mode is the inter-frame skip mode, then the reference frame used by the parent coding unit is matched with the set of available reference frames to obtain the set of candidate reference frames corresponding to the unit to be coded.

9. The method according to claim 1, characterized in that, The step of determining the candidate reference frame set corresponding to the unit to be encoded based on the parent coding unit and the available reference frame set includes: If the prediction coding type of the parent coding unit is inter-frame prediction and the inter-frame coding mode of the parent coding unit is inter-frame skip mode, then the reference frame used by the parent coding unit is matched with the set of available reference frames. If there is an intersection between the reference frames used by the parent coding unit and the reference frames in the available reference frame set, then the intersection between the reference frames used by the parent coding unit and the reference frames in the available reference frame set is determined as the candidate reference frame set corresponding to the unit to be coded. If there is no intersection between the reference frame used by the parent coding unit and the reference frames in the available reference frame set, then the available reference frame set is determined as the candidate reference frame set corresponding to the coding unit.

10. The method according to claim 9, characterized in that, The available reference frame set includes a forward available reference frame set and a backward available reference frame set; the candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set. The step of determining the intersection between the reference frames used by the parent coding unit and the reference frames in the available reference frame set as the candidate reference frame set corresponding to the coding unit includes: The reference frames whose playback order is earlier than the target video frame in the reference frames used by the parent coding unit and the set of forward available reference frames are determined as the set of forward candidate reference frames corresponding to the coding unit. The reference frames whose playback order is later than the target video frame in the reference frames used by the parent coding unit, and the set of backward available reference frames, are determined as the set of backward candidate reference frames corresponding to the coding unit.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: If there is no parent coding unit to which the unit to be coded belongs in the target video frame, then the set of available reference frames is determined as the set of candidate reference frames corresponding to the unit to be coded.

12. A video data processing apparatus, characterized in that, include: The encoding unit acquisition module is used to acquire a target video frame from video data and acquire associated encoding units related to the unit to be encoded in the target video frame; the encoding order of the associated encoding units is earlier than the encoding order of the unit to be encoded, and the associated encoding units are adjacent to the unit to be encoded; the number of associated encoding units is S, where S is a positive integer; The available set determination module is used to determine the available reference frame set corresponding to the unit to be encoded in the video data based on the associated coding units. If there is an associated coding unit with intra-frame prediction among the S associated coding units, or if S is less than the threshold for the number of coding units associated with the video encoder, then the available reference frame set is the full reference frame set constructed for the unit to be encoded. If the predicted coding type of the S associated coding units is inter-frame prediction, and S is equal to the threshold for the number of coding units, then the available reference frame set includes the reference frame in the full reference frame set that is closest to the position of the target video frame. The candidate set determination module is used to determine the candidate reference frame set corresponding to the unit to be encoded based on the parent coding unit and the available reference frame set if there is a parent coding unit to which the unit to be encoded belongs in the target video frame; the candidate reference frame set is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

13. A computer device, characterized in that, include: Processor and memory; The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to cause the computer device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method according to any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

Images