A video data processing method and device, computer equipment and a storage medium

Abstract

Embodiments of the present application provide a video data processing method and device, computer equipment and a storage medium, which are applied to cloud technology, vehicle-mounted scene, audio and video coding and decoding scene, and the method comprises the following steps: performing recursive hierarchical division on a to-be-encoded unit in a target video frame to obtain S hierarchical division modes of the to-be-encoded unit; obtaining a best encoding mode of the to-be-encoded unit from the S hierarchical division modes, and obtaining a hierarchical sub-encoded unit in the best encoding mode; if an encoding result of the hierarchical sub-encoded unit meets a motion similarity condition, performing clipping on a full-reference frame set constructed for the to-be-encoded unit according to the hierarchical sub-encoded unit to generate a candidate reference frame set corresponding to the to-be-encoded unit under a non-division mode; the candidate reference frame set is used to traverse a target reference frame for the to-be-encoded unit; and the target reference frame is used for encoding processing of the to-be-encoded unit. By using the present application, the encoding effect and the encoding efficiency of the target video frame can be considered simultaneously.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a video data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In data transmission scenarios (e.g., live streaming), it is necessary to encode the video data to be transmitted to obtain the corresponding video bitstream, thereby improving transmission efficiency. Understandably, during the encoding process, it is necessary to obtain the target video frame's encoding unit from the video data, and then perform inter-frame prediction or intra-frame prediction on the encoding unit. In the inter-frame prediction process, the inter-frame prediction mode requires the use of a reference frame selection algorithm to determine the reference frame in the video data for encoding the unit.

[0003] Current reference frame selection algorithms can acquire video frames whose encoding order precedes the target video frame. One algorithm determines the distance between these video frames and the target video frame, as well as the encoding quality of these frames themselves. It then superimposes the distance and encoding quality, sorts the results from largest to smallest, and selects the video frame with the largest value as the target reference frame for the unit to be encoded in the target video frame. However, this algorithm only considers the distance between the target reference frames and the target video frame, and the encoding quality of the target reference frames, without considering the content similarity between them. When the image content changes drastically between the target reference frames and the target video frame, the content in the target reference frames differs significantly from that in the target video frame. Encoding the target video frame based on a target reference frame with significant content differences will noticeably degrade the encoding performance. Another reference frame selection algorithm iterates through these video frames, encoding every possible combination of reference frames to find the optimal one. However, if there are a large number of video frames whose encoding order precedes the target video frame, the reference frame selection algorithm will consume a significant amount of time in traversing these frames, thus reducing the encoding efficiency of the target video frame. Therefore, current reference frame selection algorithms cannot simultaneously achieve both encoding performance and encoding efficiency. Summary of the Invention

[0004] This application provides a video data processing method, apparatus, computer equipment, and storage medium that can simultaneously consider the encoding effect and encoding efficiency of the target video frame.

[0005] One embodiment of this application provides a video data processing method, including:

[0006] The units to be encoded in the target video frame are recursively divided into S levels, resulting in S levels of division for the units to be encoded; S is a positive integer; the target video frame is a video frame in the video data.

[0007] From the S hierarchical partitioning methods, obtain the optimal encoding method for the unit to be encoded, and obtain the hierarchical sub-encoding units in the optimal encoding method;

[0008] If the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, then the full set of reference frames constructed for the unit to be encoded is pruned according to the hierarchical sub-coding unit to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

[0009] One embodiment of this application provides a video data processing apparatus, including:

[0010] The partitioning module is used to recursively partition the units to be encoded in the target video frame into S hierarchical partitioning methods; S is a positive integer; the target video frame is a video frame in the video data.

[0011] The acquisition module is used to obtain the optimal encoding method for the unit to be encoded from S hierarchical partitioning methods, and to obtain the hierarchical sub-encoding units in the optimal encoding method;

[0012] The pruning module is used to prune the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding units if the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, and to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

[0013] The partitioning module includes:

[0014] A partitioning unit is used to divide the unit to be encoded in the target video frame into S partitioning methods for the unit to be encoded; the S partitioning methods include the target unit partitioning method; the target unit partitioning method includes N partitioning sub-coding units for the unit to be encoded; N is an integer greater than 1; the N partitioning sub-coding units include the target partitioning sub-coding units;

[0015] The method acquisition unit is used to obtain the final encoding method of the sub-unit corresponding to the target sub-encoding unit;

[0016] The mode determination unit is used to determine the final encoding mode of the sub-units corresponding to the N sub-encoding units as the hierarchical partitioning mode of the unit to be encoded under the target unit partitioning mode.

[0017] The method acquisition unit is specifically used to recursively divide the target sub-encoding unit into S sub-unit hierarchical division methods if the target sub-encoding unit meets the unit division conditions.

[0018] The method acquisition unit is specifically used to obtain the optimal encoding method of the target sub-encoding unit from S sub-unit hierarchical division methods, and to obtain the sub-unit hierarchical sub-encoding unit in the optimal sub-unit encoding method.

[0019] The acquisition unit is specifically used to, if the encoding result of the sub-unit at the sub-unit level satisfies the motion similarity condition, then, based on the sub-unit level sub-coding unit, prune the full set of sub-unit reference frames constructed for the target sub-coding unit to generate a set of candidate sub-unit reference frames corresponding to the target sub-coding unit in the no-division mode; the set of candidate sub-unit reference frames is used to traverse the target sub-unit target reference frames for the target sub-coding unit; the target sub-unit target reference frames are used to encode the target sub-coding unit;

[0020] The method acquisition unit is specifically used to obtain the final encoding method of the subunit corresponding to the target sub-encoding unit from the optimal encoding method and the no-division method of the subunit.

[0021] Among them, the method acquisition unit is specifically used to obtain the sub-unit size of the target sub-coding unit;

[0022] The method for obtaining units is specifically used to determine that the target sub-encoding unit satisfies the unit partitioning condition if the size of the sub-unit is greater than or equal to the size threshold.

[0023] The method for obtaining units is specifically used to determine that if the size of a sub-unit is smaller than a size threshold, the target sub-encoding unit does not meet the unit partitioning conditions.

[0024] The method acquisition unit is specifically used to determine the no-division method as the final encoding method of the sub-unit corresponding to the target sub-division encoding unit if the target sub-encoding unit does not meet the unit division conditions.

[0025] The optimal encoding method includes M sub-encoding units of the unit to be encoded; M is an integer greater than 1; the M sub-encoding units include auxiliary sub-encoding units;

[0026] The acquisition module includes:

[0027] The first determining unit is used to determine the auxiliary sub-coding unit as the hierarchical sub-coding unit in the optimal coding scheme if the auxiliary sub-coding unit does not have a sub-coding unit.

[0028] The second determining unit is used to obtain the hierarchical sub-coding unit in the optimal coding scheme from the auxiliary sub-coding unit if the auxiliary sub-coding unit has sub-coding units.

[0029] The candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set; the full reference frame set includes a forward full reference frame set and a backward full reference frame set.

[0030] The cropping module includes:

[0031] The set acquisition unit is used to acquire the forward full reference frame set and the backward full reference frame set constructed for the unit to be encoded from the video data;

[0032] The first filtering unit is used to filter the reference frames used by the hierarchical sub-coding units in the forward full reference frame set. If there are reference frames used by the hierarchical sub-coding units in the forward full reference frame set, the reference frames filtered in the forward full reference frame set will be determined as the forward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode.

[0033] The second filtering unit is used to filter the reference frames used by the hierarchical sub-coding units in the backward full reference frame set. If there are reference frames used by the hierarchical sub-coding units in the backward full reference frame set, the reference frames filtered in the backward full reference frame set will be determined as the backward candidate reference frame set corresponding to the unit to be encoded in the non-partition mode.

[0034] The set acquisition unit is specifically used to acquire encoded video frames whose encoding order is earlier than the target video frame from the video data;

[0035] The set acquisition unit is specifically used to add the encoded video frames whose playback order is earlier than the target video frame to the forward full reference frame set constructed for the unit to be encoded if the playback order of the encoded video frames is earlier than the target video frame.

[0036] The set acquisition unit is specifically used to add the encoded video frames whose playback order is later than the target video frame to the backward full reference frame set constructed for the unit to be encoded if the playback order of the encoded video frames is later than the target video frame.

[0037] The number of hierarchical sub-coding units is P; P is an integer greater than 1; the P hierarchical sub-coding units include the target hierarchical sub-coding unit;

[0038] The device also includes:

[0039] The condition judgment module is used to obtain the inter-frame prediction mode and inter-frame prediction direction corresponding to the target level sub-coding unit;

[0040] The condition judgment module is used to determine that the encoding results of the P level sub-coding units satisfy the motion similarity condition if the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame predictions and the inter-frame prediction directions corresponding to the P level sub-coding units are all the same.

[0041] The condition judgment module is used to determine that the coding results of the P level sub-coding units do not meet the motion similarity condition if there is a level sub-coding unit among the P level sub-coding units whose inter-frame prediction mode is not translation inter-frame prediction, or if the inter-frame prediction directions corresponding to the P level sub-coding units are different.

[0042] The condition judgment module is specifically used to obtain the inter-frame prediction direction corresponding to the target level sub-coding unit; the inter-frame prediction direction corresponding to the target level sub-coding unit includes forward prediction, backward prediction and bidirectional prediction.

[0043] The condition judgment module is specifically used to obtain the motion vector corresponding to each pixel in the target-level sub-coding unit;

[0044] The condition judgment module is specifically used to determine the translation inter-frame prediction as the inter-frame prediction mode corresponding to the target level sub-coding unit if the motion vectors corresponding to each pixel in the target level sub-coding unit are the same.

[0045] The condition judgment module is specifically used to determine the non-translation inter-frame prediction as the inter-frame prediction mode corresponding to the target level sub-coding unit if there are pixels with different motion vectors in the target level sub-coding unit.

[0046] The device also includes:

[0047] The determination module is used to obtain the full set of reference frames constructed for the unit to be encoded if the encoding result of the hierarchical sub-coding unit does not meet the motion similarity condition, and to determine the full set of reference frames as the candidate reference frame set corresponding to the unit to be encoded in the non-partition mode.

[0048] The device also includes:

[0049] The parameter comparison module is used to obtain the first rate-distortion parameter of the optimal encoding method and the second rate-distortion parameter of the non-division method;

[0050] The parameter comparison module is used to determine the no-division mode as the final encoding mode corresponding to the unit to be encoded if the first rate-distortion parameter is greater than or equal to the second rate-distortion parameter.

[0051] The parameter comparison module is used to determine the optimal encoding method as the final encoding method corresponding to the unit to be encoded if the first rate-distortion parameter is less than the second rate-distortion parameter.

[0052] One embodiment of this application provides a computer device, including: a processor and a memory;

[0053] The processor is connected to a memory, which stores a computer program. When the computer program is executed by the processor, it causes the computer device to perform the method provided in the embodiments of this application.

[0054] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0055] One embodiment of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0056] Therefore, this application proposes a fast reference frame selection algorithm that fully considers the high similarity between the reference frames of an image block (i.e., the unit to be encoded) and the reference frames of its sub-blocks (i.e., hierarchical sub-coding units). Different partitions of the same image block will result in multiple reference frame selection processes. If different sub-blocks (i.e., hierarchical sub-coding units) within the image block have consistent motion trajectories (i.e., the encoding results of the hierarchical sub-coding units satisfy the motion similarity condition), then the image content covered by the image block is likely to be translated as a whole. Therefore, the reference frames of the image block and the sub-blocks (i.e., hierarchical sub-coding units) are likely to be the same. In this case, by cropping the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding units, a candidate reference frame set corresponding to the unit to be encoded without partitioning can be generated (i.e., the reference frame selection results of the hierarchical sub-coding units generated by the partitioning of the unit to be encoded are used to quickly select the reference frames of the unit to be encoded). It is understood that the fast reference frame selection algorithm provided in this application embodiment can select a set of candidate reference frames that incorporate reference frames used by hierarchical sub-coding units from all video frames. Because the reference frames in the candidate reference frame set are determined by the hierarchical sub-coding units, the reference frames in the candidate reference frame set have a high content similarity with the target video frame. In this way, this application embodiment can avoid traversing all encoded video frames (i.e., all video frames in the full reference frame set), but instead traverse the video frames in the candidate reference frame set with fewer frames. This not only reduces the traversal time, but also allows the acquisition of the target reference frame with the best encoding effect from the traversal results when traversing the candidate reference frame set to which the reference frame with high content similarity belongs. Thus, it can simultaneously consider the encoding effect and encoding efficiency of the target video frame (i.e., improve the encoding effect of the target video frame while ensuring the encoding efficiency of the target video frame; improve the encoding efficiency of the target video frame while ensuring the encoding effect of the target video frame). Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of a data interaction scenario provided in an embodiment of this application;

[0060] Figure 3 This is a flowchart illustrating a video data processing method provided in an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of a scenario for dividing units to be encoded, provided in an embodiment of this application;

[0062] Figure 5 This is a schematic diagram of a scenario for obtaining a reference frame provided in an embodiment of this application;

[0063] Figure 6 This is a flowchart illustrating a non-coding approach provided in an embodiment of this application;

[0064] Figure 7 This is a flowchart illustrating a video data processing method provided in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram illustrating a scenario of a partitioning method provided in an embodiment of this application;

[0066] Figure 9 This is a flowchart illustrating a video data processing method provided in an embodiment of this application;

[0067] Figure 10 This is a flowchart illustrating an embodiment of the present application for encoding a unit to be encoded;

[0068] Figure 11 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application;

[0069] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] For details, please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 1 As shown, this network architecture may include a server 2000 and a terminal device cluster. The terminal device cluster may specifically include one or more terminal devices; the number of terminal devices in the cluster is not limited here. Figure 1As shown, the multiple terminal devices may specifically include terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n; terminal device 3000a, terminal device 3000b, terminal device 3000c, ..., terminal device 3000n can be directly or indirectly connected to server 2000 via wired or wireless communication, so that each terminal device can interact with server 2000 through the network connection.

[0072] Each terminal device in the terminal device cluster can include: smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, in-vehicle terminals, aircraft, and other intelligent terminals with data processing capabilities. It should be understood that, as... Figure 1 Each terminal device in the terminal device cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the server 2000. This application client can include social clients, multimedia clients (e.g., video clients), entertainment clients (e.g., game clients), educational clients, live streaming clients, and other application clients with video encoding capabilities. This application client can be a standalone client or an embedded sub-client integrated into another client; no limitation is made here.

[0073] Among them, server 2000 can be the server corresponding to the application client. Server 2000 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0074] For ease of understanding, the embodiments of this application may be described in detail below. Figure 1 From the multiple terminal devices shown, one terminal device is selected as the target terminal device. For example, in the embodiments of this application, a terminal device can be selected as the target terminal device. Figure 1 The terminal device 3000c shown serves as the target terminal device, which can integrate an application client with video encoding capabilities. In this case, the target terminal device can interact with the server 2000 through the application client.

[0075] It should be understood that the video data processing method provided in this application embodiment can be executed by a computer device with video encoding capabilities. This computer device can utilize cloud technology to encode and transmit multimedia data (e.g., video data). Specifically, the video data processing method provided in this application embodiment can be executed by server 2000 (i.e., the aforementioned computer device can be server 2000), by a target terminal device (i.e., the aforementioned computer device can be the target terminal device), or by both server 2000 and the target terminal device. In other words, server 2000 can encode video data using the video data processing method provided in this application embodiment, and then send the encoded video stream to the target terminal device, which can then decode and play the video stream. Alternatively, the target terminal device can also encode video data using the video data processing method provided in this application embodiment, and then send the encoded video stream to server 2000. Optionally, the target terminal device can also send the encoded video stream to other terminal devices in the terminal device cluster (e.g., terminal device 3000a).

[0076] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to cloud computing business models. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support to be realized through cloud computing.

[0077] It is understandable that the aforementioned network framework can be applied to scenarios such as video calls, video transmission, cloud conferencing, live streaming, and cloud gaming. Specific business scenarios will not be listed here. Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. Cloud gaming technology enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server.

[0078] Cloud conferencing, based on cloud computing technology, is an efficient, convenient, and low-cost form of conferencing. Users only need an internet interface for simple and easy-to-use operations to quickly and efficiently share voice, data files, and video with teams and clients worldwide. The complex technologies involved in data transmission and processing during the meeting are handled by the cloud conferencing service provider. Currently, domestic cloud conferencing mainly focuses on SaaS (Software as a Service) models, including telephone, internet, and video services. Video conferencing based on cloud computing is called cloud conferencing. In the era of cloud conferencing, data transmission, processing, and storage are all handled by the video conferencing provider's computer resources. Users no longer need to purchase expensive hardware or install cumbersome software; they only need to open a browser and log in to the corresponding interface to conduct efficient remote meetings. Cloud conferencing systems support dynamic multi-server cluster deployment and provide multiple high-performance servers, greatly improving meeting stability, security, and availability. In recent years, video conferencing has gained popularity due to its ability to significantly improve communication efficiency, continuously reduce communication costs, and upgrade internal management, and is widely used in various fields such as transportation, finance, telecommunications, education, enterprises, and the Internet of Vehicles. Undoubtedly, with the application of cloud computing, video conferencing has become more attractive in terms of convenience, speed, and ease of use, which will surely stimulate a new wave of video conferencing applications.

[0079] It should be understood that computer equipment with video encoding capabilities (e.g., the target terminal device) can encode video data using a video encoder to obtain the corresponding video bitstream, thereby improving the transmission efficiency of the video data. For example, a video encoder can be an HEVC (High Efficiency Video Coding) video encoder, a VVC (Versatile Video Coding) video encoder, etc. The VVC video encoder is also known as an H.266 video encoder; the universal video coding standard specifies the decoding process and syntax for H.266 video encoders, as well as the encoding process and syntax for H.266 video encoders. The HEVC video encoder is also known as an H.265 video encoder.

[0080] The H.266 video encoder, as an encoding standard, achieves a bitrate approximately 50% of its predecessor, HEVC, while maintaining the same subjective quality. This is extremely beneficial for handling massive amounts of video data, as the same quality video stream requires less storage space and bandwidth. However, the encoding complexity of the H.266 video encoder is correspondingly several times higher. This is because the new standard introduces more complex encoding tools to achieve higher video compression ratios. High encoding complexity means more computational resources and longer encoding time. For low-latency services like live streaming, high encoding complexity directly degrades the user experience. Therefore, it is crucial to find a way to preserve the rate-distortion performance of the video encoder while minimizing its encoding complexity.

[0081] For ease of understanding, in this embodiment of the application, the video frame to be encoded in the video data can be referred to as the target video frame, and the basic coding unit to be encoded in the target video frame can be referred to as the coding unit. The coding unit (CU) can be a basic coding unit in an H.266 video encoder / H.265 video encoder.

[0082] It is understood that target video frames can have different video frame types (i.e., frame types). Different target video frame types result in different reference frames being selected when encoding the units to be encoded within those target video frames. The target video frame types can include a first type, a second type, and a third type. Specifically, in this application embodiment, the intra-picture (I-frame) frame type can be referred to as the first type, the bi-directional interpolated prediction frame (B-frame) frame type as the second type, and the predictive-frame (P-frame) frame type as the third type.

[0083] It is understood that the video data in the embodiments of this application can be any video data that needs to be encoded in the business scenario. For example, the video data can be directly captured by the image acquisition device (e.g., camera) in the terminal device, the video data can be recorded in real time by the image acquisition device in the terminal device during live streaming / video calls, the video data can be downloaded by the terminal device from the network, or the video data can be obtained by the terminal device from the server during gaming / conferences.

[0084] For better understanding, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram illustrating a data interaction scenario provided in an embodiment of this application. For example... Figure 2 The server 20a shown can be the one described above. Figure 1 The server 2000 in the corresponding embodiment, such as Figure 2 The terminal device 20b shown can be the one described above. Figure 1 The target terminal device in the corresponding embodiment. For ease of understanding, this application embodiment uses terminal device 20b as the sending end for sending video data and server 20a as the receiving end for receiving video data as an example for explanation.

[0085] It should be understood that terminal device 20b can acquire video data (e.g., video data 21a). Video data 21a may include one or more video frames; this embodiment does not limit the number of video frames in video data 21a. Furthermore, terminal device 20b needs to encode video data 21a using a video encoder (e.g., an H.266 video encoder) to generate a video stream associated with video data 21a.

[0086] like Figure 2As shown, when the terminal device 20b encodes the video data 21a, it can obtain the target video frame (e.g., video frame 21b) (i.e., video frame 21b is a video frame in the video data 21a) that needs to be encoded from the video data 21a, and then obtain the unit to be encoded (e.g., unit to be encoded 21c) from the video frame 21b. Further, the terminal device 20b can encode the unit to be encoded 21c based on the encoding strategy of the video encoder to obtain the compressed bitstream corresponding to the unit to be encoded 21c. It should be understood that when the terminal device 20b completes the encoding processing of each unit to be encoded in the target video frame, it can obtain the compressed bitstream corresponding to each unit to be encoded separately. Therefore, when completing the encoding processing of each video frame in the video data 21a, it can encapsulate the compressed bitstream corresponding to each unit to be encoded into a video bitstream associated with the video data 21a, thereby completing the encoding processing of the video data 21a.

[0087] The encoding strategy of a video encoder can include intra-frame prediction mode (i.e., intra-frame predictive coding) and inter-frame prediction mode (i.e., inter-frame predictive coding). Intra-frame prediction (i.e., intra-frame coding) means that the encoding of the current frame does not reference information from other frames, while inter-frame prediction (i.e., inter-frame coding) means that information from adjacent frames is used to predict the current frame. Both intra-frame and inter-frame prediction are types of encoding prediction techniques. When performing inter-frame prediction on the unit to be encoded in a target video frame, the video encoder may select one frame from either the forward reference frame list or the backward reference frame list as a reference frame (i.e., unidirectional prediction), or it may select one frame from each of the two reference frame lists, for a total of two frames, as reference frames (i.e., bidirectional prediction). Selecting one frame from the forward reference frame list as a reference frame is also called forward prediction, and selecting one frame from the backward reference frame list as a reference frame is also called backward prediction. Among them, the second type of video frames (i.e., B-frames) can be used for inter-frame prediction using one-way or two-way prediction, while the third type of video frames (i.e., P-frames) can be used for inter-frame prediction using one-way prediction.

[0088] It should be understood that the embodiments of this application can be applied to the selection of reference frames in inter-frame prediction modes, such as... Figure 2 As shown, terminal device 20b can recursively divide the unit to be encoded 21c into S hierarchical partitioning methods, where S can be a positive integer, and the value of S is determined by the encoding strategy of the video encoder. For example, S in an H.266 video encoder can be equal to 5. Specifically, the S hierarchical partitioning methods can include hierarchical partitioning method 22a, ..., hierarchical partitioning method 22b.

[0089] Furthermore, such as Figure 2As shown, the terminal device 20b can obtain the optimal encoding method for the unit to be encoded from the hierarchical partitioning methods 22a, ..., and hierarchical partitioning methods 22b. For example, the optimal encoding method among the hierarchical partitioning methods 22a, ..., and hierarchical partitioning methods 22b can be hierarchical partitioning method 22a (i.e., optimal encoding method 22a). The optimal encoding method 22a can be the hierarchical partitioning method with the lowest rate distortion performance among the hierarchical partitioning methods 22a, ..., and hierarchical partitioning methods 22b.

[0090] Furthermore, such as Figure 2 As shown, terminal device 20b can obtain the hierarchical sub-coding units in the optimal encoding method 22a. The number of hierarchical sub-coding units can be at least two. In this embodiment, the at least two hierarchical sub-coding units in the optimal encoding method 22a can be collectively referred to as hierarchical sub-coding units 21d.

[0091] Understandably, terminal device 20b can obtain video frames whose encoding order precedes video frame 21b from video data 21a, and determine the obtained video frames as the full reference frame set constructed for the unit to be encoded 21c. For example... Figure 2 As shown, if the encoding result of the hierarchical sub-coding unit 21d satisfies the motion similarity condition, the terminal device 20b can trim the full set of reference frames constructed for the unit to be encoded 21c based on the hierarchical sub-coding unit 21d to generate a candidate set of reference frames corresponding to the unit to be encoded 21c in the non-partition mode. Optionally, if the encoding result of the hierarchical sub-coding unit 21d does not satisfy the motion similarity condition, the terminal device 20b can determine the full set of reference frames constructed for the unit to be encoded 21c as the candidate set of reference frames corresponding to the unit to be encoded 21c in the non-partition mode.

[0092] Furthermore, such as Figure 2 As shown, after determining the candidate reference frame set, terminal device 20b can traverse the candidate reference frame set to obtain the target reference frame. Based on the target reference frame, it performs encoding processing on the unit to be encoded 21c to obtain the compressed bitstream of the unit to be encoded 21c in the non-division mode. Then, based on the compressed bitstream of the unit to be encoded 21c in the non-division mode or the compressed bitstream of the unit to be encoded 21c in the optimal encoding mode 22a, it generates a video bitstream associated with video data 21a. At this time, terminal device 20b can send the video bitstream associated with video data 21a to server 20a. In this way, server 20a can decode the video bitstream through a video decoder when it receives the video bitstream to obtain video data 21a.

[0093] It is understood that the compressed bitstream corresponding to the unit to be encoded (e.g., unit 21c) may include, but is not limited to, motion vectors, reference frame indices, and reference frame lists. Server 20a can use the information in the compressed bitstream to generate inter-frame predicted pixel values, i.e., to reconstruct the unit to be encoded. The reference frame index can represent the index for locating a specific reference frame in the reference frame list. Through the reference frame index, the specific reference frame used when encoding the unit to be encoded can be located in the reference frame list.

[0094] Therefore, the embodiments of this application can obtain the optimal encoding method corresponding to the unit to be encoded under the partitioning method (i.e., S hierarchical partitioning methods) when it is necessary to encode the unit to be encoded in the target video frame, and then obtain the hierarchical encoding sub-unit in the optimal encoding method from the target video frame, and determine the candidate reference frame set corresponding to the unit to be encoded under the non-partitioning method according to the reference frame used by the hierarchical encoding sub-unit. It is understandable that the reference frames in the candidate reference frame set are reference frames associated with the hierarchical coding subunit. Considering the correlation (i.e., content correlation) between the unit to be encoded and the video content in the hierarchical coding subunit, it is known that the reference frames in the candidate reference frame set have a high content similarity with the target video frame to which the unit to be encoded belongs. Thus, when encoding the unit to be encoded in the target video frame based on the candidate reference frame set, the candidate reference frame set can be traversed without traversing all the encoded video frames. This not only ensures the encoding effect of the target video frame, but also simplifies the selection of reference frames, effectively reducing the proportion of reference frame decision (i.e., reference frame selection) in the complexity of the entire encoding process, thereby reducing the computational complexity of the inter-frame encoding process of the video encoder, and thus reducing the overhead of encoding time (i.e., improving encoding efficiency), computing resources, and bandwidth resources.

[0095] The specific implementation method for determining the candidate reference frame set in video data using computer equipment with video encoding capabilities can be found below. Figures 3-10 The corresponding implementation examples.

[0096] Further, please see Figure 3 , Figure 3 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. The method can be executed by a server, a terminal device, or both. The server can be one of the aforementioned... Figure 2 The server 20a in the corresponding embodiment can be the terminal device described above. Figure 2 The corresponding embodiment is terminal device 20b. For ease of understanding, this application embodiment is described using the method executed by the terminal device as an example. The video data processing method may include the following steps S101-S103:

[0097] Step S101: Recursively divide the units to be encoded in the target video frame into S hierarchical division methods to obtain the units to be encoded.

[0098] Here, S can be a positive integer, and the target video frame is a video frame in the video data. In other words, the terminal device can obtain the video frame to be encoded from the video data and determine the obtained video frame as the target video frame. Further, the terminal device can perform image block partitioning (i.e., block partitioning) on ​​the target video frame through a video encoder to obtain one or more image blocks (i.e., coding blocks) of the target video frame, and then obtain the unit to be encoded from one or more image blocks. The purpose of image block partitioning is to process prediction more accurately, using smaller image blocks in parts with slight movement and larger image blocks on static backgrounds. In this embodiment, the coding unit CU can be referred to as an image block. The block partitioning process is accompanied by prediction and reference frame selection.

[0099] The specific process of recursively dividing the units to be encoded in the target video frame into S levels can be found below. Figure 7 The description of steps S1011-S1013 in the corresponding embodiment.

[0100] Step S102: Obtain the optimal encoding method for the unit to be encoded from the S hierarchical partitioning methods, and obtain the hierarchical sub-encoding units in the optimal encoding method;

[0101] Specifically, the terminal device can obtain the optimal encoding method for the unit to be encoded from S hierarchical partitioning methods. Specifically, the terminal device can obtain the rate-distortion performance corresponding to each of the S hierarchical partitioning methods, and determine the hierarchical partitioning method corresponding to the minimum rate-distortion performance among the S rate-distortion performances as the optimal encoding method for the unit to be encoded. The optimal encoding method includes M partitioned sub-coding units for the unit to be encoded, where M can be an integer greater than 1, and the M partitioned sub-coding units include auxiliary partitioned sub-coding units. Further, if an auxiliary partitioned sub-coding unit does not have sub-coding units, the terminal device can determine the auxiliary partitioned sub-coding unit as the hierarchical sub-coding unit in the optimal encoding method. Optionally, if an auxiliary partitioned sub-coding unit has sub-coding units, the terminal device can obtain the hierarchical sub-coding unit in the optimal encoding method from the auxiliary partitioned sub-coding units.

[0102] The specific process of obtaining the hierarchical sub-coding unit in the optimal encoding method from the auxiliary sub-coding unit can be found in the above description of obtaining the hierarchical sub-coding unit in the optimal encoding method from the unit to be encoded, and will not be repeated here.

[0103] For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a scenario for dividing units to be encoded, provided in an embodiment of this application. For example... Figure 4 The image block partitioning diagram 40a shown can be an image block partitioning diagram of the unit to be encoded obtained from video data. The image block partitioning diagram 40a can be a diagram obtained by image block partitioning processing through H.266 video encoder. This application embodiment does not limit the diagram obtained by image block partitioning processing. Here, the diagram obtained by image block partitioning processing is used as an example for explanation. Other image block partitioning diagrams obtained by video encoder can be found in the description of image block partitioning diagram 40a.

[0104] like Figure 4 As shown in the diagram 40a, the image block division diagram shows that the unit to be encoded can be divided into (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e), (image block 42a), (image block 43a, image block 43b, image block 43c), and (image block 44a, image block 44b, image block 44c).

[0105] Among them, (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e) can be divided into (image block 41a), (image block 41b, image block 41c), (image block 41d), (image block 41e), and (image block 41b, image block 41c) can be divided into (image block 41b) and (image block 41c). Optionally, (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e) can be divided into (image block 41a, image block 41b, image block 41c) and (image block 41d, image block 41e), (image block 41a, image block 41b, image block 41c) can be divided into (image block 41a), (image block 41b, image block 41c) and (image block 41d, image block 41e) can be divided into (image block 41d), (image block 41e) and (image block 41b, image block 41c) can be divided into (image block 41b), (image block 41c) Optionally, (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e) can be divided into (image block 41a, image block 41d) and (image block 41b, image block 41c, image block 41e), (image block 41a, image block 41d) can be divided into (image block 41a), (image block 41d), (image block 41b, image block 41c, image block 41e), and (image block 41b, image block 41c) can be divided into (image block 41b), (image block 41c). Similarly, the terminal device can divide (image block 43a, image block 43b, image block 43c) and (image block 44a, image block 44b, image block 44c), which will not be elaborated further here.

[0106] In the image block partitioning diagram 40a, all image blocks can be organized into a search tree (different image block partitioning diagrams can correspond to different search trees). The video encoder can traverse this block partitioning tree (i.e., the search tree) in a top-down recursive process to determine the final partitioning form of the current image block. In the search tree, the parent node can be the parent coding unit (i.e., the parent CU), and the child node can be the child coding unit (i.e., the child CU). Here, the terms parent coding unit and child coding unit are relative.

[0107] Optionally, the image block partitioning diagram 40a shows that the unit to be encoded can be divided into (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e, image block 42a) and (image block 43a, image block 43b, image block 43c, image block 44a, image block 44b, image block 44c). Optionally, the image block partitioning diagram 40a shows that the target video frame can be divided into (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e, image block 43a, image block 43b, image block 43c) and (image block 42a, image block 44a, image block 44b, image block 44c).

[0108] For ease of understanding, the image block partitioning diagram 40a can be the image block partitioning diagram corresponding to the optimal encoding method. In this case, the hierarchical sub-coding units in the optimal encoding method can include image blocks 41a, 41b, 41c, 41d, 41e, 42a, 43a, 43b, 43c, 44a, 44b, and 44c. The M partitioning sub-coding units of the unit to be encoded can include (image block 41a, image block 41b, image block 41c, image block 41d, image block 41e), (image block 42a), (image block 43a, image block 43b, image block 43c), and (image block 44a, image block 44b, image block 44c), that is, M equals 4.

[0109] The H.266 video encoder / H.265 video encoder encodes using a block partitioning method. During encoding, an image block is divided into several CUs. CUs can be nested, and a CU can be used as a new image block to be further divided into several CUs until the minimum CU size limit is reached. Therefore, the CU is the basic unit for encoding prediction.

[0110] Step S103: If the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, then the full set of reference frames constructed for the unit to be encoded is cropped according to the hierarchical sub-coding unit to generate the candidate reference frame set corresponding to the unit to be encoded in the non-partition mode.

[0111] Specifically, if the encoding results of the hierarchical sub-coding units satisfy the motion similarity condition, the terminal device can obtain a full set of reference frames constructed for the unit to be encoded from the video data. This full set of reference frames includes a forward full set and a backward full set; that is, the forward and backward full set of reference frames can be collectively referred to as the full reference frame set. In other words, the terminal device can obtain the forward and backward full reference frame sets constructed for the unit to be encoded from the video data. Furthermore, the terminal device filters reference frames used by hierarchical sub-coding units from the forward full reference frame set. If a reference frame used by a hierarchical sub-coding unit exists in the forward full reference frame set, the terminal device can determine the reference frame filtered from the forward full reference frame set as the forward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode. Similarly, the terminal device filters reference frames used by hierarchical sub-coding units from the backward full reference frame set. If a reference frame used by a hierarchical sub-coding unit exists in the backward full reference frame set, the terminal device can determine the reference frame filtered from the backward full reference frame set as the backward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode. The candidate reference frame set includes both the forward candidate reference frame set and the backward candidate reference frame set; both can be collectively referred to as the candidate reference frame set. The candidate reference frame set is used to traverse target reference frames for the unit to be encoded; the target reference frames are used for encoding the unit to be encoded.

[0112] In other words, the terminal device can match the reference frames used by the hierarchical sub-coding unit with the entire set of reference frames. Further, if there is an intersection between the reference frames used by the hierarchical sub-coding unit and the reference frames in the entire set of reference frames, the terminal device can determine the intersection of the reference frames used by the hierarchical sub-coding unit and the reference frames in the entire set of reference frames as the candidate reference frame set corresponding to the unit to be encoded in the no-partition mode. Optionally, if there is no intersection between the reference frames used by the hierarchical sub-coding unit and the reference frames in the entire set of reference frames, the terminal device can determine the entire set of reference frames as the candidate reference frame set corresponding to the unit to be encoded in the no-partition mode.

[0113] It should be understood that if the default reference frame list (i.e., the full reference frame set) generated by each coding unit in the same frame in the coding standard is the same, then the clipped reference frame list (i.e., the candidate reference frame set) must be a subset of the default reference frame list. Clipping means that the coding unit can use the reference frames used by the hierarchical sub-coding unit. In this case, if the coding result of the hierarchical sub-coding unit satisfies the motion similarity condition, the terminal device can determine the reference frames used by the hierarchical sub-coding unit as the candidate reference frame set corresponding to the coding unit in the non-partitioning mode. That is, video frames in the reference frames used by the hierarchical sub-coding unit that play before the target video frame are added to the forward candidate reference frame set, and video frames in the reference frames used by the hierarchical sub-coding unit that play after the target video frame are added to the backward candidate reference frame set.

[0114] It should be understood that the specific process by which the terminal device obtains the forward full reference frame set and the backward full reference frame set for the unit to be encoded from the video data can be described as follows: The terminal device can obtain encoded video frames whose encoding order is earlier than the target video frame from the video data. Further, if the playback order of the encoded video frames is earlier than the target video frame, the terminal device can add the encoded video frames whose playback order is earlier than the target video frame to the forward full reference frame set constructed for the unit to be encoded; if the playback order of the encoded video frames is later than the target video frame, the terminal device can add the encoded video frames whose playback order is later than the target video frame to the backward full reference frame set constructed for the unit to be encoded. In other words, the terminal device can add encoded video frames whose playback order is earlier than the target video frame to the forward full reference frame set, and add encoded video frames whose playback order is later than the target video frame to the forward full reference frame set.

[0115] It is understood that during inter-frame prediction, the video encoder can construct a reference frame list for the target video frame. This reference frame list comprises two parts: a forward reference frame list (i.e., the complete set of forward reference frames) and a backward reference frame list (i.e., the complete set of backward reference frames). The forward reference frame list contains video frames whose encoding order and playback order both precede the current frame (i.e., the target video frame), while the backward reference frame list contains video frames whose encoding order precedes the current frame (i.e., the target video frame) and whose playback order follows the current frame (i.e., the target video frame). This embodiment does not limit the number of video frames in the reference frame list.

[0116] The number of hierarchical sub-coding units is P, where P can be an integer greater than 1. It can be understood that the terminal device can determine the associated reference frame set as the union of the reference frames used by the P hierarchical sub-coding units. Specifically, the terminal device can define the reference frames in the associated reference frame set whose playback order is earlier than the target video frame as the forward associated reference frame set, and the reference frames in the associated reference frame set whose playback order is later than the target video frame as the backward associated reference frame set. The forward and backward associated reference frame sets can be collectively referred to as the associated reference frame set. In other words, the terminal device can add the reference frames in the P hierarchical sub-coding units whose playback order is earlier than the target video frame to the forward associated reference frame set, and add the reference frames in the P hierarchical sub-coding units whose playback order is later than the target video frame to the backward associated reference frame set. Therefore, the terminal device can determine the intersection of the forward associated reference frame set and the forward full reference frame set as the forward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode; and determine the intersection of the backward associated reference frame set and the backward full reference frame set as the backward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode.

[0117] For ease of understanding, this application embodiment uses the example of a forward-associated reference frame set including reference frames whose playback order is earlier than the target video frame, and a backward-associated reference frame set including reference frames whose playback order is later than the target video frame. Optionally, if the forward-associated reference frame set does not include reference frames (i.e., the associated reference frame set does not include reference frames whose playback order is earlier than the target video frame), the terminal device can determine that the forward candidate reference frame set corresponding to the unit to be encoded in the non-partitioning mode is an empty set, or determine the full forward reference frame set as the forward candidate reference frame set corresponding to the unit to be encoded in the non-partitioning mode; if the backward-associated reference frame set does not include reference frames (i.e., the associated reference frame set does not include reference frames whose playback order is later than the target video frame), the terminal device can determine that the backward candidate reference frame set corresponding to the unit to be encoded in the non-partitioning mode is an empty set, or determine the full backward reference frame set as the backward candidate reference frame set corresponding to the unit to be encoded in the non-partitioning mode.

[0118] For ease of understanding, this application embodiment will be described using the example of a forward full reference frame set including encoded video frames whose playback order is earlier than the target video frame, and a backward full reference frame set including encoded video frames whose playback order is later than the target video frame.

[0119] For example, taking P = 3 as an example, the P hierarchical sub-coding units can include hierarchical sub-coding unit P1, hierarchical sub-coding unit P2, and hierarchical sub-coding unit P3. Hierarchical sub-coding units P1, P2, and P3 all use bidirectional prediction. The forward reference frames and backward reference frames used by hierarchical sub-coding units P1, P2, and P3 are (x0, y0), (x0, y1), and (x1, y2), respectively. Then, the forward reference frame list (i.e., the forward candidate reference frame set) of the unit to be coded without partitioning is pruned to {x0, x1}, and the backward reference frame list (i.e., the backward candidate reference frame set) is pruned to {y0, y1, y2}.

[0120] Optionally, embodiments of this application may add a limitation on the size of the unit to be encoded (i.e., add a size limit). For example, when the number of pixels of the unit to be encoded exceeds a pixel threshold (e.g., 512), the fast strategy provided by embodiments of this application is executed. The fast strategy is steps S101-S103 in embodiments of this application.

[0121] For easier understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating a scenario for acquiring a reference frame, as provided in an embodiment of this application. Figure 5 The diagram shows a bidirectional prediction of the unit to be encoded. Video frame 53c can be the target video frame, video frame set 53a can be the forward candidate reference frame set corresponding to the target video frame 53c (i.e., forward candidate reference frame set 53a), and video frame set 53b can be the backward candidate reference frame set corresponding to the target video frame 53c (i.e., backward candidate reference frame set 53b).

[0122] The forward candidate reference frame set 53a may include multiple video frames, and the backward candidate reference frame set 53b may include multiple video frames. This embodiment does not limit the number of video frames in the forward candidate reference frame set 53a, nor does it limit the number of video frames in the backward candidate reference frame set 53b. For ease of understanding, this embodiment will be described using the example that both the forward candidate reference frame set 53a and the backward candidate reference frame set 53b include 3 video frames. The forward candidate reference frame set 53a may include video frames 50a, 50b, and 50c, and the backward candidate reference frame set 53b may include video frames 51a, 51b, and 51c.

[0123] like Figure 5As shown, the target video frame 53c may include a unit to be encoded 52a, the video frame 50c may include an encoding unit 52b, and the video frame 51b may include an encoding unit 52c. When the video frame 50c and the video frame 51b are determined as reference frames used by the target video frame, the video encoder can perform encoding processing on the unit to be encoded 52a based on the encoding units 52b and 52c. At this time, the unit to be encoded 52a selects one frame as a reference frame (e.g., video frame 50c and video frame 51b) from the forward candidate reference frame set 53a and the backward candidate reference frame set 53b respectively and performs motion search. The encoding units 52b and 52c can be referred to as reference blocks.

[0124] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of a process without code division provided in an embodiment of this application. For example... Figure 6 As shown, the terminal device can execute step S11 to obtain the unit to be encoded from the target video frame. Further, the terminal device can execute step S12 to construct a reference frame list (i.e., a full set of reference frames) for the unit to be encoded using the default method. That is, the terminal device can obtain a full set of reference frames constructed for the unit to be encoded from the video data.

[0125] Furthermore, such as Figure 6 As shown, the terminal device can execute step S13 to determine whether the unit to be encoded meets the requirements of reference frame list pruning. That is, the terminal device can determine whether the encoding result of the hierarchical sub-coding unit meets the motion similarity condition. It can be understood that if the encoding result of the hierarchical sub-coding unit meets the motion similarity condition, then the unit to be encoded meets the requirements of reference frame list pruning; optionally, if the encoding result of the hierarchical sub-coding unit does not meet the motion similarity condition, then the unit to be encoded does not meet the requirements of reference frame list pruning.

[0126] like Figure 6 As shown, if the unit to be encoded meets the requirements of reference frame list pruning, the terminal device can execute step S14 to prune the reference frame list of the unit to be encoded. That is, the terminal device can prune the full set of reference frames constructed for the unit to be encoded according to the hierarchical sub-coding unit. Optionally, if the unit to be encoded does not meet the requirements of reference frame list pruning, the terminal device can skip executing step S14.

[0127] Furthermore, such as Figure 6As shown, the terminal device can execute step S15, which involves performing inter-frame prediction on the unit to be encoded, i.e., generating a set of candidate reference frames corresponding to the unit to be encoded in the non-partitioning mode, traversing the target reference frames in the candidate reference frame set, and performing encoding processing on the unit to be encoded based on the target reference frames. Further, the terminal device can execute step S16, which indicates the end of the non-partitioning encoding of the unit to be encoded.

[0128] It should be understood that the terminal device can obtain the encoding strategy of the video encoder (e.g., an H.266 video encoder) and perform encoding processing on the unit to be encoded based on the encoding strategy. The encoding modes associated with the encoding strategy may include inter-frame prediction mode and intra-frame prediction mode. Thus, when performing inter-frame prediction processing on the unit to be encoded, the terminal device can determine the reference video frame associated with the unit to be encoded based on the frame type of the target video frame. Different video compression standards may correspond to different reference video frames. If the frame type of the target video frame is a B-frame (i.e., the second type) or a P-frame (i.e., the third type), the terminal device can execute steps S101-S103; optionally, if the frame type of the target video frame is an I-frame (i.e., the first type), the terminal device does not need to execute steps S101-S103.

[0129] Therefore, this application proposes a fast reference frame selection algorithm that fully considers the high similarity between the reference frames of an image block (i.e., the unit to be encoded) and the reference frames of its sub-blocks (i.e., hierarchical sub-coding units). Different partitions of the same image block will result in multiple reference frame selection processes. If different sub-blocks (i.e., hierarchical sub-coding units) within the image block have consistent motion trajectories (i.e., the encoding results of the hierarchical sub-coding units satisfy the motion similarity condition), then the image content covered by the image block is likely to be translated as a whole. Therefore, the reference frames of the image block and the sub-blocks (i.e., hierarchical sub-coding units) are likely to be the same. In this case, by cropping the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding units, a candidate reference frame set corresponding to the unit to be encoded without partitioning can be generated (i.e., the reference frame selection results of the hierarchical sub-coding units generated by the partitioning of the unit to be encoded are used to quickly select the reference frames of the unit to be encoded). It is understood that the fast reference frame selection algorithm provided in this application embodiment can select a set of candidate reference frames that incorporate reference frames used by hierarchical sub-coding units from all video frames. Because the reference frames in the candidate reference frame set are determined by the hierarchical sub-coding units, the reference frames in the candidate reference frame set have a high content similarity with the target video frame. In this way, this application embodiment can avoid traversing all encoded video frames (i.e., all video frames in the full reference frame set), but instead traverse the video frames in the candidate reference frame set with fewer frames. This not only reduces the traversal time, but also allows the acquisition of the target reference frame with the best encoding effect from the traversal results when traversing the candidate reference frame set to which the reference frame with high content similarity belongs. Thus, it can simultaneously consider the encoding effect and encoding efficiency of the target video frame (i.e., improve the encoding effect of the target video frame while ensuring the encoding efficiency of the target video frame; improve the encoding efficiency of the target video frame while ensuring the encoding effect of the target video frame).

[0130] Further, please see Figure 7 , Figure 7 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. The video data processing method may include the following steps S1011-S1013, and steps S1011-S1013 are... Figure 3 A specific embodiment of step S101 in the corresponding embodiment.

[0131] Step S1011: Divide the unit to be encoded in the target video frame into units to obtain S unit division methods for the unit to be encoded.

[0132] The S unit partitioning methods include the target unit partitioning method, which in turn includes N sub-coding units of the unit to be encoded. Here, N can be an integer greater than 1, and the N sub-coding units include the target sub-coding unit, which can be used as a new unit to be encoded.

[0133] For easier understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating a scenario of a partitioning method provided in an embodiment of this application. For example... Figure 8 The diagram shows the encoding unit under S-unit partitioning and no partitioning methods. For example, as shown... Figure 8 The diagrams shown represent the six division methods specified by the H.266 video encoder. Diagram 80a represents a no-division diagram, diagram 81a represents a horizontal two-division diagram, diagram 82a represents a vertical two-division diagram, diagram 83a represents a four-division diagram, diagram 84a represents a horizontal three-division diagram, and diagram 85a represents a vertical three-division diagram.

[0134] like Figure 8 As shown, schematic diagram 80a can divide the unit to be encoded into 1 image block, schematic diagrams 81a and 82a can divide the unit to be encoded into 2 image blocks, schematic diagrams 84a and 85a can divide the unit to be encoded into 3 image blocks, and schematic diagram 83a can divide the unit to be encoded into 4 image blocks.

[0135] Among them, schematic diagram 80a can be divided into image block 80b; schematic diagram 81a can be divided into image block 81b and image block 81c; schematic diagram 82a can be divided into image block 82b and image block 82c; schematic diagram 84a can be divided into image block 84b, image block 84c and image block 84d; schematic diagram 85a can be divided into image block 85b, image block 85c and image block 85d; and schematic diagram 83a can be divided into image block 83b, image block 83c, image block 83d and image block 83e.

[0136] In other words, if the target unit is divided as shown in diagram 81a, then the N sub-coding units of the target unit division method can specifically include image block 81b and image block 81c, i.e., N equals 2; if the target unit is divided as shown in diagram 82a, then the N sub-coding units of the target unit division method can specifically include image block 82b and image block 82c, i.e., N equals 2; if the target unit is divided as shown in diagram 84a, then the N sub-coding units of the target unit division method can specifically include image block 84b, image block 84c, and image block 84d, i.e., N equals 3; if the target unit is divided as shown in diagram 85a, then the N sub-coding units of the target unit division method can specifically include image block 85b, image block 85c, and image block 85d, i.e., N equals 3; if the target unit is divided as shown in diagram 83a, then the N sub-coding units of the target unit division method can specifically include image block 83b, image block 83c, image block 83d, and image block 83e, i.e., N equals 4.

[0137] It should be understood that, apart from the undivided method shown in diagram 80a, other sub-blocks (which can also be called image blocks) obtained from other divisions can continue to be divided in these six ways until the block size reaches the division limit. For example, image block 81b can continue to be divided according to diagram 82a; or image block 81b can continue to be divided according to diagram 80a (i.e., image block 81b is undivided).

[0138] Step S1012: Obtain the final encoding method of the sub-unit corresponding to the target sub-coding unit;

[0139] Specifically, if the target sub-coding unit satisfies the unit partitioning condition, the terminal device can recursively partition the target sub-coding unit to obtain S sub-unit hierarchical partitioning methods for the target sub-coding unit. Further, the terminal device can obtain the optimal sub-unit encoding method for the target sub-coding unit from the S sub-unit hierarchical partitioning methods, and obtain the sub-unit hierarchical sub-coding units from the optimal sub-unit encoding method. Further, if the sub-unit encoding results of the sub-unit hierarchical sub-coding units satisfy the motion similarity condition, the terminal device can trim the full set of sub-unit reference frames constructed for the target sub-coding unit according to the sub-unit hierarchical sub-coding units, generating a set of candidate sub-unit reference frames corresponding to the target sub-coding unit in the no-partitioning mode. The candidate sub-unit reference frame set is used to traverse the target sub-unit target reference frames for the target sub-coding unit; the target sub-unit target reference frames are used to encode the target sub-coding unit. Further, the terminal device can obtain the final sub-unit encoding method corresponding to the target sub-coding unit from the optimal sub-unit encoding method and the no-partitioning method.

[0140] The specific process of the terminal device recursively dividing the target sub-coding unit into S sub-units can be found in the description above of recursively dividing the unit to be encoded into S sub-units. It will not be repeated here.

[0141] The specific process by which the terminal device obtains the optimal encoding method for the target sub-coding unit from the S sub-unit hierarchical partitioning methods can be found in the description above regarding obtaining the optimal encoding method for the unit to be encoded from the S hierarchical partitioning methods, and will not be repeated here. Similarly, the specific process by which the terminal device obtains the sub-unit hierarchical sub-coding units within the optimal sub-unit encoding method can be found in the description above regarding obtaining the hierarchical sub-coding units within the optimal encoding method, and will not be repeated here.

[0142] The specific process of cropping the full set of reference frames for a subunit according to the subunit hierarchical sub-coding units to generate a set of candidate reference frames for a subunit can be found in the description above of cropping the full set of reference frames according to the hierarchical sub-coding units to generate a set of candidate reference frames, and will not be repeated here.

[0143] The specific process of obtaining the final encoding scheme of the sub-unit corresponding to the target sub-encoding unit from the optimal encoding scheme and the no-division scheme can be found below. Figure 9 The corresponding embodiment describes the final encoding method corresponding to the unit to be encoded obtained from the optimal encoding method and the no-division method.

[0144] It should be understood that the terminal device can obtain the sub-unit size of the target sub-coding unit. Further, if the sub-unit size is greater than or equal to a size threshold, the terminal device can determine that the target sub-coding unit satisfies the unit partitioning condition; alternatively, if the sub-unit size is less than the size threshold, the terminal device can determine that the target sub-coding unit does not satisfy the unit partitioning condition. Therefore, the unit partitioning condition refers to the condition that the obtained sub-unit size of the target sub-coding unit is greater than or equal to the size threshold. This application embodiment does not limit the specific value of the size threshold.

[0145] Optionally, if the target sub-coding unit does not meet the unit partitioning conditions, the terminal device may determine the no-partitioning method as the final encoding method of the sub-unit corresponding to the target sub-coding unit.

[0146] Step S1013: Determine the final encoding method of the sub-units corresponding to the N sub-encoding units as the hierarchical division method of the unit to be encoded under the target unit division method.

[0147] It should be understood that S unit partitioning methods can recursively generate S hierarchical partitioning methods, and one unit partitioning method can recursively generate one hierarchical partitioning method. The specific process by which the terminal device determines the corresponding hierarchical partitioning method for the unit to be encoded under other unit partitioning methods besides the target unit partitioning method can be found in the description of determining the corresponding hierarchical partitioning method for the unit to be encoded under the target unit partitioning method; it will not be repeated here.

[0148] It should be understood that the hierarchical partitioning method corresponding to the unit to be encoded under the target unit partitioning method can be as described above. Figure 3 The optimal encoding method in the corresponding embodiment may not necessarily be the same as described above, as the hierarchical division method corresponding to the unit to be encoded under the target unit division method. Figure 3 The optimal encoding method in the corresponding embodiment. For better understanding, please refer to [link to previous document]. Figure 4 If the image block partitioning diagram 40a is the image block partitioning diagram corresponding to the optimal encoding method, and the hierarchical partitioning method corresponding to the unit to be encoded under the target unit partitioning method is as described above... Figure 3 The optimal encoding method in the corresponding embodiment is shown in the image block partitioning diagram 40a, which is an image block partitioning diagram of the hierarchical partitioning method corresponding to the unit to be encoded under the target unit partitioning method.

[0149] Therefore, the embodiments of this application can divide the unit to be encoded in the target video frame into units, obtaining S unit division methods for the unit to be encoded. Then, the hierarchical division method corresponding to the unit to be encoded under the S unit division methods is determined recursively. The S hierarchical division methods represent the best encoding result of the unit to be encoded under the S unit division methods, and the best encoding method represents the best encoding result of the unit to be encoded under the S hierarchical division methods. In this way, when determining the candidate reference frame set of the unit to be encoded under the no-division method based on the best encoding method, the accuracy of the obtained candidate reference frame set can be improved.

[0150] Further, please see Figure 9 , Figure 9 This is a flowchart illustrating a video data processing method provided in an embodiment of this application. The method can be executed by a server, a terminal device, or both. The server can be one of the aforementioned... Figure 2 The server 20a in the corresponding embodiment can be the terminal device described above. Figure 2 The corresponding embodiment is terminal device 20b. For ease of understanding, this application embodiment is described using the method executed by the terminal device as an example. The video data processing method may include the following steps S201-S208:

[0151] Step S201: Recursively divide the units to be encoded in the target video frame into S hierarchical division methods to obtain the units to be encoded into S hierarchical division methods.

[0152] Here, S can be a positive integer; the target video frame is a video frame in the video data. The specific process by which the terminal device recursively divides the units to be encoded in the target video frame into S levels can be found above. Figure 7 The descriptions of steps S1011-S1013 in the corresponding embodiments will not be repeated here.

[0153] Step S202: Obtain the optimal encoding method for the unit to be encoded from the S hierarchical partitioning methods, and obtain the hierarchical sub-encoding units in the optimal encoding method;

[0154] The number of hierarchical sub-coding units is P, where P can be an integer greater than 1; the P hierarchical sub-coding units include the target hierarchical sub-coding unit. The specific process of obtaining the optimal encoding method for the unit to be encoded from the S hierarchical partitioning methods, and obtaining the hierarchical sub-coding units from the optimal encoding method, can be found above. Figure 3 The description of step S102 in the corresponding embodiments will not be repeated here.

[0155] Step S203: Obtain the inter-frame prediction mode and inter-frame prediction direction corresponding to the target level sub-coding unit;

[0156] Specifically, the terminal device can obtain the inter-frame prediction direction corresponding to the target-level sub-coding unit. This inter-frame prediction direction includes forward prediction, backward prediction, and bidirectional prediction. Further, the terminal device can obtain the motion vector corresponding to each pixel in the target-level sub-coding unit. This motion vector, also known as a motion vector, describes the offset vector of a position in a video frame relative to its position in a reference frame; it is a vector that marks the positional relationship between the current block and the reference block during inter-frame prediction. Further, if the motion vectors corresponding to each pixel in the target-level sub-coding unit are all the same, the terminal device can determine translational inter-frame prediction as the inter-frame prediction mode corresponding to the target-level sub-coding unit. Optionally, if there are pixels with different motion vectors in the target-level sub-coding unit, the terminal device can determine non-translational inter-frame prediction as the inter-frame prediction mode corresponding to the target-level sub-coding unit.

[0157] It should be understood that if the inter-frame prediction direction corresponding to the target level sub-coding unit is forward prediction, then each pixel in the target level sub-coding unit can include a forward motion vector, that is, each pixel can include one motion vector; optionally, if the inter-frame prediction direction corresponding to the target level sub-coding unit is backward prediction, then each pixel in the target level sub-coding unit can include a backward motion vector, that is, each pixel can include one motion vector; optionally, if the inter-frame prediction direction corresponding to the target level sub-coding unit is bidirectional prediction, then each pixel in the target level sub-coding unit can include both a forward motion vector and a backward motion vector, that is, each pixel can include two motion vectors.

[0158] Therefore, it can be understood that if the inter-frame prediction direction corresponding to the target level sub-coding unit is forward prediction, then the motion vector of each pixel in the target level sub-coding unit is the same in the forward direction, which means that the motion vectors corresponding to each pixel in the target level sub-coding unit are all the same. Optionally, if the inter-frame prediction direction corresponding to the target level sub-coding unit is backward prediction, then the motion vector of each pixel in the target level sub-coding unit is the same in the backward direction, which means that the motion vectors corresponding to each pixel in the target level sub-coding unit are all the same. Optionally, if the inter-frame prediction direction corresponding to the target level sub-coding unit is bidirectional prediction, then the motion vector of each pixel in the target level sub-coding unit is the same in the backward direction and the motion vector of each pixel in the target level sub-coding unit is the same in the forward direction, that is, the motion vectors of each pixel in the target level sub-coding unit are the same in both directions, which means that the motion vectors corresponding to each pixel in the target level sub-coding unit are all the same.

[0159] It is understood that the terminal device can determine whether the coding results of the P level sub-coding units satisfy the motion similarity condition based on the inter-frame prediction modes and inter-frame prediction directions corresponding to the P level sub-coding units respectively. The process of confirming that the coding results of the P level sub-coding units satisfy the motion similarity condition can be found in steps S204-S205 below; optionally, the process of confirming that the coding results of the P level sub-coding units do not satisfy the motion similarity condition can be found in steps S206-S207 below.

[0160] Step S204: If the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame prediction and the inter-frame prediction directions corresponding to the P level sub-coding units are all the same, then it is determined that the coding results of the P level sub-coding units satisfy the motion similarity condition.

[0161] For example, if the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame prediction and the inter-frame prediction directions corresponding to the P level sub-coding units are all forward prediction, then the terminal device can determine that the coding results of the P level sub-coding units satisfy the motion similarity condition.

[0162] Therefore, the motion similarity condition refers to the condition that the inter-frame prediction modes corresponding to the P sub-coding units are all translation inter-frame predictions, and the inter-frame prediction directions corresponding to the P sub-coding units are all the same.

[0163] Optionally, if the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame prediction, the terminal device can determine that the coding results of the P level sub-coding units satisfy the motion similarity condition; alternatively, if the inter-frame prediction directions corresponding to the P level sub-coding units are all the same, the terminal device can determine that the coding results of the P level sub-coding units satisfy the motion similarity condition.

[0164] Step S205: Based on the hierarchical sub-coding units, the full set of reference frames constructed for the unit to be encoded is trimmed to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-division mode.

[0165] The specific process by which the terminal device prunes the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding units to generate the candidate set of reference frames corresponding to the unit to be encoded in the non-partitioning mode can be found in the above. Figure 3 The description of step S103 in the corresponding embodiments will not be repeated here.

[0166] In other words, if the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, the terminal device can prune the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding unit to generate the candidate set of reference frames corresponding to the unit to be encoded in the non-partition mode.

[0167] Step S206: If there is a sub-coding unit among the P sub-coding units whose inter-frame prediction mode is not translation inter-frame prediction, or if the inter-frame prediction directions corresponding to the P sub-coding units are different, then it is determined that the coding results of the P sub-coding units do not meet the motion similarity condition.

[0168] Optionally, if there is a sub-coding unit among the P sub-coding units whose inter-frame prediction mode is not translation inter-frame prediction, the terminal device can determine that the coding results of the P sub-coding units do not meet the motion similarity condition; alternatively, if the inter-frame prediction directions corresponding to the P sub-coding units are different, the terminal device can determine that the coding results of the P sub-coding units do not meet the motion similarity condition.

[0169] Step S207: Obtain the full set of reference frames constructed for the unit to be encoded, and determine the full set of reference frames as the candidate set of reference frames corresponding to the unit to be encoded in the no-partition mode;

[0170] The specific process by which the terminal device obtains the full set of reference frames constructed for the unit to be encoded can be found in the above. Figure 3 The description of step S103 in the corresponding embodiments will not be repeated here.

[0171] In other words, if the encoding result of the hierarchical sub-coding unit does not meet the motion similarity condition, the terminal device can obtain the full set of reference frames constructed for the unit to be encoded, and determine the full set of reference frames as the candidate set of reference frames corresponding to the unit to be encoded in the non-partition mode.

[0172] It is understandable that the candidate reference frame set generated in steps S205 and S207 can be used to traverse the target reference frame for the unit to be encoded; the target reference frame can be used to encode the unit to be encoded and generate the compressed bitstream corresponding to the unit to be encoded.

[0173] The candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set. It should be understood that the specific process of the terminal device traversing the target reference frame in the candidate reference frame set can be described as follows: the terminal device can determine the video frame type of the target video frame. The video frame type of the target video frame can be used to guide the video encoder in selecting reference frames from the candidate reference frame set for encoding the target video frame. In this embodiment, the reference frames obtained by traversing the candidate reference frame set can be referred to as target reference frames. Further, if the video frame type is a one-way prediction type (i.e., the third type), the terminal device can traverse the target reference frames used for encoding the unit to be encoded from either the forward or backward candidate reference frame set. Optionally, if the video frame type is a two-way prediction type (i.e., the second type), the terminal device can traverse the target reference frames used for encoding the unit to be encoded from the forward, backward, or two-way reference frame set. The two-way reference frame set includes both the forward and backward candidate reference frame sets. In other words, if the video frame type is bidirectional prediction type, the terminal device can traverse the target reference frames used for encoding the unit to be encoded in the forward candidate reference frame set or the backward candidate reference frame set; or, the terminal device can traverse the target reference frames used for encoding the unit to be encoded in both the forward candidate reference frame set and the backward candidate reference frame set.

[0174] It should be understood that when a coding unit (CU) (e.g., the unit to be encoded) attempts not to be divided, the video encoder needs to select a suitable prediction mode for the CU. Prediction modes can be broadly categorized into inter-frame prediction and intra-frame prediction. Inter-frame prediction can be further divided into translational inter-frame prediction and affine inter-frame prediction, depending on the motion type. In translational inter-frame prediction, the motion vector of each pixel in the CU is the same; in affine inter-frame prediction, the motion vector of each pixel in the CU may be different. Affine inter-frame prediction is suitable for scaling and rotational motions, and non-translational inter-frame prediction can include affine inter-frame prediction.

[0175] Step S208: Obtain the final encoding method corresponding to the unit to be encoded from the optimal encoding method and the no-division method.

[0176] Specifically, the terminal device can obtain a first rate-distortion parameter for the optimal encoding method and a second rate-distortion parameter for the no-division method. Further, if the first rate-distortion parameter is greater than or equal to the second rate-distortion parameter, the terminal device can determine the no-division method as the final encoding method corresponding to the unit to be encoded; optionally, if the first rate-distortion parameter is less than the second rate-distortion parameter, the terminal device can determine the optimal encoding method as the final encoding method corresponding to the unit to be encoded.

[0177] In other words, the terminal device can obtain the first rate-distortion parameter of the optimal encoding method and the second rate-distortion parameter of the no-division method. Further, if the first rate-distortion parameter is greater than the second rate-distortion parameter, the terminal device can determine the no-division method as the final encoding method corresponding to the unit to be encoded; alternatively, if the first rate-distortion parameter is less than the second rate-distortion parameter, the terminal device can determine the optimal encoding method as the final encoding method corresponding to the unit to be encoded; alternatively, if the first rate-distortion parameter is equal to the second rate-distortion parameter, the terminal device can determine either the optimal encoding method or the no-division method as the final encoding method corresponding to the unit to be encoded.

[0178] It should be understood that the terminal device can determine the video frames in the candidate reference frame set as the reference video frames associated with the unit to be encoded. The video encoder does not determine how to specifically select the reference video frames for encoding processing. Different selections will result in different encoding effects. In order to obtain the best encoding effect, the video encoder can encode every possible combination of reference frames, which includes highly complex motion search and motion compensation, thereby obtaining the combination of reference frames with the best encoding effect. Here, the encoding effect in the embodiments of this application can be understood as distortion. The encoding effect can be measured using rate-distortion cost. The encoding effect under rate-distortion cost can also be called rate-distortion performance. Rate-distortion performance can be measured using rate-distortion parameters (e.g., first rate-distortion parameter, second rate-distortion parameter).

[0179] It should be understood that the basic idea of ​​inter-frame prediction is to utilize the temporal correlation of video data to select the region with the most similar pixel distribution in one or two previously encoded frames to predict the current CU (i.e., the unit to be encoded). Then, only the location information of this similar region (i.e., the horizontal and vertical coordinates of the similar region in the video frame) and the pixel difference between the CU to be encoded and the similar region need to be encoded. Generally, the smaller the pixel difference, the fewer bytes need to be transmitted, and the higher the encoding efficiency. If the encoder ultimately selects not the most reasonable region for prediction, it can still generate a standard-compliant bitstream, but the encoding quality will be compromised. Finding this most reasonable region is a computationally very complex process, which the encoder often achieves by comparing pixels one by one; this process is also called motion search.

[0180] Therefore, this application embodiment can use recursion to implement a bottom-up encoding architecture. This encoding architecture can encode small blocks first, then large blocks. The key here is that the no-division method is encoded after the continuing-division method. In this way, the entire block division process will recursively go to the smallest sub-CU and encode the no-division method layer by layer upwards. At this time, when a CU (i.e., the unit to be encoded) is about to try no-division encoding, if it can continue to be divided, then the encoding of its various division methods has been completed, and the video encoder has the best encoding result of the current CU under the condition of continued division. This application embodiment will sequentially query the encoding results of each sub-CU in the current best encoding division form. If the current best encoding result meets the requirements, the reference frame list used by the current CU when performing no-division encoding will be pruned.

[0181] For easier understanding, please refer to Figure 10 , Figure 10 This is a flowchart illustrating an embodiment of the encoding of a unit to be encoded provided in this application. For example... Figure 10 As shown, the terminal device can execute step S21 to obtain the unit to be encoded from the target video frame. Further, the terminal device can execute step S22 to encode using a partitioning method, sequentially encoding each sub-coding unit (i.e., partitioning the sub-coding unit). This involves dividing the unit to be encoded into S unit partitioning methods, obtaining S unit partitioning methods for the unit to be encoded, and then sequentially encoding each sub-coding unit under the S unit partitioning methods to generate the hierarchical partitioning method corresponding to the unit to be encoded under each of the S unit partitioning methods. For example, the S unit partitioning methods may include a target unit partitioning method. The terminal device can sequentially encode N sub-coding units under the target unit partitioning method to generate the hierarchical partitioning method corresponding to the unit to be encoded under the target unit partitioning method.

[0182] Furthermore, such as Figure 10As shown, the terminal device can execute step S23, which uses a non-division method for encoding. That is, the terminal device can generate a set of candidate reference frames corresponding to the unit to be encoded in the non-division method based on S level division methods, and then perform encoding processing on the unit to be encoded based on the set of candidate reference frames to obtain the encoding result of the unit to be encoded in the non-division method.

[0183] Furthermore, such as Figure 10 As shown, the terminal device can execute step S24, which compares the rate-distortion performance of various partitioning methods and selects the optimal partitioning method (i.e., the final encoding method). Specifically, step S24 compares the rate-distortion performance of the optimal encoding method with that of the non-partitioning method, and obtains the final encoding method corresponding to the unit to be encoded from the optimal encoding method and the non-partitioning method. Further, the terminal device can execute step S25, which indicates that the encoding of the unit to be encoded has ended.

[0184] Therefore, the embodiments of this application can recursively divide the units to be encoded in the target video frame into S hierarchical division methods to obtain the S hierarchical division methods of the units to be encoded. Then, the hierarchical sub-coding units in the optimal encoding method of the S hierarchical division methods are obtained. Based on the inter-frame prediction mode and inter-frame prediction direction corresponding to the hierarchical sub-coding units, the candidate reference frame set corresponding to the unit to be encoded in the no-division method is determined. Then, the unit to be encoded is encoded based on the candidate reference frame set. Thus, the final encoding method corresponding to the unit to be encoded can be obtained from the optimal encoding method and the no-division method. Therefore, when encoding the target video frame based on the final encoding method, both the encoding effect and encoding efficiency of the target video frame can be taken into account.

[0185] Further, please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a video data processing device provided in an embodiment of this application. The video data processing device 1 may include: a segmentation module 11, an acquisition module 12, and a cropping module 13; further, the video data processing device 1 may also include: a condition judgment module 14, a determination module 15, and a parameter comparison module 16.

[0186] The partitioning module 11 is used to recursively partition the units to be encoded in the target video frame to obtain S partitioning methods for the units to be encoded; S is a positive integer; the target video frame is a video frame in the video data;

[0187] The partitioning module 11 includes: a partitioning unit 111, a mode acquisition unit 112, and a mode determination unit 113;

[0188] The partitioning unit 111 is used to partition the unit to be encoded in the target video frame to obtain S unit partitioning methods for the unit to be encoded; the S unit partitioning methods include the target unit partitioning method; the target unit partitioning method includes N partitioning sub-coding units for the unit to be encoded; N is an integer greater than 1; the N partitioning sub-coding units include the target partitioning sub-coding units;

[0189] Method acquisition unit 112 is used to acquire the final encoding method of the subunit corresponding to the target sub-encoding unit;

[0190] Among them, the method acquisition unit 112 is specifically used to recursively divide the target sub-encoding unit into S sub-unit hierarchical division methods if the target sub-encoding unit meets the unit division conditions.

[0191] The method acquisition unit 112 is specifically used to obtain the optimal encoding method of the target sub-encoding unit from the S sub-unit hierarchical division methods, and to obtain the sub-unit hierarchical sub-encoding unit in the optimal encoding method of the sub-unit.

[0192] The acquisition unit 112 is specifically used to, if the sub-unit encoding result of the sub-unit hierarchical sub-coding unit satisfies the motion similarity condition, then according to the sub-unit hierarchical sub-coding unit, the full set of sub-unit reference frames constructed for the target partitioning sub-coding unit is cropped to generate the sub-unit candidate reference frame set corresponding to the target partitioning sub-coding unit in the non-partitioning mode; the sub-unit candidate reference frame set is used to traverse the sub-unit target reference frames for the target partitioning sub-coding unit; the sub-unit target reference frames are used to encode the target partitioning sub-coding unit;

[0193] The method acquisition unit 112 is specifically used to obtain the final encoding method of the subunit corresponding to the target partitioned sub-encoding unit from the optimal encoding method and the no-partition method of the subunit.

[0194] Among them, the method acquisition unit 112 is specifically used to acquire the sub-unit size of the target sub-coding unit;

[0195] The method of obtaining unit 112 is specifically used to determine that the target sub-encoding unit satisfies the unit division condition if the size of the sub-unit is greater than or equal to the size threshold.

[0196] The method for obtaining unit 112 is specifically used to determine that if the size of the sub-unit is less than the size threshold, the target sub-encoding unit does not meet the unit partitioning condition.

[0197] The mode acquisition unit 112 is specifically used to determine the no-division mode as the final encoding mode of the sub-unit corresponding to the target sub-division encoding unit if the target sub-encoding unit does not meet the unit division conditions.

[0198] The mode determination unit 113 is used to determine the final encoding mode of the sub-units corresponding to the N sub-encoding units as the hierarchical division mode of the unit to be encoded under the target unit division mode.

[0199] The specific implementation methods of the partitioning unit 111, the method acquisition unit 112, and the method determination unit 113 can be found in the above description. Figure 7 In the corresponding embodiments, steps S1011-S1013 and... Figure 3 The description of step S101 in the corresponding embodiments will not be repeated here.

[0200] The acquisition module 12 is used to obtain the best encoding method for the unit to be encoded from S hierarchical partitioning methods, and to obtain the hierarchical sub-encoding units in the best encoding method;

[0201] The optimal encoding method includes M sub-encoding units of the unit to be encoded; M is an integer greater than 1; the M sub-encoding units include auxiliary sub-encoding units;

[0202] The acquisition module 12 includes: a first determining unit 121 and a second determining unit 122;

[0203] The first determining unit 121 is used to determine the auxiliary sub-coding unit as the hierarchical sub-coding unit in the optimal coding scheme if the auxiliary sub-coding unit does not have a sub-coding unit.

[0204] The second determining unit 122 is used to obtain the hierarchical sub-coding unit in the optimal coding method from the auxiliary sub-coding unit if the auxiliary sub-coding unit has sub-coding units.

[0205] The specific implementation methods of the first determining unit 121 and the second determining unit 122 can be found in the above description. Figure 3 The description of step S102 in the corresponding embodiments will not be repeated here.

[0206] The pruning module 13 is used to prune the full set of reference frames constructed for the unit to be encoded according to the hierarchical sub-coding unit if the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, and generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

[0207] The candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set; the full reference frame set includes a forward full reference frame set and a backward full reference frame set.

[0208] The cropping module 13 includes: a collection acquisition unit 131, a first filtering unit 132, and a second filtering unit 133;

[0209] The set acquisition unit 131 is used to acquire the forward full reference frame set and the backward full reference frame set constructed for the unit to be encoded from the video data;

[0210] Among them, the set acquisition unit 131 is specifically used to acquire encoded video frames whose encoding order is earlier than the target video frame from the video data;

[0211] The set acquisition unit 131 is specifically used to add the encoded video frames whose playback order is earlier than the target video frame to the forward full reference frame set constructed for the unit to be encoded if the playback order of the encoded video frames is earlier than the target video frame.

[0212] The set acquisition unit 131 is specifically used to add the encoded video frames whose playback order is later than the target video frame to the backward full reference frame set constructed for the unit to be encoded if the playback order of the encoded video frames is later than the target video frame.

[0213] The first filtering unit 132 is used to filter the reference frames used by the hierarchical sub-coding unit in the forward full reference frame set. If there are reference frames used by the hierarchical sub-coding unit in the forward full reference frame set, the reference frames filtered in the forward full reference frame set will be determined as the forward candidate reference frame set corresponding to the unit to be encoded in the no-division mode.

[0214] The second filtering unit 133 is used to filter the reference frames used by the hierarchical sub-coding units in the backward full reference frame set. If there are reference frames used by the hierarchical sub-coding units in the backward full reference frame set, the reference frames filtered in the backward full reference frame set will be determined as the backward candidate reference frame set corresponding to the unit to be encoded in the non-division mode.

[0215] The specific implementation methods of the set acquisition unit 131, the first filtering unit 132, and the second filtering unit 133 can be found in the above description. Figure 3 The description of step S103 in the corresponding embodiments will not be repeated here.

[0216] Optionally, the number of hierarchical sub-coding units is P; P is an integer greater than 1; the P hierarchical sub-coding units include the target hierarchical sub-coding unit;

[0217] Condition judgment module 14 is used to obtain the inter-frame prediction mode and inter-frame prediction direction corresponding to the target level sub-coding unit;

[0218] The condition judgment module 14 is used to determine that the encoding results of the P level sub-coding units satisfy the motion similarity condition if the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame predictions and the inter-frame prediction directions corresponding to the P level sub-coding units are all the same.

[0219] The condition judgment module 14 is used to determine that the encoding results of the P level sub-coding units do not meet the motion similarity condition if there is a level sub-coding unit among the P level sub-coding units whose inter-frame prediction mode is not translation inter-frame prediction, or if the inter-frame prediction directions corresponding to the P level sub-coding units are different.

[0220] Among them, the condition judgment module 14 is specifically used to obtain the inter-frame prediction direction corresponding to the target level sub-coding unit; the inter-frame prediction direction corresponding to the target level sub-coding unit includes forward prediction, backward prediction and bidirectional prediction.

[0221] Condition judgment module 14 is specifically used to obtain the motion vector corresponding to each pixel in the target level sub-coding unit;

[0222] The condition judgment module 14 is specifically used to determine the translation inter-frame prediction as the inter-frame prediction mode corresponding to the target level sub-coding unit if the motion vectors corresponding to each pixel in the target level sub-coding unit are the same.

[0223] The condition judgment module 14 is specifically used to determine the non-translation inter-frame prediction as the inter-frame prediction mode corresponding to the target level sub-coding unit if there are pixels with different motion vectors in the target level sub-coding unit.

[0224] Optionally, the determining module 15 is used to obtain a full set of reference frames constructed for the unit to be encoded if the encoding result of the hierarchical sub-coding unit does not meet the motion similarity condition, and to determine the full set of reference frames as the candidate reference frame set corresponding to the unit to be encoded in the non-partition mode.

[0225] Optionally, parameter comparison module 16 is used to obtain the first rate-distortion parameter of the optimal encoding method and the second rate-distortion parameter of the non-division method;

[0226] The parameter comparison module 16 is used to determine the no-division mode as the final encoding mode corresponding to the unit to be encoded if the first rate distortion parameter is greater than or equal to the second rate distortion parameter.

[0227] The parameter comparison module 16 is used to determine the optimal encoding method as the final encoding method corresponding to the unit to be encoded if the first rate-distortion parameter is less than the second rate-distortion parameter.

[0228] The specific implementation methods of the partitioning module 11, the acquisition module 12, and the trimming module 13 can be found in the above description. Figure 3In the corresponding embodiments, steps S101-S103 and... Figure 7 The descriptions of steps S1011-S1013 in the corresponding embodiments will not be repeated here. The specific implementation methods of the condition judgment module 14, the determination module 15, and the parameter comparison module 16 can be found above. Figure 9 The descriptions of steps S201-S208 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0229] Further, please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 12 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. In some embodiments, the user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. Optionally, the network interface 1004 may include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 12 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0230] In such Figure 8 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0231] The units to be encoded in the target video frame are recursively divided into S levels, resulting in S levels of division for the units to be encoded; S is a positive integer; the target video frame is a video frame in the video data.

[0232] From the S hierarchical partitioning methods, obtain the optimal encoding method for the unit to be encoded, and obtain the hierarchical sub-encoding units in the optimal encoding method;

[0233] If the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, then the full set of reference frames constructed for the unit to be encoded is pruned according to the hierarchical sub-coding unit to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded.

[0234] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 7 or Figure 9 The description of the video data processing method in the corresponding embodiments can also be performed as described above. Figure 11 The description of the video data processing apparatus 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0235] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned video data processing device 1. When the processor executes the computer program, it can execute the aforementioned... Figure 3 , Figure 7 or Figure 9 The description of the video data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0236] Furthermore, it should be noted that this application also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 3 , Figure 7 or Figure 9 The description of the video data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program product embodiments related to this application, please refer to the description of the method embodiments of this application.

[0237] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0238] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video data processing method, characterized in that, include: The unit to be encoded in the target video frame is recursively divided into S levels to obtain the S levels of the unit to be encoded; where S is a positive integer; and the target video frame is a video frame in the video data. From the S hierarchical partitioning methods, obtain the optimal encoding method for the unit to be encoded, and obtain the hierarchical sub-encoding unit in the optimal encoding method; If the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, then the full set of reference frames constructed for the unit to be encoded is cropped according to the hierarchical sub-coding unit to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded. Wherein, the number of the hierarchical sub-coding units is P; where P is an integer greater than 1; the method further includes: If the inter-frame prediction modes corresponding to the P hierarchical sub-coding units are all translational inter-frame predictions, and the inter-frame prediction directions corresponding to the P hierarchical sub-coding units are all the same, then the coding results of the P hierarchical sub-coding units are determined to satisfy the motion similarity condition. If any of the P hierarchical sub-coding units has an inter-frame prediction mode that is not the translation inter-frame prediction mode, or if the inter-frame prediction directions corresponding to the P hierarchical sub-coding units are different, then it is determined that the coding results of the P hierarchical sub-coding units do not satisfy the motion similarity condition.

2. The method according to claim 1, characterized in that, The recursive hierarchical partitioning of the units to be encoded in the target video frame to obtain S hierarchical partitioning methods for the units to be encoded includes: The target video frame is divided into units to be encoded, resulting in S unit division methods for the unit to be encoded; the S unit division methods include the target unit division method; the target unit division method includes N sub-coding units of the unit to be encoded; N is an integer greater than 1; the N sub-coding units include the target sub-coding unit. Obtain the final encoding method of the sub-unit corresponding to the target sub-encoding unit; The final encoding method of each of the N sub-encoding units is determined as the hierarchical division method of the unit to be encoded under the target unit division method.

3. The method according to claim 2, characterized in that, The step of obtaining the final encoding method of the sub-unit corresponding to the target sub-coding unit includes: If the target sub-coding unit satisfies the unit partitioning condition, then the target sub-coding unit is recursively partitioned hierarchically to obtain the S sub-unit hierarchical partitioning method of the target sub-coding unit; From the S sub-unit hierarchical partitioning methods, obtain the optimal sub-unit encoding method of the target partitioned sub-coding unit, and obtain the sub-unit hierarchical sub-coding unit in the optimal sub-unit encoding method; If the sub-unit encoding result of the sub-unit hierarchical sub-coding unit satisfies the motion similarity condition, then the full set of sub-unit reference frames constructed for the target partitioning sub-coding unit is truncated according to the sub-unit hierarchical sub-coding unit to generate the sub-unit candidate reference frame set corresponding to the target partitioning sub-coding unit in the no-partition mode; the sub-unit candidate reference frame set is used to traverse the sub-unit target reference frames for the target partitioning sub-coding unit; the sub-unit target reference frames are used to encode the target partitioning sub-coding unit; The final encoding scheme of the subunit corresponding to the target partitioned sub-encoding unit is obtained from the optimal encoding scheme of the subunit and the no-partition scheme.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the sub-unit size of the target sub-coding unit; If the size of the sub-unit is greater than or equal to the size threshold, then the target sub-encoding unit is determined to satisfy the unit partitioning condition; If the size of the sub-unit is smaller than the size threshold, then the target sub-encoding unit is determined not to meet the unit partitioning condition.

5. The method according to claim 3, characterized in that, The method further includes: If the target sub-coding unit does not meet the unit partitioning condition, then the non-partitioning method is determined as the final encoding method of the sub-unit corresponding to the target sub-coding unit.

6. The method according to claim 1, characterized in that, The optimal encoding method includes M sub-encoding units of the unit to be encoded; M is an integer greater than 1; the M sub-encoding units include auxiliary sub-encoding units; The step of obtaining the hierarchical sub-coding units in the optimal encoding method includes: If the auxiliary partitioning sub-coding unit does not have a sub-coding unit, then the auxiliary partitioning sub-coding unit is determined as the hierarchical sub-coding unit in the optimal encoding method; If the auxiliary partitioning sub-coding unit has sub-coding units, then the hierarchical sub-coding unit in the optimal encoding method is obtained from the auxiliary partitioning sub-coding unit.

7. The method according to claim 1, characterized in that, The candidate reference frame set includes a forward candidate reference frame set and a backward candidate reference frame set; the full reference frame set includes a forward full reference frame set and a backward full reference frame set. The step of pruning the full set of reference frames constructed for the unit to be encoded based on the hierarchical sub-coding units to generate a set of candidate reference frames corresponding to the unit to be encoded in the non-partitioning mode includes: Obtain the forward full reference frame set and the backward full reference frame set constructed for the unit to be encoded from the video data; The reference frames used by the hierarchical sub-coding unit are filtered from the forward full reference frame set. If there are reference frames used by the hierarchical sub-coding unit in the forward full reference frame set, the reference frames filtered from the forward full reference frame set are determined as the forward candidate reference frame set corresponding to the unit to be encoded in the no-partition mode. The reference frames used by the hierarchical sub-coding unit are selected from the backward full reference frame set. If the reference frames used by the hierarchical sub-coding unit exist in the backward full reference frame set, the reference frames selected from the backward full reference frame set are determined as the backward candidate reference frame set corresponding to the unit to be encoded under the no-division mode.

8. The method according to claim 7, characterized in that, The step of obtaining the forward full reference frame set and the backward full reference frame set constructed from the video data for the unit to be encoded includes: Obtain encoded video frames whose encoding order is earlier than the target video frame from the video data; If the playback order of the encoded video frames is earlier than that of the target video frames, then the encoded video frames whose playback order is earlier than that of the target video frames are added to the forward full reference frame set constructed for the unit to be encoded. If the playback order of the encoded video frame is later than that of the target video frame, then the encoded video frame whose playback order is later than that of the target video frame is added to the backward full reference frame set constructed for the unit to be encoded.

9. The method according to claim 1, characterized in that, The P hierarchical sub-coding units include target hierarchical sub-coding units; The method further includes: Obtain the inter-frame prediction direction corresponding to the target level sub-coding unit; the inter-frame prediction direction corresponding to the target level sub-coding unit includes forward prediction, backward prediction, and bidirectional prediction; Obtain the motion vector corresponding to each pixel in the target-level sub-coding unit; If the motion vectors corresponding to each pixel in the target-level sub-coding unit are the same, then the translation inter-frame prediction is determined as the inter-frame prediction mode corresponding to the target-level sub-coding unit. If there are pixels with different motion vectors in the target-level sub-coding unit, then the non-translation inter-frame prediction is determined as the inter-frame prediction mode corresponding to the target-level sub-coding unit.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: If the encoding result of the hierarchical sub-coding unit does not meet the motion similarity condition, then a full set of reference frames constructed for the unit to be encoded is obtained, and the full set of reference frames is determined as the candidate reference frame set corresponding to the unit to be encoded under the no-partitioning mode.

11. The method according to any one of claims 1-9, characterized in that, The method further includes: Obtain the first rate-distortion parameter of the optimal encoding method and the second rate-distortion parameter of the non-division method; If the first rate-distortion parameter is greater than or equal to the second rate-distortion parameter, then the non-division method is determined as the final encoding method corresponding to the unit to be encoded; If the first rate-distortion parameter is less than the second rate-distortion parameter, then the optimal encoding method is determined as the final encoding method corresponding to the unit to be encoded.

12. A video data processing apparatus, characterized in that, include: The partitioning module is used to recursively partition the units to be encoded in the target video frame to obtain S partitioning methods for the units to be encoded. S is a positive integer; the target video frame is a video frame in the video data. The acquisition module is used to acquire the optimal encoding method of the unit to be encoded from the S hierarchical partitioning methods, and to acquire the hierarchical sub-encoding units in the optimal encoding method; The pruning module is used to prune the full set of reference frames constructed for the unit to be encoded according to the hierarchical sub-coding unit if the encoding result of the hierarchical sub-coding unit satisfies the motion similarity condition, thereby generating a set of candidate reference frames corresponding to the unit to be encoded in the non-partition mode; the set of candidate reference frames is used to traverse the target reference frames for the unit to be encoded; the target reference frames are used to encode the unit to be encoded. The number of hierarchical sub-coding units is P; where P is an integer greater than 1; the device further includes: The condition judgment module is used to determine that the encoding results of the P level sub-coding units satisfy the motion similarity condition if the inter-frame prediction modes corresponding to the P level sub-coding units are all translation inter-frame predictions and the inter-frame prediction directions corresponding to the P level sub-coding units are all the same. The condition judgment module is further configured to determine that the encoding results of the P level sub-coding units do not meet the motion similarity condition if there is a level sub-coding unit among the P level sub-coding units whose inter-frame prediction mode is not translation inter-frame prediction, or if the inter-frame prediction directions corresponding to the P level sub-coding units are different.

13. A computer device, characterized in that, include: Processor and memory; The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to cause the computer device to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method according to any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

Images