Video processing method and device and related equipment

Abstract

The invention provides a video processing method and device and related equipment, and relates to the technical field of video processing. The method comprises the following steps: performing feature extraction on a coding unit in a video to be processed to obtain feature information of the coding unit; performing priority decision on a plurality of coding modes in a coding mode pool according to the feature information of the coding units to obtain a coding mode sequence corresponding to the coding units; the sequence of the coding modes is the sequence of the priorities of the coding modes from high to low; and performing mode search on the plurality of coding modes based on the coding mode sequence corresponding to the coding unit to obtain a target coding mode of the coding unit. According to the method, the search sequence of the encoding modes can be adaptively adjusted, so that an encoder search architecture supports top-down search and bottom-up search, high-dimensional information is fully utilized in the mode search process, the search time and the calculation amount are reduced, and the speed of an encoder is increased.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the video coding framework, in order to achieve efficient video compression, the encoder searches for the optimal combination of coding tools among many combinations of coding tools through Rate-Distortion Optimization (RDO). RDO search mainly includes the selection of prediction mode and division method, but the full search method has huge computational complexity and high time consumption, which becomes the speed bottleneck of the encoder. In related technologies, in order to reduce the search time, the encoder adopts a fast algorithm. However, the search architecture can only perform top-down search (i.e., search large blocks first, then search small blocks), and the search order of the prediction mode is fixed, which makes it difficult to fully utilize high-dimensional information, limiting the speed of the encoder.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a video processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product to overcome the above problems or at least partially solve the above problems.

[0005] An embodiment of the present disclosure provides a video processing method, the method comprising: extracting features from a coding unit in a video to be processed to obtain feature information of the coding unit; making priority decisions on a plurality of coding modes in a coding mode pool according to the feature information of the coding unit to obtain a coding mode sequence corresponding to the coding unit; the coding mode sequence is a sequence of coding modes from high to low priority; based on the coding mode sequence corresponding to the coding unit, performing a mode search on the plurality of coding modes to obtain a target coding mode of the coding unit.

[0006] In some embodiments of the present disclosure, the method of performing a pattern search on the multiple coding modes based on the coding mode sequence corresponding to the coding unit to obtain a target coding mode of the coding unit includes: selecting a current coding mode based on the coding mode sequence corresponding to the coding unit; encoding the coding unit based on the current coding mode to obtain a coding quality indicator corresponding to the current coding mode; selecting a next coding mode of the current coding mode based on the coding mode sequence corresponding to the coding unit, determining the next coding mode as a new current coding mode, and encoding based on the new current coding mode until the coding of the multiple coding modes is completed; obtaining the coding quality indicators corresponding to the multiple coding modes, and obtaining the target coding mode of the coding unit according to the coding quality indicators corresponding to the multiple coding modes.

[0007] In some embodiments of the present disclosure, the encoding processing of the encoding unit based on the current encoding mode includes: determining whether to trigger the current encoding mode according to the encoding information of the encoding mode of the encoding unit that has completed the encoding processing; if it is determined that the current encoding mode is triggered, encoding processing is performed on the encoding unit based on the current encoding mode to obtain an encoding quality indicator corresponding to the current encoding mode; if it is determined that the current encoding mode is not triggered, determining to skip the current encoding mode.

[0008] In some embodiments of the present disclosure, priority decisions are made on multiple coding modes in a coding mode pool based on feature information of the coding unit to obtain a coding mode order corresponding to the coding unit, including: inputting the feature information of the coding unit into a pre-trained mode decision model, and outputting the coding mode order corresponding to the coding unit; the pre-trained mode decision model includes a fully connected network.

[0009] In some embodiments of the present disclosure, the pre-trained mode decision model is trained according to the following method: determine an initial mode decision model; obtain coding unit training samples, feature information of the coding unit training samples, and a coding mode sequence corresponding to the coding unit training samples; input the feature information of the coding unit training samples into the initial mode decision model, and output a predicted coding mode sequence corresponding to the coding unit training samples; calculate a model loss based on the coding mode sequence corresponding to the coding unit training samples and the predicted coding mode sequence corresponding to the coding unit training samples; adjust model parameters of the initial mode decision model based on the model loss to obtain the pre-trained mode decision model.

[0010] In some embodiments of the present disclosure, the method also includes: obtaining a prediction mode and a division method; the prediction mode includes one or more of an intra-frame prediction mode, an inter-frame fusion prediction mode, an advanced motion vector prediction mode and an adaptive motion vector precision prediction mode, and the division method includes one or more of a quadtree division method, a ternary tree division method and a binary tree division method; adding the prediction mode and the division method to the coding mode pool.

[0011] In some embodiments of the present disclosure, the method also includes: splitting the prediction mode to obtain a first prediction sub-mode and a second prediction sub-mode; the first prediction sub-mode is a coarse-grained search mode including a prediction process, and the second prediction sub-mode is a fine-grained search mode including a transform quantization reconstruction process.

[0012] In some embodiments of the present disclosure, the feature information of the coding unit includes one or more of pre-analysis feature information, feature information of adjacent coding units, feature information of a parent block, and quantization parameter information; wherein the pre-analysis feature information includes one or more of intra-frame loss, inter-frame loss, and texture complexity obtained in the pre-analysis stage; the feature information of adjacent coding units includes one or more of a division depth of an adjacent coding unit, a coding mode of an adjacent coding unit, a rate-distortion value of an adjacent coding unit, a motion vector of an adjacent coding unit, and a reference image of an adjacent coding unit; the feature information of the parent block includes one or more of an optimal coding mode of the parent block and rate-distortion values ​​of each coding mode of the parent block.

[0013] An embodiment of the present disclosure provides a video processing device, which includes: a feature extraction module, configured to extract features from a coding unit in a video to be processed, and obtain feature information of the coding unit; a priority decision module, configured to make priority decisions on multiple coding modes in a coding mode pool according to the feature information of the coding unit, and obtain a coding mode sequence corresponding to the coding unit; the coding mode sequence is a descending order of the priority of the coding modes; and a mode search module, configured to perform a mode search on the multiple coding modes based on the coding mode sequence corresponding to the coding unit, and obtain a target coding mode of the coding unit.

[0014] An embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the above-mentioned video processing method.

[0015] An embodiment of the present disclosure provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above-mentioned video processing method.

[0016] The embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above-mentioned video processing method is implemented.

[0017] The video processing method provided by the embodiment of the present disclosure extracts features from the coding units in the processed video to obtain feature information, and makes priority decisions on multiple coding modes in the coding mode pool based on the feature information to obtain the coding mode order corresponding to the coding units, and then performs a mode search based on the order to obtain the target coding mode. In this way, the search order of the coding modes can be adaptively adjusted so that the encoder search architecture supports top-down search (i.e., searching large blocks first, then searching small blocks) and bottom-up search (i.e., searching small blocks first, then searching large blocks), and then in the mode search process, the high-dimensional information from its parent block and child block can be fully utilized, thereby helping the encoder to more accurately skip the search process of some modes, reduce the search time and calculation amount, and improve the speed of the encoder.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A flow chart of performing pattern search based on RDO method in the related art is shown;

[0021] Figure 2 A schematic diagram showing an exemplary system architecture to which the video processing method according to an embodiment of the present disclosure can be applied;

[0022] Figure 3 A flow chart of a video processing method according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of the training process of the pre-trained mode decision model of an embodiment of the present disclosure is shown;

[0024] Figure 5 A flowchart of performing a mode search based on the coding mode sequence corresponding to the coding unit to obtain a target coding mode according to an embodiment of the present disclosure is shown;

[0025] Figure 6 A flowchart of another video processing method according to an embodiment of the present disclosure is shown;

[0026] Figure 7 A flowchart of another video processing method according to an embodiment of the present disclosure is shown;

[0027] Figure 8 A block diagram of a video processing device according to an embodiment of the present disclosure is shown;

[0028] Fig. 9 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0030] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] The collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of the user personal information involved in this disclosure are in compliance with the relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0032] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the term "plurality" refers to two or more; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0035] In the video coding framework, in order to achieve efficient video compression, the encoder searches for the optimal coding tool combination among many coding tool combinations through RDO. Figure 1 FIG. 1 shows a flow chart of performing pattern search based on RDO in the related art. Figure 1 ,The process of pattern searching based on RDO method in the related technology includes the following steps.

[0036] Step S101: for a coding unit in a video image, determine that the coding unit is a current processing unit.

[0037] Step S102, extract low-dimensional information of the current processing unit to obtain low-dimensional information of the current processing unit.

[0038] Step S103, based on the low-dimensional information of the current processing unit, a mode skipping decision is made for the current processing unit. Specifically, based on the low-dimensional information of the current processing unit, it is evaluated that some encoding modes may not be optimal, and these encoding modes are skipped.

[0039] Step S104, searching for a prediction mode for the current processing unit to obtain an optimal prediction mode for the current processing unit.

[0040] Step S105, making a partitioning and pruning decision for the current processing unit.

[0041] Step S106, if no pruning is performed, the current processing unit is divided into sub-coding units, and the sub-coding unit is determined to be the new current processing unit, and the process returns to step S102, and the above low-dimensional information extraction, mode skipping decision, prediction mode search, and partitioning and pruning decision are repeated for the new current processing unit. The recursive process will continue until the predetermined minimum coding unit size is reached or no further division is required.

[0042] Step S107: If pruning is performed, or after pruning analysis is performed and the current coding unit is returned, the optimal coding mode of the coding unit is obtained.

[0043] An analysis of the RDO search architecture of the related art shows that it has the following disadvantages:

[0044] (1) From the perspective of block division, this search architecture can only perform a top-down search, that is, first search large blocks and then search small blocks. This means that each coding unit can only use the high-dimensional information of its parent block during RDO search, but cannot use the high-dimensional information of its child blocks, which limits the encoder's ability to accelerate. In addition, if the video content is more suitable for deep division, this top-down search order is not conducive to the encoder quickly locating its appropriate division level, wasting time.

[0045] (2) The search order of prediction modes is fixed, that is, the prediction mode that comes first cannot utilize the high-dimensional information provided by the prediction mode that comes later. The earlier the order, the less high-dimensional information that can be used. This limits the acceleration capability of the prediction mode that comes first.

[0046] In order to solve the above problems, the embodiments of the present disclosure provide a video processing method, an apparatus and related equipment, which can adaptively adjust the search order of coding modes, so that the encoder search architecture supports top-down search (i.e., searching large blocks first, then searching small blocks) and bottom-up search (i.e., searching small blocks first, then searching large blocks), so that the high-dimensional information from its parent blocks and child blocks can be fully utilized in the mode search process, thereby helping the encoder to more accurately skip the search process of some modes, reduce the search time and calculation amount, and improve the speed of the encoder.

[0047] Figure 2 The diagram shows an exemplary system architecture to which the video processing method according to the embodiment of the present disclosure can be applied.

[0048] like Figure 2 As shown, the system architecture may include a server 201, a network 202, and a client 203. The network 202 is used to provide a medium for a communication link between the client 203 and the server 201. The network 202 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0049] In an exemplary embodiment, the client 203 for data transmission with the server 201 may include but is not limited to a smart phone, a tablet computer, a laptop computer, a smart speaker, a digital assistant, an AR (Augmented Reality) device, a VR (Virtual Reality) device, a smart wearable device, etc., or the client 203 may also be a personal computer, such as a laptop computer and a desktop computer, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0050] Server 201 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 201 can also be a server of a network platform, which can be, for example, a trading platform, a live broadcast platform, a social platform, or an audio platform, etc., which is not limited in the embodiments of the present disclosure. Among them, the server can be a single server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0051] In some embodiments of the present disclosure, the video processing process of server 201 may be: extracting features from coding units in the video to be processed to obtain feature information of the coding units; making priority decisions on multiple coding modes in a coding mode pool based on the feature information of the coding units to obtain a coding mode order corresponding to the coding units; the coding mode order is the order of the coding modes from high to low priority; based on the coding mode order corresponding to the coding units, performing a mode search on multiple coding modes to obtain a target coding mode for the coding units.

[0052] In addition, it should be noted that Figure 2 What is shown is only one application environment of the method for processing video encoding parameters provided by the present disclosure. Figure 2 The numbers of the clients 203, networks 202 and servers 201 are merely illustrative, and any number of clients, networks and servers may be provided according to actual needs.

[0053] Figure 3 A flowchart of a video processing method according to an embodiment of the present disclosure is shown. Figure 3 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 2The server 201 in the embodiment is also as follows Figure 2 Client 203 in, and Figure 2 The server 201 and the client 203 in the embodiment jointly implement the video processing method, but the present disclosure is not limited thereto. Figure 3 The video processing method provided by the embodiment of the present disclosure includes the following steps.

[0054] Step S310: extract features from coding units in the video to be processed to obtain feature information of the coding units.

[0055] In the embodiments of the present disclosure, the video to be processed refers to a video waiting to be encoded, and the video to be processed includes at least one video frame to be processed, each video frame to be processed is an image in the video data, and each video frame to be processed is divided into at least one encoding unit, wherein the encoding unit refers to the basic processing unit in the encoding process.

[0056] In the disclosed embodiment, feature extraction is performed on a coding unit in a processed video to obtain feature information of the coding unit, such as spatial features, temporal features, statistical features, features of adjacent coding units, features of a parent block, and quantization parameters.

[0057] In some embodiments of the present disclosure, the feature information of the coding unit includes one or more of pre-analysis feature information, feature information of adjacent coding units, feature information of a parent block, and quantization parameter information.

[0058] In the disclosed embodiment, the pre-analysis feature information refers to information obtained by performing a preliminary analysis on the coding unit before formal encoding. The information in the pre-analysis stage is mainly used to guide subsequent coding decisions, such as selecting an appropriate prediction mode, division method, etc.

[0059] The pre-analysis feature information includes one or more of the intra-frame loss, inter-frame loss, and texture complexity obtained in the pre-analysis stage. Of course, in addition to the intra-frame loss, inter-frame loss, and texture complexity, other information may also be obtained in the pre-analysis stage, such as content type, region of interest, etc., which is not limited in the embodiments of the present disclosure.

[0060] Intra-frame loss refers to the difference between the coding unit and its prediction value, which is used to evaluate the degree of distortion of the coding unit in intra-frame prediction mode. Inter-frame loss refers to the difference between the prediction value of the coding unit and its reference frame, which is used to evaluate the degree of distortion of the coding unit in inter-frame prediction mode. Texture complexity is used to describe the texture characteristics of the coding unit, reflecting the complexity of the image content, and is used to assist in selecting an appropriate prediction mode.

[0061] In the disclosed embodiment, the feature information of an adjacent coding unit refers to relevant information of other coding units adjacent to the coding unit. In video coding, the coding unit and its adjacent coding units usually have certain similarities or correlations, so the feature information of the adjacent coding units can help the coding unit make reasonable coding decisions.

[0062] The feature information of the adjacent coding unit includes one or more of the division depth of the adjacent coding unit, the coding mode of the adjacent coding unit, the rate-distortion value of the adjacent coding unit, the motion vector of the adjacent coding unit, and the reference image of the adjacent coding unit. Of course, in addition to the division depth, coding mode, rate-distortion value, motion vector, and reference image of the adjacent coding unit, the feature information of the adjacent coding unit may also include other information, such as residual energy, coding cost, prediction direction, etc. of the adjacent coding unit, which is not limited in the embodiments of the present disclosure.

[0063] The feature information of the adjacent coding unit may include one or more of the following options: a split depth of the adjacent coding unit, an encoding mode of the adjacent coding unit, a rate-distortion value of the adjacent coding unit, a motion vector of the adjacent coding unit, and a reference image of the adjacent coding unit.

[0064] The division depth of adjacent coding units is used to indicate the hierarchical depth of adjacent coding units in hierarchical division, and is used to guide the division strategy of the coding unit. If the division depth of adjacent coding units is deeper, it may mean that the coding unit also needs to be divided into finer granularity.

[0065] The coding mode of the adjacent coding unit refers to the coding method used by the adjacent coding unit, including intra-frame prediction mode, inter-frame prediction mode, etc., which is used to guide the coding mode selection of the coding unit. If the adjacent coding unit selects a certain mode, it may mean that the coding unit is also suitable for the mode.

[0066] The rate-distortion value of adjacent coding units is an important indicator for measuring coding quality. It is usually expressed as a combination of bit rate and distortion after encoding, and is used to evaluate the encoding effect of coding units in different modes. If the rate-distortion value of an adjacent coding unit in a certain mode is good, it may mean that the coding unit is also suitable for this mode.

[0067] The motion vector of the adjacent coding unit indicates the motion direction and magnitude of the adjacent coding unit in the inter-frame prediction, and is used to guide the inter-frame prediction of the coding unit. If the motion vectors of the adjacent coding units have a certain regularity, it may mean that the coding unit also has similar motion characteristics.

[0068] The reference image of the adjacent coding unit is a reference frame used in inter-frame prediction, which is used to guide the inter-frame prediction of the coding unit. If a reference image is used by an adjacent coding unit, it may mean that the coding unit is also suitable to use the same reference image.

[0069] In the disclosed embodiment, the characteristic information of the parent block refers to the information of the larger coding unit to which the coding unit belongs. Since the characteristic information of the parent block usually affects the coding mode selection of the coding unit, this information can be used to guide the coding decision of the coding unit.

[0070] The characteristic information of the parent block includes one or more of the optimal coding mode of the parent block and the rate-distortion values ​​of each coding mode of the parent block. Of course, in addition to the optimal coding mode of the parent block and the rate-distortion values ​​of each coding mode of the parent block, the characteristic information of the parent block may also include other information, such as the coding block size of the parent block, the directionality information of the parent block, etc., which is not limited in the embodiments of the present disclosure.

[0071] The optimal coding mode of the parent block refers to the optimal coding mode selected by the parent block during the coding process, which can be obtained by RDO. The optimal coding mode of the parent block is used to guide the selection of the coding mode of the coding unit. Specifically, if the parent block selects a certain mode, the coding unit may also be suitable for the mode, especially in the scenario where the parent block and the coding unit have similar characteristics.

[0072] The rate-distortion value of each coding mode of the parent block refers to the rate-distortion value of the parent block in different coding modes during the encoding process, which may include a combination of bit rate and distortion, and is used to evaluate the encoding effect of the coding unit in different modes. The rate-distortion value of the parent block can provide information about which modes perform better in the parent block, thereby helping the coding unit make a more reasonable mode selection.

[0073] In the disclosed embodiment, the quantization parameter information of the coding unit refers to a parameter used to control the degree of quantization during the video encoding process, which affects the quality and bit rate of the encoded video. The quantization parameter information of the coding unit may include one or more of the current quantization parameter, the quantization step size, the change of the quantization parameter, and the historical information of the quantization parameter.

[0074] It should be noted that in addition to pre-analysis feature information, feature information of adjacent coding units, feature information of parent blocks and quantization parameter information, the feature information of the coding unit may also include other feature information, such as pixel value features, edge information, frequency domain features, etc., which is not limited in this embodiment of the present disclosure.

[0075] In the video processing method provided by the embodiment of the present disclosure, feature information of the coding unit is obtained by extracting features from the coding unit in the video to be processed, and various characteristics of the coding unit can be captured comprehensively and accurately, including the loss and complexity of the pre-analysis stage, the coding characteristics of adjacent coding units, the coding mode and performance of the parent block, and quantization parameter information. These feature information reflects the content characteristics and coding requirements of the coding unit, and provides an important basis for subsequent decisions on the coding mode sequence.

[0076] Step S320, making a priority decision on multiple coding modes in the coding mode pool according to the characteristic information of the coding unit, and obtaining a coding mode order corresponding to the coding unit; the coding mode order is the order of the coding modes from high priority to low priority.

[0077] In the disclosed embodiment, the coding mode pool is a pre-constructed mode pool, and multiple coding modes are stored in the coding mode pool. According to the characteristic information of the coding unit, priority decisions are made on multiple coding modes in the coding mode pool to obtain the coding mode order corresponding to the coding unit. This order reflects the arrangement of the priority of the coding mode from high to low. For example, a machine learning model (such as a decision tree, random forest, neural network, etc.) or a heuristic algorithm (such as a rule-based classifier) ​​can be used to evaluate the applicability of each coding mode in the coding mode pool to obtain the coding mode order corresponding to the coding unit.

[0078] Step S330: Based on the coding mode sequence corresponding to the coding unit, a mode search is performed on multiple coding modes to obtain a target coding mode of the coding unit.

[0079] In the disclosed embodiment, the coding modes are evaluated one by one according to the coding mode sequence corresponding to the coding unit, and each coding mode is coded to evaluate the performance corresponding to the coding mode, and finally the optimal coding mode is selected as the target coding mode of the coding mode.

[0080] The video processing method provided by the embodiment of the present disclosure extracts features from the coding units in the processed video to obtain feature information, and makes priority decisions on multiple coding modes in the coding mode pool based on the feature information to obtain the coding mode order corresponding to the coding units, and then performs a mode search based on the order to obtain the target coding mode. In this way, the search order of the coding modes can be adaptively adjusted so that the encoder search architecture supports top-down search (i.e., searching large blocks first, then searching small blocks) and bottom-up search (i.e., searching small blocks first, then searching large blocks), and then in the mode search process, the high-dimensional information from its parent block and child block can be fully utilized, thereby helping the encoder to more accurately skip the search process of some modes, reduce the search time and calculation amount, and improve the speed of the encoder.

[0081] Next, the steps of video processing in the embodiment of the present disclosure are further described.

[0082] In step S310, feature extraction is performed on the coding unit in the video to be processed to obtain feature information of the coding unit.

[0083] In this step, feature information of the coding units in the processed video is obtained by extracting features of the coding units. The feature information reflects the content characteristics and coding requirements of the coding units, and provides an important basis for the subsequent decision on the coding mode sequence.

[0084] The characteristic information of the coding unit may include one or more of pre-analysis characteristic information, characteristic information of adjacent coding units, characteristic information of the parent block and quantization parameter information. The pre-analysis characteristic information includes one or more of intra-frame loss, inter-frame loss and texture complexity obtained in the pre-analysis stage; the characteristic information of the adjacent coding unit includes one or more of the division depth of the adjacent coding unit, the coding mode of the adjacent coding unit, the rate-distortion value of the adjacent coding unit, the motion vector of the adjacent coding unit and the reference image of the adjacent coding unit; the characteristic information of the parent block includes one or more of the optimal coding mode of the parent block and the rate-distortion value of each coding mode of the parent block.

[0085] In step S320, priority decisions are made on multiple coding modes in the coding mode pool according to the characteristic information of the coding unit to obtain a coding mode order corresponding to the coding unit; the coding mode order is the order of the coding modes from high priority to low priority.

[0086] In this step, priority decisions are made on multiple coding modes in the coding mode pool based on the characteristic information of the coding unit to obtain the coding mode order corresponding to the coding unit. This order reflects the arrangement of the priority of the coding modes from high to low, so that the search process can first focus on the coding mode that is most likely to obtain the best coding effect, thereby optimizing the coding effect.

[0087] In some embodiments of the present disclosure, the video processing method also includes: obtaining a prediction mode and a division method; the prediction mode includes one or more of an intra-frame prediction mode, an inter-frame fusion prediction mode and an advanced motion vector prediction mode, and the division method includes one or more of a quadtree division method, a ternary tree division method and a binary tree division method; adding the prediction mode and the division method to the coding mode pool.

[0088] In the disclosed embodiments, the prediction mode refers to a method for predicting video frames according to the temporal or spatial correlation between video frames in order to reduce data redundancy and improve compression efficiency during the video encoding process. The prediction mode includes one or more of the intra-frame prediction mode (Intra), the inter-frame fusion prediction mode (Merge), the Advanced Motion Vector Prediction mode (AMVP) and the Adaptive Motion Vector Resolution (AMVR).

[0089] Intra can use the pixel values ​​in the current frame for prediction. It predicts the pixel values ​​in the current coding block by analyzing the encoded pixel values ​​around the current coding block. Merge can use the temporal correlation between adjacent frames for prediction and use the motion information of adjacent blocks to predict the current block. AMVP is a complex inter-frame prediction mode. It is a process of predicting the current block by analyzing the motion vector between the current block and its reference frame and using the reference block at the corresponding position of the reference frame. AMVR is a technology in the inter-frame prediction mode. AMVR technology is based on inter-frame prediction and further improves the prediction accuracy and coding efficiency by adaptively selecting motion vectors of different precisions.

[0090] It should be noted that, in addition to Intra, Merge, AMVP and AMVR, the prediction mode may also include other modes, such as Skip Mode (ie, an inter-frame skip mode), which is not limited in the embodiments of the present disclosure.

[0091] In the embodiments of the present disclosure, the splitting method refers to a method of splitting a video frame or coding block into smaller, easier-to-process parts in order to more effectively organize and manage video data during the video encoding process. The splitting method includes one or more of a quadtree splitting method (QT Split), a ternary tree splitting method (TT Split), and a binary tree splitting method (BT Split).

[0092] QT Split is a recursive partitioning method, which means that the current coding block will be divided into 4 sub-blocks of the same size. TT Split is similar to the quadtree partitioning method, but each node has at most three child nodes. In BT Split, each node has at most two child nodes, which are divided into a left subtree and a right subtree.

[0093] It should be noted that, in addition to QT Split, TT Split and BT Split, the division method may also include other methods, such as 4-partition division, 3-partition division, etc., which is not limited in the embodiments of the present disclosure.

[0094] In the disclosed embodiment, after the prediction mode and the division method are obtained, these prediction modes and the division method are added to the coding mode pool, so as to utilize these prediction modes and the division method to constitute multiple coding modes in the coding mode pool, and subsequently for the coding unit, the multiple coding modes are prioritized to obtain the coding mode order corresponding to the coding unit, and then the multiple coding modes in the coding mode pool are traversed in this order to obtain the target coding mode.

[0095] In the video processing method provided by the embodiment of the present disclosure, the coding mode pool includes prediction modes (such as Intra, Merge and AMVP) and division methods (such as QT Split, TT Split and BT Split). These prediction modes and division methods are uniformly placed in the coding mode pool to form multiple coding modes, and multiple coding modes are supported to be arranged in any order, so that the coding process can more flexibly adjust the priorities of multiple coding modes according to the characteristic information of the coding unit, thereby supporting top-down and bottom-up bidirectional division searches.

[0096] In some embodiments of the present disclosure, the video processing method also includes: splitting the prediction mode to obtain a first prediction sub-mode and a second prediction sub-mode; the first prediction sub-mode is a coarse-grained search mode including a prediction process, and the second prediction sub-mode is a fine-grained search mode including a transform quantization reconstruction process.

[0097] In the disclosed embodiment, for each prediction mode, the prediction mode is split into a first prediction sub-mode and a second prediction sub-mode.

[0098] The first prediction sub-mode is a coarse-grained search mode with only the prediction process, which can also be called the coarse-grained search mode. This mode focuses on the preliminary screening and rapid positioning of the prediction process. The second prediction sub-mode is a fine-grained search mode that includes transformation, quantization and reconstruction, which can also be called the fine-grained search mode. This mode involves more complex operations such as transformation, quantization and reconstruction to more accurately evaluate the performance of the prediction mode.

[0099] In the video processing method provided by the embodiment of the present disclosure, the prediction mode is further divided into a coarse-grained search mode and a fine-grained search mode, thereby enhancing the flexibility and scalability of the mode search architecture.

[0100] In some embodiments of the present disclosure, priority decisions are made on multiple coding modes in a coding mode pool based on feature information of a coding unit to obtain a coding mode order corresponding to the coding unit, including: inputting the feature information of the coding unit into a pre-trained mode decision model, and outputting the coding mode order corresponding to the coding unit; the pre-trained mode decision model includes a fully connected network.

[0101] In the disclosed embodiment, the pre-trained mode decision model includes a fully connected network. The fully connected network is composed of multiple layers of neurons, and each layer of neurons is connected to all neurons in the previous layer. The input layer receives the feature information of the coding unit, and the output layer outputs the predicted coding mode sequence. Between the input layer and the output layer, multiple hidden layers may be included for extracting and integrating feature information.

[0102] In a possible implementation, the specific implementation of obtaining the coding mode sequence corresponding to the coding unit based on the pre-trained mode decision model is as follows:

[0103] (1) The feature information of the coding unit is input into a pre-trained mode decision model, wherein the feature information of the coding unit includes pre-analysis feature information, feature information of adjacent coding units, feature information of the parent block, and quantization parameter information, and the feature information is usually represented in the form of a vector.

[0104] (2) The pre-trained pattern decision model extracts and transforms the input feature information. This process may include convolution operations, pooling operations, etc. to extract more abstract and useful features.

[0105] (3) After feature extraction and transformation, the pre-trained pattern decision model will apply nonlinear activation functions (such as ReLU, Sigmoid, etc.) to introduce nonlinear factors, so that the model can process more complex input data.

[0106] (4) The feature information will pass through one or more fully connected layers. In the fully connected layer, each neuron is connected to all neurons in the previous layer, and a new feature vector is output through weighted summation and activation function processing.

[0107] (5) The feature information reaches the output layer. In the output layer, the pre-trained mode decision model calculates a prediction vector based on the input feature information and the current parameter settings. Each element in this prediction vector corresponds to a mode in the coding mode pool, and the value of the element (such as probability, score, etc.) reflects the priority of the mode when processing the coding unit.

[0108] (6) Based on the prediction vector obtained from the output layer, the pre-trained mode decision model will sort or select it to obtain the coding mode order corresponding to the coding unit.

[0109] Specifically, the elements in the prediction vector are sorted according to the numerical size to obtain an ordered list. The elements in this list are arranged in order from high to low, reflecting the priority of different coding modes. According to the sorted list, the coding modes corresponding to the first few (such as the first N) elements with the highest numerical values ​​are selected as the predicted coding mode order. This number N can be set according to actual needs to balance coding efficiency and video quality.

[0110] In the video processing method provided by the embodiment of the present disclosure, the priority order of the coding mode can be quickly determined according to the characteristic information of the coding unit through the pre-trained mode decision model, and the pre-trained mode decision model can be integrated into the encoder as an independent module to facilitate updating or expanding the model to adapt to new coding standards or requirements.

[0111] In the disclosed embodiment, the process of obtaining the coding mode sequence corresponding to the coding unit is based on a pre-trained mode decision model. In the model training stage, a large amount of training data and optimization algorithms are needed to continuously adjust and optimize the parameters and structure of the model to ensure that the model can accurately predict the coding mode sequence. Figure 4 FIG. 1 is a schematic diagram showing the training process of the pre-trained mode decision model of the embodiment of the present disclosure. Figure 4 , follow the steps below to train and obtain the pre-trained mode decision model.

[0112] Step S410, determining an initial mode decision model.

[0113] In this step, a suitable model is selected as the initial mode decision model. This model can be a fully connected network, a neural network, such as a convolutional neural network, a recurrent neural network, a long short-term memory network, etc., or a decision tree, a random forest, a support vector machine, etc. Of course, in the embodiment of the present disclosure, a suitable model can be selected according to actual needs, data characteristics, and computing resources.

[0114] Step S420: Obtain coding unit training samples, feature information of the coding unit training samples, and a coding mode sequence corresponding to the coding unit training samples.

[0115] Among them, the coding unit training sample refers to the video data block used for training, and the feature information refers to the data extracted from the coding unit that can reflect its characteristics, such as pre-analysis feature information, feature information of adjacent coding units, feature information of parent blocks, and quantization parameter information. The coding mode order refers to the order of the coding modes actually used for each coding unit, which is the target that the model needs to predict.

[0116] Step S430: input the feature information of the coding unit training samples into the initial mode decision model, and output the predicted coding mode sequence corresponding to the coding unit training samples.

[0117] In this step, the feature information of the coding unit is input into the initial mode decision model, and the model outputs the predicted coding mode sequence based on the input feature information. This is the forward propagation process of the model.

[0118] Step S440, calculating the model loss according to the coding mode sequence corresponding to the coding unit training samples and the prediction coding mode sequence corresponding to the coding unit training samples.

[0119] In this step, the model loss is calculated, that is, the difference between the predicted coding mode sequence and the actual coding mode sequence. Among them, the loss function includes mean square error, cross entropy loss, etc. Of course, in the embodiment of the present disclosure, the loss function can be selected according to the specific task and goal.

[0120] Step S450, adjusting the model parameters of the initial mode decision model according to the model loss to obtain a pre-trained mode decision model.

[0121] In this step, optimization algorithms such as gradient descent, stochastic gradient descent, etc. are used to update the model parameters according to the calculated loss. This process is usually called back propagation and parameter update. Through multiple iterations of this process, the model gradually learns how to predict the correct coding pattern sequence from the input feature information. When the performance of the model no longer improves, or when the preset number of iterations is reached, the training process ends and a pre-trained mode decision model is obtained.

[0122] In the video processing method provided by the embodiment of the present disclosure, the pre-trained mode decision model can capture the complex relationship between the feature information of the coding unit and the optimal coding mode by learning from a large amount of training data, so that the model can more accurately predict the coding mode sequence suitable for the current coding unit; and, because the mode decision model is trained based on a large amount of training data, it has a strong generalization ability, that is, the model can process different types of coding units and output a reasonable coding mode sequence, so as to adapt to different video content and encoding requirements.

[0123] In step S330, based on the coding mode sequence corresponding to the coding unit, a mode search is performed on multiple coding modes to obtain a target coding mode of the coding unit.

[0124] In this step, based on the coding mode order corresponding to the coding unit, multiple coding modes are traversed to obtain the target coding mode of the coding unit. Since it is performed in descending order of priority of the coding modes, it can reduce the search time and calculation amount while ensuring the coding effect, thereby improving the speed of the encoder.

[0125] In some embodiments of the present disclosure, based on the coding mode sequence corresponding to the coding unit, a pattern search is performed on multiple coding modes to obtain a target coding mode of the coding unit, including: selecting a current coding mode based on the coding mode sequence corresponding to the coding unit; encoding the coding unit based on the current coding mode to obtain a coding quality indicator corresponding to the current coding mode; selecting the next coding mode of the current coding mode based on the coding mode sequence corresponding to the coding unit, determining the next coding mode as a new current coding mode, and encoding based on the new current coding mode until the coding process of multiple coding modes is completed; obtaining coding quality indicators corresponding to multiple coding modes, and obtaining the target coding mode of the coding unit according to the coding quality indicators corresponding to the multiple coding modes.

[0126] In the disclosed embodiment, based on the coding mode sequence corresponding to the coding unit, that is, in the order of the priority of the coding mode from high to low, the coding mode with the highest priority is selected as the current coding mode, and the coding unit is encoded using the current coding mode to obtain a coding quality indicator corresponding to the current coding mode, such as a rate-distortion value. Based on the coding mode sequence corresponding to the coding unit, the next coding mode of the current coding mode is selected as the new current coding mode, and the coding unit is encoded using the new current coding mode to obtain a coding quality indicator corresponding to the new current coding mode. After traversing all coding modes and obtaining corresponding coding quality indicators, the target coding mode of the coding unit is determined based on these coding quality indicators. For example, the coding quality indicator is a rate-distortion value, and the minimum value among the rate-distortion values ​​of the coding modes is obtained, and the coding mode corresponding to the minimum value is determined as the target coding mode.

[0127] In some embodiments of the present disclosure, encoding processing is performed on a coding unit based on a current encoding mode, including: determining whether to trigger the current encoding mode based on encoding information of the encoding mode of the encoding unit that has completed encoding processing; if it is determined that the current encoding mode is triggered, encoding processing is performed on the coding unit based on the current encoding mode to obtain an encoding quality indicator corresponding to the current encoding mode; if it is determined that the current encoding mode is not triggered, determining to skip the current encoding mode.

[0128] In the disclosed embodiment, the coding information includes one or more of rate-distortion information, motion information of the inter-frame mode, and reference information. Among them, rate-distortion information refers to an indicator that measures the trade-off between the quality of the video after encoding and the bit rate required for encoding. The motion information of the inter-frame mode means that if the coding unit adopts inter-frame prediction (i.e., prediction using information of adjacent frames), motion information will be generated to describe the position change of the coding unit between adjacent frames. Reference information is used to indicate which other coding units or frames the coding unit refers to during the encoding process.

[0129] In the disclosed embodiment, during the video encoding process, after the encoding unit completes the encoding process based on the encoding mode, it will generate the encoding information of the encoding mode. When the encoding unit is encoded based on the current encoding mode, the encoding information of the encoding mode that has completed the encoding process can be used to determine whether to trigger the current encoding mode. If it is determined to be triggered, the encoding unit is encoded based on the current encoding mode, and if it is determined not to be triggered, the current encoding mode is skipped.

[0130] For example, if the expected rate-distortion performance of the current coding mode (obtained based on some prediction or estimation) is not as good as the rate-distortion performance of the coding mode that has completed the coding process, it may be decided to skip the current coding mode. For another example, if there is a significant inconsistency or redundancy in motion information or reference information between the current coding mode and the coding mode that has completed the coding process, it may cause the current coding mode to be skipped.

[0131] Figure 5 The flowchart of the embodiment of the present disclosure is shown, which performs a mode search based on the coding mode sequence corresponding to the coding unit to obtain the target coding mode. Figure 5 , follow the steps below to perform pattern search to obtain the target encoding pattern.

[0132] Step S501 : based on the coding mode sequence corresponding to the coding unit, select a coding mode with the highest priority among multiple coding modes, and determine the coding mode with the highest priority as the current coding mode.

[0133] Step S502, judging whether the current coding mode is triggered according to the coding information of the coding mode that has completed the coding process, if so, executing step S503, if not, executing step S504.

[0134] Step S503: perform encoding processing on the encoding unit based on the current encoding mode to obtain an encoding quality indicator corresponding to the current encoding mode.

[0135] Step S504: skip the current encoding mode.

[0136] Step S505, determining whether all encoding modes in the plurality of encoding modes have been traversed, if not, executing step S506, and if so, executing step S507.

[0137] Step S506: based on the coding mode sequence corresponding to the coding unit, select the next coding mode, and determine the next coding mode as the current coding mode.

[0138] Step S507: Analyze the coding quality indicators of all coding modes that have completed the coding process to obtain the target coding mode of the coding unit.

[0139] The coding mode that has completed the coding process in step S502 refers to the coding mode that has completed the coding process before the current coding mode. When traversing the current coding mode, it is determined whether to trigger the coding process of the current coding mode according to the coding information of the coding mode that has completed the coding process before the current coding mode, that is, it is determined whether to skip the current coding mode. Since the coding mode sequence is in descending order of priority, the information extracted from the coding mode with a high priority that is traversed first can assist in determining whether the coding mode with a low priority can be skipped.

[0140] Furthermore, since there is a skipped coding mode among the multiple coding modes, in step S507, the coding quality indicators of all the coding modes that have completed the coding process are analyzed, and finally the target coding mode of the coding unit is obtained. The coding quality indicator may include a rate-distortion value, and the rate-distortion values ​​of all the coding modes that have completed the coding process are analyzed, and the minimum value is selected, and the coding mode corresponding to the minimum value is determined as the target coding mode of the coding unit.

[0141] In the video processing method implemented in the present invention, a mode search is performed based on the coding mode sequence corresponding to the coding unit, so that the coding mode can be searched in an orderly manner, meaningless random attempts are avoided, and the waste of computing resources is reduced; by considering multiple coding modes and selecting based on their coding quality indicators, a coding mode that achieves the best balance between coding quality and compression efficiency, that is, a target coding mode, can be found; when traversing the current coding mode, the coding information of the coding mode that has completed the coding process is used to guide whether to trigger the current coding mode, which can make full use of high-dimensional information, improve the accuracy of decision-making, further reduce unnecessary coding attempts, reduce search time and calculation amount, and improve the speed of the encoder.

[0142] To facilitate understanding, a specific embodiment is provided below to illustrate the video processing method of the embodiment of the present disclosure. Figure 6 A flowchart of another video processing method according to an embodiment of the present disclosure is shown. Figure 6 A specific implementation process of performing mode search on a coding unit to determine the optimal coding mode of the coding unit in an embodiment of the present disclosure is provided, referring to Figure 6 , may include the following steps.

[0143] Step S601: for a coding unit in a video image, feature extraction is performed on the coding unit to obtain feature information of the coding unit, such as pre-analysis feature information, feature information of adjacent coding units, feature information of a parent block, and quantization parameter information.

[0144] Step S602: input the feature information of the coding unit into a pre-trained mode decision model, perform mode priority decision on multiple coding modes in the coding mode pool, and obtain the coding mode order corresponding to the coding unit.

[0145] Among them, the coding mode pool includes different prediction modes and division methods, such as Intra, Merge, AMVP, AMVR, QT Split, TT Split and BT Split.

[0146] Step S603: traverse multiple coding modes in the coding mode pool according to the coding mode sequence corresponding to the coding unit to obtain the current coding mode.

[0147] Step S604: for the current coding mode, determine whether to trigger the current coding mode. Specifically, determine based on the coding information of the coding mode that has completed the coding process.

[0148] Step S605: If it is determined that the current coding mode is triggered, the coding unit is coded based on the current coding mode to obtain a coding quality indicator corresponding to the current coding mode, and the current coding mode is ended.

[0149] Step S606: If it is determined that the current encoding mode is not triggered, then the current encoding mode is ended.

[0150] Step S607, determine whether to traverse all coding modes. If all coding modes are traversed, end the mode traversal and execute step S608. Otherwise, return to step S603 and select the next coding mode as the new current coding mode according to the coding mode sequence corresponding to the coding unit.

[0151] Step S608: Compare the encoding quality indicators corresponding to all the encoding modes that have completed the encoding process to obtain the optimal encoding mode, that is, the target encoding mode.

[0152] according to Figure 6 The illustrated embodiment shows that the video processing method abstracts different prediction modes and division methods into search tasks for coding units, and puts them in a coding mode pool for unified management, supporting the arbitrary arrangement of the search order of all modes in the coding mode pool; when the coding unit needs to perform a mode search, it will adaptively select the most likely mode from the coding mode pool for encoding based on various low-dimensional (i.e., characteristic information of the coding unit) and high-dimensional information (i.e., encoding information of the coding mode that has completed the encoding process), and the encoded high-dimensional information can be further used for the selection and triggering of the next mode.

[0153] Figure 7 A flowchart of another video processing method according to an embodiment of the present disclosure is shown. Figure 7Provided is a specific implementation process of first determining the division method and then performing the prediction mode search during the mode search of the coding unit in the embodiment, referring to Figure 7 , may include the following steps.

[0154] Step S701: obtaining a coding unit in a video image.

[0155] Step S702, determine whether it is divisible.

[0156] Step S703: If it is determined that the coding unit can be divided, the coding unit is divided to obtain multiple sub-coding units, and the process returns to step S702 to further determine whether the sub-coding units obtained by the division can be divided.

[0157] Step S704: If it is determined that the coding unit cannot be divided, a divided coding unit is obtained, wherein the divided coding unit includes a sub-coding unit.

[0158] Step S705: extract feature information from the sub-coding unit to obtain feature information of the sub-coding unit, wherein the feature information of the sub-coding unit is high-dimensional information, which may include sub-block division depth, sub-block mode ratio, sub-block reference frame and sub-block motion information distribution.

[0159] Step S706: Based on the characteristic information of the sub-coding unit, a mode skipping decision is made for the coding unit. Specifically, based on the characteristic information of the sub-coding unit, it is evaluated that some coding modes may not be optimal, and these coding modes are skipped.

[0160] Step S707: perform a prediction mode search on the coding unit to obtain an optimal prediction mode for predicting the coding unit.

[0161] Figure 7 The embodiment shown provides that when performing a pattern search, the division method can be performed first and then the prediction mode search can be performed. Specifically, when searching for a coding unit, the coding mode order corresponding to the coding unit can be determined first, wherein the priority of the division method in the coding mode order corresponding to the coding unit is higher than the priority of the prediction mode, so the division method is performed first and then the prediction mode search is performed. In this way, it can support bottom-up division search, that is, first search large blocks and then search small blocks, so that the coding unit can use the high-dimensional information of its sub-blocks during the search to improve the encoding speed; and it can support upward or downward division search starting from any division level, so that the division level for starting the search can be adaptively selected according to the characteristics of the video content.

[0162] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0163] Figure 8 FIG. 1 is a block diagram of a video processing device according to an embodiment of the present disclosure. Figure 8 As shown, the device 800 includes a feature extraction module 810 , a priority decision module 820 and a pattern search module 830 .

[0164] The feature extraction module 810 is configured to extract features from the coding unit in the video to be processed to obtain feature information of the coding unit. The priority decision module 820 is configured to make a priority decision on multiple coding modes in the coding mode pool according to the feature information of the coding unit to obtain the coding mode sequence corresponding to the coding unit; the coding mode sequence is the sequence of the priority of the coding mode from high to low. The mode search module 830 is configured to perform a mode search on multiple coding modes based on the coding mode sequence corresponding to the coding unit to obtain the target coding mode of the coding unit.

[0165] In some embodiments of the present disclosure, the pattern search module 830 is further configured to: select a current coding mode based on the coding mode sequence corresponding to the coding unit; perform encoding processing on the coding unit based on the current coding mode to obtain a coding quality indicator corresponding to the current coding mode; select the next coding mode of the current coding mode based on the coding mode sequence corresponding to the coding unit, determine the next coding mode as the new current coding mode, and perform encoding processing based on the new current coding mode until multiple coding modes complete encoding processing; obtain coding quality indicators corresponding to multiple coding modes, and obtain the target coding mode of the coding unit according to the coding quality indicators corresponding to the multiple coding modes.

[0166] In some embodiments of the present disclosure, the pattern search module 830 is further configured to: determine whether to trigger the current coding mode based on the coding information of the coding mode of the coding unit that has completed the coding process; if it is determined that the current coding mode is triggered, encode the coding unit based on the current coding mode to obtain a coding quality indicator corresponding to the current coding mode; if it is determined that the current coding mode is not triggered, determine to skip the current coding mode.

[0167] In some embodiments of the present disclosure, the priority decision module 820 is configured to: input feature information of the coding unit into a pre-trained mode decision model, and output the coding mode sequence corresponding to the coding unit; the pre-trained mode decision model includes a fully connected network.

[0168] In some embodiments of the present disclosure, the device 800 also includes a model training module 840, which is configured to: determine an initial mode decision model; obtain coding unit training samples, feature information of the coding unit training samples, and a coding mode sequence corresponding to the coding unit training samples; input the feature information of the coding unit training samples into the initial mode decision model, and output a predicted coding mode sequence corresponding to the coding unit training samples; calculate the model loss based on the coding mode sequence corresponding to the coding unit training samples and the predicted coding mode sequence corresponding to the coding unit training samples; adjust the model parameters of the initial mode decision model based on the model loss to obtain a pre-trained mode decision model.

[0169] In some embodiments of the present disclosure, the device 800 also includes a mode acquisition module 850, which is configured to: obtain a prediction mode and a division method; the prediction mode includes one or more of an intra-frame prediction mode, an inter-frame fusion prediction mode, an advanced motion vector prediction mode, and an adaptive motion vector precision prediction mode, and the division method includes one or more of a quadtree division method, a ternary tree division method, and a binary tree division method; and add the prediction mode and the division method to the coding mode pool.

[0170] In some embodiments of the present disclosure, the pattern acquisition module 850 is also configured to: split the prediction pattern to obtain a first prediction sub-pattern and a second prediction sub-pattern; the first prediction sub-pattern is a coarse-grained search mode including a prediction process, and the second prediction sub-pattern is a fine-grained search mode including a transform quantization reconstruction process.

[0171] In some embodiments of the present disclosure, the feature information of a coding unit includes one or more of pre-analysis feature information, feature information of adjacent coding units, feature information of a parent block, and quantization parameter information; wherein the pre-analysis feature information includes one or more of intra-frame loss, inter-frame loss, and texture complexity obtained in the pre-analysis stage; the feature information of adjacent coding units includes one or more of the division depth of adjacent coding units, the coding mode of adjacent coding units, the rate-distortion value of adjacent coding units, the motion vector of adjacent coding units, and the reference image of adjacent coding units; the feature information of a parent block includes one or more of the optimal coding mode of the parent block and the rate-distortion values ​​of each coding mode of the parent block.

[0172] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0173] Fig. 9 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure. It should be noted that: Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0174] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0175] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 3 Follow the steps shown in .

[0176] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0177] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0178] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0179] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 940. As shown, the network adapter 940 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0180] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0181] The program product for implementing the above method according to the embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0182] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0183] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0184] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0185] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0186] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0187] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0188] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0190] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: The method comprises: Extract features from coding units in the video to be processed to obtain feature information of the coding units; According to the characteristic information of the coding unit, priority decisions are made on multiple coding modes in the coding mode pool to obtain a coding mode sequence corresponding to the coding unit; the coding mode sequence is a sequence of coding modes from high to low priority; Based on the coding mode sequence corresponding to the coding unit, a mode search is performed on the multiple coding modes to obtain a target coding mode of the coding unit.

2. The method according to claim 1, characterized in that The performing a mode search on the multiple coding modes based on the coding mode sequence corresponding to the coding unit to obtain a target coding mode of the coding unit includes: Selecting a current encoding mode based on the encoding mode sequence corresponding to the encoding unit; Performing encoding processing on the encoding unit based on the current encoding mode to obtain an encoding quality indicator corresponding to the current encoding mode; Based on the coding mode sequence corresponding to the coding unit, selecting a next coding mode of the current coding mode, determining the next coding mode as a new current coding mode, and performing coding processing based on the new current coding mode until the coding processing of the multiple coding modes is completed; Obtain encoding quality indicators corresponding to the multiple encoding modes, and obtain a target encoding mode of the encoding unit according to the encoding quality indicators corresponding to the multiple encoding modes.

3. The method according to claim 2, characterized in that The performing encoding processing on the encoding unit based on the current encoding mode includes: Determining whether to trigger the current encoding mode according to the encoding information of the encoding mode of the encoding unit that has completed the encoding process; If it is determined that the current encoding mode is triggered, encoding processing is performed on the encoding unit based on the current encoding mode to obtain an encoding quality indicator corresponding to the current encoding mode; If it is determined that the current encoding mode is not triggered, it is determined to skip the current encoding mode.

4. The method according to claim 1, characterized in that: The step of making a priority decision on a plurality of coding modes in a coding mode pool according to the characteristic information of the coding unit to obtain a coding mode sequence corresponding to the coding unit includes: The feature information of the coding unit is input into a pre-trained mode decision model, and the coding mode sequence corresponding to the coding unit is output; the pre-trained mode decision model includes a fully connected network.

5. The method according to claim 4, characterized in that The pre-trained mode decision model is obtained by training according to the following method: Determine the initial mode decision model; Obtaining a coding unit training sample, feature information of the coding unit training sample, and a coding mode sequence corresponding to the coding unit training sample; Inputting feature information of the coding unit training samples into the initial mode decision model, and outputting a predicted coding mode sequence corresponding to the coding unit training samples; Calculating a model loss according to a coding mode sequence corresponding to the coding unit training samples and a predicted coding mode sequence corresponding to the coding unit training samples; The model parameters of the initial mode decision model are adjusted according to the model loss to obtain the pre-trained mode decision model.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a prediction mode and a partitioning method; the prediction mode includes one or more of an intra-frame prediction mode, an inter-frame fusion prediction mode, an advanced motion vector prediction mode, and an adaptive motion vector precision prediction mode; the partitioning method includes one or more of a quadtree partitioning method, a ternary tree partitioning method, and a binary tree partitioning method; The prediction mode and the partitioning method are added to the coding mode pool.

7. The method according to claim 6, characterized in that The method further comprises: The prediction mode is split to obtain a first prediction sub-mode and a second prediction sub-mode; the first prediction sub-mode is a coarse-grained search mode including a prediction process, and the second prediction sub-mode is a fine-grained search mode including a transform quantization reconstruction process.

8. The method according to any one of claims 1 to 7, characterized in that: The characteristic information of the coding unit includes one or more of pre-analysis characteristic information, characteristic information of adjacent coding units, characteristic information of parent blocks and quantization parameter information; wherein, The pre-analysis feature information includes one or more of the intra-frame loss, inter-frame loss and texture complexity obtained in the pre-analysis stage; the feature information of the adjacent coding unit includes one or more of the division depth of the adjacent coding unit, the coding mode of the adjacent coding unit, the rate-distortion value of the adjacent coding unit, the motion vector of the adjacent coding unit and the reference image of the adjacent coding unit; the feature information of the parent block includes one or more of the optimal coding mode of the parent block and the rate-distortion values ​​of each coding mode of the parent block.

9. A video processing device, characterized in that: The device comprises: A feature extraction module is configured to extract features from coding units in the video to be processed to obtain feature information of the coding units; A priority decision module is configured to make a priority decision on a plurality of coding modes in a coding mode pool according to the characteristic information of the coding unit, and obtain a coding mode sequence corresponding to the coding unit; the coding mode sequence is a sequence of coding modes from high priority to low priority; The mode search module is configured to perform a mode search on the multiple coding modes based on the coding mode sequence corresponding to the coding unit to obtain a target coding mode of the coding unit.

10. An electronic device, characterized in that: include: processor; A memory for storing the processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the video processing method according to any one of claims 1 to 8. 11 . A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the video processing method according to any one of claims 1 to 8 is implemented.

Images