Coding mode determination method and device
By obtaining the encoding characteristics of video data and training machine learning models, the problem of insufficient feature extraction complexity and universality in video encoding technology is solved, and high-versatility and low-complexity coding mode decisions are achieved.
Patent Information
- Application Number
- CN202010134314.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-03-02
AI Technical Summary
In the existing video encoding technology, the statistical-based model decision-making method has poor generality, while the machine learning-based method has high complexity and is difficult to take into account both high generality and low complexity.
By obtaining the encoding characteristics of video data, selecting the encoding mode based on preset criteria, and combining the features and patterns into training data, the machine learning model is trained, the feature dimension is reduced, and the model universality and efficiency are improved.
While ensuring the accuracy of coding mode decision-making, the dimension of machine learning features is reduced, and high versatility and low complexity are provided, which solves the problem of insufficient feature extraction complexity and universality in the prior art.
Smart Images

Figure CN113347415B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video coding technology, and in particular to a method and device for determining a coding mode. Background Art
[0002] In video coding technology, in order to avoid too high a bit rate after coding, the video data is usually compressed and coded. In this way, after coding, the video data will be lost, that is, the video data before coding is not exactly the same as the video data restored after coding and decoding. In this way, after coding and decoding, the video data is distorted. The smaller the bit rate after coding, the greater the distortion after decoding, and vice versa. Therefore, a problem in video coding is how to balance the bit rate and distortion after coding.
[0003] Currently, in video coding technology, options are usually set for multiple aspects of video coding, for example, macroblock type, inter-frame prediction direction, motion vector and other options are set. Each of these options has multiple possible choices, and different choices correspond to different coding modes.
[0004] In different encoding modes, the bit rate after encoding and the distortion after decoding are also different.
[0005] In order to select the best coding mode from a variety of different coding modes, that is, to make a coding mode decision, for example, a rate-distortion optimization criterion is adopted. Under the rate-distortion optimization criterion, the rate-distortion cost is used to indicate the relationship between the bit rate and distortion after encoding, so that the coding mode with the most optimized rate-distortion cost is determined as the best coding mode.
[0006] The existing mode decision methods include statistical mode decision methods and machine learning-based mode decision methods. In the statistical mode decision method, the previous encoding results are usually counted and the mode decision is made based on the statistical results. The machine learning-based mode decision methods include traditional machine learning-based methods and deep learning-based methods.
[0007] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present invention and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology section of the present invention. Summary of the invention
[0008] The inventors have found that in existing video coding technologies, although the statistical-based pattern decision method is simple and has low complexity, it has poor versatility and the accuracy of the decision results cannot be guaranteed because the statistical objects usually come from limited data information. In the pattern decision method based on machine learning, if the features on which the machine learning is based are artificially designed (traditional machine learning methods), in order to maintain a high degree of versatility, it is necessary to select high-dimensional features (i.e., features composed of multiple types of information), but the selection of high-dimensional features will also lead to higher training complexity and feature extraction complexity. If the features on which the machine learning is based are automatically selected by the machine (methods based on deep learning), the complexity of the learning method is usually higher.
[0009] In order to solve the above problems or similar problems, an embodiment of the present invention provides a method and device for determining a coding mode, which can have both high versatility and low complexity.
[0010] According to a first aspect of an embodiment of the present invention, a method for determining a coding mode is provided, wherein the method comprises: acquiring video data and a first feature of the coded data obtained by encoding the video data using a first coding mode, the first feature being related to the first coding mode used to encode the video data; selecting a second coding mode based on the video data based on a preset criterion; combining the first feature and the second coding mode as training data to train a machine learning model to obtain a first model; and processing a third feature of the coded data obtained by encoding using a third coding mode based on the first model to determine a fourth coding mode, the third feature being related to the third coding mode.
[0011] According to a second aspect of an embodiment of the present invention, a coding mode determination device is provided, wherein the device includes: an acquisition unit, which acquires video data and a first feature of the coded data obtained by encoding the video data using a first coding mode, the first feature being related to the first coding mode used to encode the video data; a first processing unit, which selects a second coding mode based on the video data and based on preset criteria; and a second processing unit, which combines the first feature and the second coding mode as training data to train a machine learning model to obtain a first model, and processes a third feature of the coded data obtained by encoding using a third coding mode based on the first model to determine a fourth coding mode, the third feature being related to the third coding mode.
[0012] One beneficial effect of an embodiment of the present invention is that by using a first feature related to a first encoding mode selected when encoding video data (i.e., the feature has a high correlation with the optimal encoding mode determined during encoding) as at least a part of the object of machine learning, a first model for processing a third feature of encoded data encoded using a third encoding mode is obtained. In this way, the dimension of the feature on which the machine learning is based can be reduced (i.e., the type of information involved in the feature is reduced) while ensuring the same accuracy of the coding mode decision, thereby providing both high versatility and low complexity.
[0013] With reference to the following description and accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the manner in which the principles of the present invention can be adopted. It should be understood that the embodiments of the present invention are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of the present invention include many changes, modifications and equivalents.
[0014] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.
[0015] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the implementation of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0017] Figure 1 It is a schematic diagram of the coding mode determination method of embodiment 1 of the present invention.
[0018] Figure 2 It is a schematic diagram of the application environment of the coding mode determination method of embodiment 1 of the present invention.
[0019] Figure 3 It is a schematic diagram of step 101 of embodiment 1 of the present invention.
[0020] Figure 4 It is a schematic diagram of step 103 of embodiment 1 of the present invention.
[0021] Figure 5 It is a schematic diagram of a coding mode determination device according to embodiment 2 of the present invention.
[0022] Figure 6 It is a schematic diagram of the acquisition unit 501 of Embodiment 2 of the present invention.
[0023] Figure 7 It is a schematic diagram of the first processing unit 502 of Embodiment 2 of the present invention. DETAILED DESCRIPTION
[0024] With reference to the accompanying drawings, the above and other features of the present invention will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present invention are specifically disclosed, which show some embodiments in which the principles of the present invention can be adopted, and it should be understood that the present invention is not limited to the described embodiments, on the contrary, the present invention includes all modifications, variations and equivalents falling within the scope of the appended claims.
[0025] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0026] In the embodiments of the present invention, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or time sequence of these elements, etc. These elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms.
[0027] In the embodiments of the present invention, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.
[0028] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent to" versus "directly adjacent to," etc.).
[0029] It should also be noted that, in some alternative implementations, the functions / actions mentioned may occur in a different order than that indicated in the accompanying drawings. For example, depending on the functions / actions involved, two figures shown in succession may actually be performed substantially simultaneously or may sometimes be performed in the reverse order.
[0030] Various implementations of the embodiments of the present invention are described below in conjunction with the accompanying drawings. These implementations are only exemplary and are not intended to limit the embodiments of the present invention.
[0031] Example 1
[0032] This embodiment 1 provides a method for determining a coding mode.
[0033] Figure 1 FIG. 1 is a schematic diagram of the coding mode determination method of this embodiment. Figure 2 As shown, the method includes:
[0034] Step 101, obtaining video data and a first feature of encoded data obtained by encoding the video data using a first encoding mode, where the first feature is related to the first encoding mode used to encode the video data;
[0035] Step 103, selecting a second encoding mode based on the video data and a preset criterion;
[0036] Step 105, combining the first feature and the second encoding mode as training data to train a machine learning model to obtain a first model;
[0037] Step 107: Processing a third feature of the encoded data obtained by encoding using the third encoding mode based on the first model to determine a fourth encoding mode, wherein the third feature is related to the third encoding mode.
[0038] Figure 2 FIG. 1 is a schematic diagram of the application environment of the method of this embodiment. Figure 2As shown, the video data 201 is composed of multiple data frames 201-1, 201-2, ..., 201-n. The video data 201 is input to the encoder 202 for video encoding. In the video encoding process, the multiple data frames are encoded respectively. When encoding each data frame, the data frame can be divided into macroblocks, and encoded based on the macroblocks in a specific encoding mode. The above method of this embodiment can be used to determine the encoding mode adopted by the encoder 202.
[0039] In this embodiment, the first feature is a feature extracted from the encoded data obtained by encoding using the first encoding mode, and the first feature is related to the first encoding mode selected during encoding. In other words, the first feature is correlated with the optimal encoding mode determined during encoding. The optimal encoding mode can be selected based on a preset criterion. For example, the preset criterion can be a rate-distortion cost criterion.
[0040] In the above embodiment, a first feature related to the first encoding mode selected when encoding the video data is used as at least a part of the object of machine learning (that is, the feature has a high correlation with the optimal encoding mode determined during encoding), thereby obtaining a first model for processing the third feature of the encoded data encoded using the third encoding mode. In this way, the dimension of the feature on which the machine learning is based can be reduced (that is, the type of information involved in the feature is reduced) while ensuring the same coding mode decision accuracy, thereby providing both high versatility and low complexity.
[0041] Figure 3 FIG. 1 is a schematic diagram of an implementation method of step 101 of this embodiment. Figure 3 As shown, the above step 101 may include:
[0042] Step 301, obtaining video data;
[0043] Step 303, encoding the video data by an encoder to obtain encoded data; and
[0044] Step 305: decode the encoded data obtained by the encoder through a decoder to obtain a first feature of the encoded data.
[0045] However, this embodiment is not limited to the above implementation manner, and the first feature may be directly obtained from the encoder without using a decoder.
[0046] In this embodiment, the encoder may be a hardware encoder, a software encoder, or a combination of the two. The decoder may be a hardware decoder, a software decoder, or a combination of the two.
[0047] In one embodiment, the encoder is a hardware encoder, and the decoder is a software encoder. In step 303, the video data is hardware-encoded by the hardware encoder to obtain coded data. In step 305, the coded data obtained by the hardware encoder is software-decoded by the software decoder to obtain the first feature of the coded data. In this way, a higher encoding speed can be obtained, and the hardware encoding and software decoding can be performed asynchronously. Therefore, the increase in complexity of the method brought by the encoding and decoding steps 303 and 305 is very low. However, the present embodiment is not limited to this.
[0048] In this embodiment, the first feature may include mode information of the current coding block. The coding block may be, for example, a macroblock contained in a video data frame. The mode information of the coding block may be, for example, macroblock type information in an H.264 coding block, for example, 16×16, 8×16, etc.; the mode information of the coding block may also be, for example, inter-frame prediction direction information of CTU, PU, TU, B frame in an H.265 coding block, for example, L0, L1, bidirectional, etc.; the mode information of the coding block may also be, for example, motion vector estimation accuracy information in an H.264 coding block, for example, integer accuracy, 1 / 4 accuracy, etc.
[0049] The first feature may include only the above-mentioned mode information, or may further include an intermediate result generated by an encoder for encoding video data using the first encoding mode, for example, a coding block flag, a coding quantization coefficient, the number of bits consumed by the coding block, or other intermediate results generated by the encoder.
[0050] Figure 4 FIG. 1 is a schematic diagram of step 103 of this embodiment. Figure 4 As shown, step 103 may include:
[0051] Step 401, extracting a second feature from video data; and
[0052] Step 403: Select a second encoding mode based on the second feature and a preset criterion.
[0053] In the above step 401, the second feature can be extracted by the method in the prior art. In the above step 403, the preset criterion can be, for example, a rate-distortion cost criterion, but this embodiment is not limited thereto. The preset criterion can be satisfied as long as it can make the second encoding mode selected according to the criterion a mode with a high compression rate.
[0054] The second feature is a feature for mode decision extracted from video data in an existing coding mode decision method. For example, the second feature may be motion information of the current coding block, CU division depth information, etc., or motion information of peripheral blocks of the current coding block, CU division depth information, etc., or motion information of peripheral blocks of coding blocks at the same position of the reference image of the current coding block, CU division depth information, etc., or texture features generated outside the coding process, global motion information, etc.
[0055] In this embodiment, in step 105, a conventional training method of machine learning can be used to train the machine learning model, thereby obtaining a first model; in step 107, a conventional inference method of machine learning can be used to process the third feature related to the third coding mode of the input coded data (i.e., the coded data to be processed) obtained by encoding using the third coding mode, thereby determining the fourth coding mode. The explanation and specific implementation of the third feature can be similar to the above-mentioned first feature, but since the third feature and the first feature correspond to different coded data (i.e., used as training data and data to be processed, respectively), the different coded data may use the same or different coding modes, and therefore, the third feature and the first feature may be the same or different.
[0056] Furthermore, in step 105 and step 107, the machine learning may be traditional machine learning or deep learning.
[0057] Through the coding mode determination method of this embodiment, it is possible to reduce the dimension of the features on which machine learning is based (i.e., reduce the types of information involved in the features) while ensuring the same coding model decision accuracy, thereby providing higher versatility and lower complexity at the same time.
[0058] Example 2
[0059] This embodiment 2 provides a coding mode decision device. The same contents as those in embodiment 1 are not described in detail here, and the differences between embodiment 2 and embodiment 1 are described below.
[0060] Figure 5 FIG. 1 is a schematic diagram of the coding mode decision device of this embodiment. Figure 5As shown, the coding mode determination device 500 includes an acquisition unit 501, a first processing unit 502, and a second processing unit 503. The acquisition unit 501 acquires video data and a first feature of the coded data obtained by encoding the video data using a first coding mode, and the first feature is related to the first coding mode used to encode the video data; the first processing unit 502 selects a second coding mode based on the video data and a preset criterion; the second processing unit 503 combines the first feature and the second coding mode as training data to train a machine learning model to obtain a first model, and processes a third feature of the coded data obtained by encoding using a third coding mode based on the first model to determine a fourth coding mode, and the third feature is related to the third coding mode.
[0061] The first feature and the third feature of this embodiment are the same as those in Embodiment 1 and will not be described again here.
[0062] Figure 6 FIG. 5 is a schematic diagram of the acquisition unit 501 of this embodiment. Figure 6 As shown, the acquisition unit 501 may include a video data acquisition unit 601, an encoder 602, and a decoder 603. The video data acquisition unit 601 acquires video data, the encoder 602 encodes the video data to obtain encoded data, and the decoder 603 decodes the encoded data obtained by the encoder 602 to obtain the first feature of the encoded data.
[0063] In this embodiment, the encoder 602 may be a hardware encoder that performs hardware encoding on the video data to obtain encoded data; the decoder 603 may be a software decoder that performs software decoding on the encoded data obtained by the hardware encoder to obtain the first feature of the encoded data.
[0064] Figure 7 FIG. 5 is a schematic diagram of the first processing unit 502 of this embodiment. Figure 7 As shown, the first processing unit 502 may include an extraction unit 701 and a determination unit 702. The extraction unit 701 extracts the second feature from the video data; the determination unit 702 selects the second encoding mode based on the second feature and a preset criterion.
[0065] The first processing unit 502 and the second processing unit 503 can be directly embodied as hardware, software modules executed by a processor, or a combination of the two. They can correspond to various software modules of a computer program flow or to various hardware modules. These software modules can correspond to Figure 1 The steps 103, 105 and 107 shown in FIG. These hardware modules can be implemented by solidifying these software modules using, for example, a field programmable gate array (FPGA).
[0066] The hardware modules corresponding to the first processing unit 502 and the second processing unit 503 can be independently configured or configured in the same hardware device.
[0067] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in the memory of the device or in a memory card that can be inserted into the device. For example, if the device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.
[0068] The coding mode determination device of this embodiment may be arranged in an electronic device.
[0069] In the embodiment of the present invention, the term "electronic device" includes user devices and network devices. The user devices include but are not limited to smart phones, tablet computers, etc.; the network devices include but are not limited to a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a super virtual computer consisting of a group of loosely coupled computer sets. The computer device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, VPN network, etc.
[0070] It should be noted that the user equipment, network equipment and network are only examples. Other existing or future computer equipment or networks that are applicable to the present application should also be included in the scope of protection of the present application and included here by reference.
[0071] Through the coding mode determination device of this embodiment, it is possible to reduce the dimension of the features on which machine learning is based (i.e., reduce the types of information involved in the features) while ensuring the same coding mode decision accuracy, thereby providing higher versatility and lower complexity at the same time.
[0072] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.
Claims
1. A method for determining a coding mode, characterized in that: The method comprises: Obtaining video data and a first feature of encoded data obtained by encoding the video data using a first encoding mode, wherein the first feature is related to the first encoding mode used to encode the video data; wherein the first feature is correlated with an optimal encoding mode determined based on a rate-distortion cost criterion during encoding, and the first feature includes mode information of a current encoding block, wherein the mode information includes at least one of the following: macroblock type information, inter-frame prediction direction information, or motion vector estimation accuracy information; According to the video data, extract a second feature from the video data, and select a second encoding mode based on the second feature and a preset criterion; wherein the preset criterion is the rate-distortion cost criterion, the second feature is a feature for mode decision extracted from the video data in an existing encoding mode decision method, and the second feature includes at least one of the following: motion information of the current encoding block, CU division depth information of the current encoding block, motion information of surrounding blocks of the current encoding block, CU division depth information of surrounding blocks of the current encoding block, motion information of surrounding blocks of the encoding block at the same position of a reference image of the current encoding block, CU division depth information of surrounding blocks of the encoding block at the same position of a reference image of the current encoding block, texture features generated outside the encoding process, and global motion information generated outside the encoding process; Combine the first feature and the second encoding mode as training data to train a machine learning model to obtain a first model; and Based on the first model, a third feature of the encoded data obtained by encoding using the third encoding mode is processed to determine a fourth encoding mode, the third feature is related to the third encoding mode, the third feature corresponds to the encoded data to be processed, and the specific implementation of the third feature is similar to the first feature.
2. The method according to claim 1, wherein: The first feature also includes an intermediate result generated by an encoder for encoding the video data using a first encoding mode.
3. The method according to claim 1, wherein: The acquiring of video data and the first feature of the video data comprises: Get video data; Encoding the video data by an encoder to obtain encoded data; and The encoded data obtained by the encoder is decoded by a decoder to obtain the first feature.
4. A coding mode determination device, characterized in that: The device comprises: An acquisition unit is configured to acquire video data and a first feature of encoded data obtained by encoding the video data using a first encoding mode, wherein the first feature is related to the first encoding mode used to encode the video data; wherein the first feature is correlated with an optimal encoding mode determined based on a rate-distortion cost criterion during encoding, and the first feature includes mode information of a current encoding block, wherein the mode information includes at least one of the following: macroblock type information, inter-frame prediction direction information, or motion vector estimation accuracy information; a first processing unit, which extracts a second feature from the video data according to the video data, and selects a second coding mode based on the second feature and a preset criterion; wherein the preset criterion is the rate-distortion cost criterion, the second feature is a feature for mode decision extracted from the video data in an existing coding mode decision method, and the second feature includes at least one of the following: motion information of the current coding block, CU division depth information of the current coding block, motion information of surrounding blocks of the current coding block, CU division depth information of surrounding blocks of the current coding block, motion information of surrounding blocks of a coding block at the same position of a reference image of the current coding block, CU division depth information of surrounding blocks of a coding block at the same position of a reference image of the current coding block, texture features generated outside the coding process, and global motion information generated outside the coding process; and A second processing unit combines the first feature and the second encoding mode as training data to train a machine learning model to obtain a first model, and processes a third feature of the encoded data encoded using the third encoding mode based on the first model to determine a fourth encoding mode, wherein the third feature is related to the third encoding mode, the third feature corresponds to the encoded data to be processed, and the specific implementation of the third feature is similar to the first feature.
5. The device according to claim 4, wherein: The acquisition unit comprises: A video data acquisition unit, which acquires video data; an encoder, which encodes the video data to obtain encoded data; and A decoder is configured to decode the encoded data obtained by the encoder to obtain the first feature.
