Video processing method and device

Abstract

The embodiment of the invention provides a video processing method and device, and the method comprises the steps: calculating the proportion information of an affine coding unit of a reference frame of a coding unit under the condition that the coding unit corresponds to a target time domain layer; skipping affine motion estimation processing of the coding unit under the condition that the proportion information is smaller than a preset proportion threshold value and the coding unit does not select an affine merging mode; the affine motion estimation is skipped for the coding unit, so that the video coding efficiency is effectively improved on the premise of ensuring the video visual quality.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of video processing technology, and more particularly to a video processing method and device. Background Art

[0002] With the development of computer and Internet technology, video coding technology has become more and more mature; in order to improve coding efficiency and ensure video quality during video coding, affine motion estimation technology has been applied. Affine motion estimation technology uses an affine motion model, which can describe non-translational motions such as rotation, scaling, and stretching. In video coding, affine motion estimation is used for inter-frame prediction coding, which can effectively improve coding efficiency and video quality. However, for coding units where there is no affine motion, if the affine motion estimation method is still used to encode the video frames, not only will the video quality not be improved, but the video coding efficiency will also be seriously affected. The performance benefits of affine motion estimation at this encoding stage will be very small. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a video processing method. One or more embodiments of this specification also relate to a video processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a first video processing method is provided, including:

[0005] When the coding unit corresponds to the target temporal layer, calculating the proportion information of the affine coding unit of the reference frame of the coding unit;

[0006] When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, the affine motion estimation process of the encoding unit is skipped.

[0007] According to a second aspect of the embodiments of this specification, a video processing device is provided, including:

[0008] A calculation information module, configured to calculate the proportion information of the affine coding unit of the reference frame of the coding unit when the coding unit corresponds to the target temporal layer;

[0009] The skip processing module is configured to skip the affine motion estimation processing of the encoding unit when the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode.

[0010] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0011] Memory and processor;

[0012] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned video processing method are implemented.

[0013] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned video processing method are implemented.

[0014] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned video processing method when executed by a processor.

[0015] The video processing method provided in this embodiment can adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency under the premise of ensuring video quality. When the encoding unit in the current encoding stage corresponds to the target time domain layer, the proportion information of the affine encoding unit of the reference frame of the encoding unit can be calculated, so as to reflect the influence range of the affine motion estimation in the video frame encoding stage through the proportion information of the affine encoding unit. When the proportion information is less than the preset proportion threshold and the encoding unit does not select the affine merge mode, it means that the influence is small. Even if the affine motion estimation is used for subsequent encoding processing, the improvement of the video quality is limited. Therefore, the affine motion estimation processing of the encoding unit can be skipped, and only the non-affine motion estimation processing can be performed, so as to effectively improve the video encoding efficiency under the premise of ensuring the visual quality of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of a video processing method provided by an embodiment of this specification;

[0017] Figure 2a is a flow chart of a video processing method provided by an embodiment of this specification;

[0018] Figure 2b is a schematic diagram of frame processing in a video processing method provided by an embodiment of this specification;

[0019] Figure 3 is a flow chart of a second video processing method provided by an embodiment of this specification;

[0020] Figure 4 is a flowchart of a third video processing method provided by an embodiment of this specification;

[0021] Figure 5 is a flowchart of a fourth video processing method provided by an embodiment of this specification;

[0022] Figure 6 is a processing flow chart of a video processing method provided by an embodiment of this specification;

[0023] Figure 7 is a structural schematic diagram of a video processing device provided by an embodiment of this specification;

[0024] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0025] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] First, the terms involved in one or more embodiments of this specification are explained.

[0030] Affine Motion Estimation: (AME) is a technique used in video coding and image processing to describe complex motions (such as rotation, scaling, and tilt). It goes beyond simple translational motion models and can more accurately capture the motion trajectory of objects in video frames. In the video coding process, Affine motion estimation is used for inter-frame prediction to improve coding efficiency and video quality.

[0031] In this specification, a video processing method is provided. This specification also relates to a video processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0032] See also Figure 1 As shown in the schematic diagram, the video processing method provided by this embodiment can adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency under the premise of ensuring video quality. When the coding unit in the current encoding stage corresponds to the target time domain layer, the proportion information of the affine coding unit of the reference frame of the coding unit can be calculated, so as to reflect the influence range of the affine motion estimation in the video frame encoding stage through the proportion information of the affine coding unit. When the proportion information is less than the preset proportion threshold and the coding unit does not select the affine merge mode, it means that the influence is small. Even if the affine motion estimation is used for subsequent encoding processing, the improvement of the video quality is limited. Therefore, the affine motion estimation processing of the coding unit can be skipped, and only the non-affine motion estimation processing can be performed, so as to effectively improve the video encoding efficiency under the premise of ensuring the visual quality of the video.

[0033] See also Figure 2a , Figure 2a A flow chart of a video processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0034] Step S202: when the coding unit corresponds to the target temporal layer, calculate the proportion information of the affine coding unit of the reference frame of the coding unit.

[0035] The video processing method provided in this embodiment can be applied to any scenario that does not limit codec (coder and decoder) and video encoding and decoding, such as live broadcast, on-demand, transcoding and other applications. Through the video processing method provided in this embodiment, it is possible to adaptively decide whether to skip affine motion estimation, thereby ensuring that the video encoding efficiency is improved under the premise of video quality, and more computing resources can be saved. This embodiment uses the application of the video processing method in video playback software as an example to illustrate the video processing method. The description of other scenarios can refer to the same or corresponding description content in this embodiment, and this embodiment will not be repeated in detail here.

[0036] Specifically, the target time domain layer specifically refers to the time domain layer in the video encoding process, which needs to use the video processing method provided in this embodiment to adaptively decide whether to skip affine motion estimation. The time domain layer using the video processing method in this embodiment can be set according to actual needs; for example, the video segment is frame 0 to frame 7. After the video frames are divided during the encoding process, they are divided into 3 time domain layers, namely, the low time domain layer (including frame 0, frame 4), the medium time domain layer (including frame 2, frame 6), and the high time domain layer (including frame 1, frame 3, frame 5, frame 7). At this time, the high time domain layer can be selected as the target time domain layer for subsequent processing of whether to skip affine motion estimation. During specific implementation, the setting of the target time domain layer can be set according to actual needs, and this embodiment does not make any limitation here.

[0037] Correspondingly, the coding unit specifically refers to the basic unit obtained by dividing the video frames contained in the target time domain layer at the current moment in the process of encoding the video clip. Subsequent processing of the coding unit can determine whether it is necessary to skip affine motion estimation, thereby improving the video coding efficiency without affecting the video quality; in video coding standards such as High Efficiency Video Coding (HEVC), video frames can be divided into multiple coding units, which can be further subdivided into smaller blocks (such as prediction units PU, transform units TU, etc.) for more refined coding processing. The reference frame specifically refers to the information in the previous or subsequent frame (called the reference frame) that is usually referenced during the encoding of the coding unit, and the referenced video frame is the reference frame of the coding unit, so that when inter-frame prediction is performed on the coding unit, the coding processing of the coding unit can be completed in combination with the pixel information in the reference frame. The redundant information of the current frame is reduced by combining the reference frame through motion estimation and compensation.

[0038] Correspondingly, the affine coding unit specifically refers to a coding unit that performs affine motion estimation in the encoding stage after dividing the reference frame to obtain multiple reference frame coding units. Correspondingly, the affine area information specifically refers to the area information corresponding to the affine coding unit, which is used for subsequent calculation of the proportion relative to the reference frame area information, and then decides whether to skip the affine motion estimation processing of the coding unit. Correspondingly, the proportion information specifically refers to the proportion of the affine coding unit in the reference frame relative to all the coding units in the reference frame. The proportion information can be area proportion information or quantity proportion information, and this embodiment does not make any limitation here.

[0039] Based on this, in order to be able to adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency while ensuring the video quality, the temporal layer of the coding unit associated with the current encoding stage can be detected first, and when the coding unit corresponds to the target temporal layer, the decision to skip affine motion estimation can be made. At this time, the proportion information of the affine coding unit in the reference frame of the coding unit can be calculated first, so that the decision whether to skip the affine motion estimation process can be made by analyzing the proportion information, thereby effectively improving the video encoding efficiency while ensuring the visual quality of the video.

[0040] Furthermore, in order to be able to decide whether to skip the motion estimation processing of the coding unit, it can be completed in combination with the proportion information of the affine motion estimation in the reference frame. Since affine motion estimation is a technology for describing complex motions (such as rotation, scaling and tilt) in video coding, in the video coding process, affine motion estimation is used for inter-frame prediction to improve coding efficiency and video quality; therefore, the decision whether to skip affine motion estimation can be determined in combination with the size of the range of influence of affine motion estimation on the video frame, that is, the larger the range of affine motion estimation in the video frame, the larger the range of influence will be. At this time, skipping this mechanism will affect the video quality, otherwise it will not affect the video quality. Therefore, in order to measure the range of influence, it can be determined by calculating the proportion information. Therefore, for the reference frame of the coding unit, the proportion information of the radial coding unit in the reference frame can be determined, so as to facilitate the subsequent mapping of whether it is necessary to perform affine motion estimation on the coding unit through the proportion information.

[0041] It can be understood that if the affine motion estimation ratio in the reference frame is low, it means that the influence range of the affine motion estimation is small, and the coding unit does not perform affine motion estimation at this time, which will not seriously affect the video quality. On this basis, the range affected by the affine motion estimation can be determined by calculating the corresponding ratio information of the affine coding unit, and whether the coding unit needs to skip the affine motion estimation process.

[0042] Furthermore, after obtaining the video clip to be encoded, considering that the video clip will be divided into several time domain layers in the encoding stage, each time domain layer contains a certain number of video frames, and these time domain layers have different importance and priority, and the frames of the high time domain layer need to refer to the video frames of the low time domain layer when encoding and decoding, in order to avoid skipping affine motion estimation and affecting the video quality, the video frames of the high time domain layer can be selected for affine motion estimation skipping decision, and the video frames of other time domain layers are encoded and decoded using standard processing methods. In this embodiment, the specific implementation method is as follows:

[0043] A video clip is obtained, and a video frame sequence is obtained by performing frame processing on the video clip; the video frames included in the video frame sequence are divided, and at least two time domain layers are determined according to the division result, wherein the video frames included in each time domain layer constitute the video frame sequence; when encoding the video frames included in each time domain layer, and when the time domain layer corresponding to the coding unit processed in the current encoding stage is the target time domain layer, a step of calculating the proportion information of the affine coding unit of the reference frame of the coding unit is performed.

[0044] Specifically, the video frame sequence refers to a sequence of multiple video frames arranged in time order after the video clip is framed. Correspondingly, the time domain layer refers to the time domain layers obtained by dividing the video frame sequence, and each time domain layer contains a certain number of video frames, and the video frames contained in each time domain layer will constitute a video frame sequence.

[0045] Based on this, in order to ensure that the video quality is not affected, after obtaining the video clip, the video clip can be first processed by dividing the video clip into frames to obtain a video frame sequence; at this time, the video frames contained in the video frame sequence can be divided according to the preset division method, and at least two time domain layers can be determined according to the division results, and the video frames contained in each time domain layer constitute a video frame sequence. On this basis, since each time domain layer has different importance and encoding and decoding priorities, the video frames contained in each time domain layer can be encoded in sequence according to the encoding and decoding priority. In the case where the time domain layer corresponding to the current encoding stage is the target time domain layer, it means that the time domain layer for encoding processing at this time is the high time domain layer, and the video frame encoding and decoding of the high time domain layer refers to the video frame of the low time domain layer, so the adaptive decision affine motion estimation skipping process can be performed on the coding unit of the high time domain layer, so the step of determining the coding unit corresponding to the target time domain layer can be continued for subsequent decision processing.

[0046] In practical applications, since the encoding and decoding of video frames of a high temporal layer needs to refer to video frames of a low temporal layer, the highest temporal layer can be directly selected as the target temporal layer for subsequent processing.

[0047] For example, after a user submits an on-demand movie A command through the video player application X installed on the mobile phone, the server of the video player application X will respond to the user's on-demand command and send the video of movie A to the mobile phone, so that the mobile phone can play movie A after decoding the video. After the video player application X on the mobile phone receives the video clip sent by the server, during the video encoding process, such as Figure 2b As shown, video clip frames 0 to 7 will be divided into three temporal layers, namely low temporal layer 0 (including frame 0 and frame 4), medium temporal layer 1 (including frame 2 and frame 6), and high temporal layer 2 (including frame 1, frame 3, frame 5, and frame 7); wherein, frame 2 in medium temporal layer 1 and frame 1 in high temporal layer 2 will be coded and decoded with reference to frame 0 in low temporal layer 0, frame 3 in high temporal layer 2 will be coded and decoded with reference to frame 2 in medium temporal layer 1, frame 6 in medium temporal layer 1 and frame 5 in high temporal layer 2 will be coded and decoded with reference to frame 4 in low temporal layer 0, and frame 7 in high temporal layer 2 will be coded and decoded with reference to frame 6 in medium temporal layer 1. On this basis, Affine motion estimation skip decision can be made for the coding unit of high temporal layer 2.

[0048] In summary, by performing affine motion estimation skipping decision on the coding unit of the target time domain layer, it is possible to avoid affecting the video quality in the case of skipping.

[0049] Further, when it is determined that only the coding unit of the high temporal layer needs to be subjected to the affine motion estimation skip decision, during the video segment encoding and decoding stage, it is necessary to detect whether the coding unit corresponds to the high temporal layer, so as to decide whether to start the affine motion estimation skip decision processing. In this embodiment, the specific implementation method is as follows:

[0050] Determine the coding unit to be detected corresponding to the current encoding moment; load the unit identification information corresponding to the coding unit to be detected, and detect whether the coding unit to be detected meets the affine motion estimation skip decision condition according to the unit identification information; if so, determine that the time domain layer corresponding to the coding unit to be detected is the target time domain layer, and use the coding unit to be detected as the coding unit; if not, perform standard encoding processing on the coding unit to be detected.

[0051] Specifically, the coding unit to be detected specifically refers to the coding unit that the encoder needs to encode and process at the current encoding moment. Correspondingly, the unit identification information specifically refers to the Tid of the coding unit to be detected, and the unit identification information can reflect the time domain layer to which the coding unit to be detected belongs. Correspondingly, the affine motion estimation skip decision condition is the condition for detecting the unit identification information, and then determining whether the coding unit belongs to the time domain layer that needs to skip affine motion estimation.

[0052] Based on this, when the encoder encodes the target video, the coding unit to be detected corresponding to the current encoding moment can be determined first; at this time, the unit identification information corresponding to the coding unit to be detected can be loaded first, and the coding unit to be detected can be detected according to the unit identification information to determine whether it meets the affine motion estimation skip decision condition; if not, it means that the coding unit to be detected does not correspond to the high temporal layer, so the coding unit can be encoded in a standard encoding method. If yes, it means that the coding unit to be detected corresponds to the high temporal layer (target temporal layer), so the temporal layer corresponding to the coding unit to be detected can be the target temporal layer, and the coding unit to be detected can be used as the coding unit at this time, so that the affine motion estimation skip decision can be made for the coding unit of the high temporal layer later.

[0053] In practical applications, the affine motion estimation skip decision condition can be set as a Tid threshold as needed, and the Tid threshold is the threshold for detecting whether a coding unit corresponds to a high temporal layer.

[0054] Continuing with the above example, during the encoding stage for the video clips of movie A, since the encoding and decoding of the video frames of each time domain layer needs to be completed by the coding unit, in order to be able to make an Affine motion estimation skip decision for the coding unit of the high time domain layer 2, the current coding unit and its corresponding Tid can be determined during the encoding stage. At this time, the Tid can be compared with the preset Tid threshold. When the Tid corresponding to the current coding unit is higher than the Tid threshold, it means that the video frame that needs to be encoded and processed by the current coding unit belongs to the high time domain layer 2. Therefore, the Affine motion estimation skip decision mechanism can be enabled to implement subsequent decisions on whether Affine motion estimation needs to be skipped.

[0055] In summary, by determining the time domain layer to which the coding unit belongs by detecting the unit identification information, it is possible to enable subsequent affine motion estimation skipping decisions when the coding unit corresponds to a high time domain layer, thereby completing video encoding while ensuring video quality.

[0056] Furthermore, when it is determined that the coding unit to be processed in the current coding stage corresponds to the target temporal layer, in order to make a subsequent decision on whether to skip the affine motion estimation of the coding unit, the reference frame corresponding to the coding unit can be determined, so that the area information ratio calculation can be completed in combination with the reference frame. In this embodiment, the specific implementation method is as follows:

[0057] Determine multiple candidate reference frames, and determine a candidate coding area corresponding to the coding unit in each candidate reference frame; calculate the pixel similarity between the coding unit and each candidate coding area, and select a reference frame of the coding unit from the multiple candidate reference frames according to the pixel similarity calculation result.

[0058] Specifically, the candidate reference frame refers to the reference frame selected by the encoder for the coding unit corresponding to the current coding moment. In practical applications, the candidate reference frame can be screened by the encoder according to the rate-distortion optimization principle, and the higher the content similarity between the selected candidate reference frame and the coding unit, the smaller the prediction error. Therefore, the encoder can refer to the correlation between the coding unit and the reference frame when selecting. Correspondingly, the candidate coding area refers to the area in each candidate reference frame that matches the area corresponding to the coding unit, which is used to analyze the pixel similarity of the same position between the candidate reference frame and the coding unit. Correspondingly, pixel similarity refers to the determination of the similarity between the candidate coding area and the coding unit by calculating the similarity of pixel values.

[0059] Based on this, when the encoder screens reference frames for a coding unit, it can first determine multiple candidate reference frames, and on this basis, it can determine the candidate coding area corresponding to the coding unit in each candidate reference frame; at this time, the pixel similarity between the coding unit and each candidate coding area can be calculated, and then the candidate reference frame to which the candidate coding area with the highest similarity belongs can be selected according to the pixel similarity calculation result as the reference frame of the coding unit for subsequent use.

[0060] In practical applications, in addition to the above-mentioned method of selecting reference frames, it can also be implemented based on features, that is, the encoder can map candidate reference frames to feature space, and then calculate the similarity between the coding unit and the candidate reference frame in the feature space, and then screen the reference frame that matches the coding unit. It can also be specified by the coding standard, that is, the encoder can define a reference frame list (Reference Picture Set, RPS) according to a preset standard to manage reference frames, and select a suitable reference frame according to the provisions in the RPS and the characteristics of the current coding unit. In specific implementation, the reference frame selection method can be set according to actual needs, and this embodiment does not make any restrictions here.

[0061] Continuing with the above example, when the encoder performs encoding on the video frame 5 included in the high temporal layer 2, it will first divide the video frame 5 into multiple coding units, and then perform encoding processing on each coding unit. During the encoding process, for any coding unit, the encoder can pre-select multiple candidate reference frames. At this time, a candidate coding area with the same position as the coding unit can be determined in each candidate reference frame, and then by calculating the pixel similarity between each candidate coding area and the coding unit, it is determined that the candidate reference frame 1 has the highest correlation with the coding unit, so it can be used as the reference frame corresponding to the current coding unit, so as to perform affine motion estimation skip decision processing for the current coding unit later.

[0062] In summary, by calculating pixel similarity to filter reference frames for coding units, the correlation between reference frames and coding units can be guaranteed, and then accurate affine motion estimation can be made to ensure video quality.

[0063] Furthermore, in order to accurately calculate the area ratio and decide whether to skip the affine motion estimation, it is necessary to first screen out the affine coding unit and then determine its corresponding ratio information, so that the affine motion estimation decision processing can be performed. In this embodiment, the specific implementation method is as follows:

[0064] Traversing a plurality of initial coding units in a reference frame of the coding unit;

[0065] An initial coding unit for affine motion estimation is selected according to the traversal result as the affine coding unit, and the proportion information of the affine coding unit is calculated.

[0066] Specifically, the initial coding unit specifically refers to all coding units obtained after dividing the reference frame. Based on this, after determining that the coding unit corresponds to the target time domain layer, in order to intuitively and accurately reflect the influence of affine motion estimation in the coding unit encoding the video frame, it can be reflected by calculating the proportion information. Before this, multiple initial coding units in the reference frame of the coding unit can be traversed to select the coding unit for affine motion estimation as the affine coding unit according to the traversal results; thereafter, the proportion information of the affine coding unit can be calculated, so as to realize the subsequent decision of affine motion estimation in combination with the proportion information.

[0067] In practical applications, if the proportion information is the area proportion information of the affine coding unit, it can be determined by calculating the width-height product based on the coding size information corresponding to the affine coding unit; in addition, the proportion information corresponding to the affine coding unit can also be determined by counting the number of pixels in the video frame area, and this embodiment does not make any limitations here.

[0068] In summary, by screening the affine coding unit when calculating the proportion information, it can be ensured that the calculated proportion information intuitively reflects the proportion of affine motion estimation of the coding unit corresponding to the reference frame, and then can be used as a factor condition for deciding whether the coding unit skips the affine motion estimation, thereby making accurate decisions while reducing the impact.

[0069] Furthermore, considering that the encoding process of a video frame may require multiple encoding units to complete, different calculation methods can be selected according to needs for the proportion information corresponding to the affine encoding unit. In this embodiment, on the one hand: the affine area information of each affine encoding unit is summed to obtain local affine area information, the global area information of the reference frame is determined, and the area proportion information of the local affine area information relative to the global area information is calculated, and the area proportion information is used as the proportion information of the affine encoding unit.

[0070] On the other hand, determine the initial unit number of the multiple initial coding units and the affine unit number of the multiple affine coding units, calculate the quantity ratio information of the affine unit number relative to the initial unit number, and use the quantity ratio information as the ratio information of the affine coding unit.

[0071] Specifically, the local affine area information specifically refers to the area information obtained by summing the affine area information corresponding to each affine coding unit. Correspondingly, the global area information specifically refers to the area information corresponding to the reference frame. Correspondingly, the initial unit number specifically refers to the number corresponding to all coding units contained in the reference frame. Correspondingly, the affine unit number specifically refers to the number corresponding to the coding units that perform affine motion estimation processing in the reference frame.

[0072] Based on this, when there are multiple affine coding units, if the proportion information is calculated according to the area proportion, the affine area information of each affine coding unit can be summed to obtain the local affine area information; at the same time, the global area information of the reference frame is obtained, and on this basis, the area proportion information of the local affine area information relative to the global area information is calculated, and the area proportion information can be used as the proportion information corresponding to the affine coding unit, thereby reflecting the influence range of the affine motion estimation.

[0073] If the proportion information is calculated according to the proportion of the number of units, the initial unit number of multiple initial coding units in the reference frame and the affine unit number of multiple affine coding units can be determined. On this basis, the quantity proportion information of the affine unit number relative to the initial unit number can be calculated. At this time, the quantity proportion information can be used as the proportion information of the affine coding unit; thereby reflecting the influence range of the affine motion estimation.

[0074] In practical applications, when calculating the proportion information, in addition to the above calculations based on the area proportion and the number proportion, the proportion information can also be calculated by calculating the number of pixels corresponding to the affine coding unit and the number of pixels corresponding to the reference frame, and by calculating the proportion of the number of pixels. In specific implementation, the calculation method for the proportion information can be selected according to actual needs, and this embodiment does not make any limitation here.

[0075] Using the above example, after determining that the current coding unit CU needs to make an Affine motion estimation skip decision, you can first obtain the reference frame corresponding to the current coding unit CU, and at the same time determine the coding units involving the affine transformation model in all coding units CU of the reference frame as the affine coding unit AffineCU; on this basis, you can calculate the accumulation of the area corresponding to the affine coding unit AffineCU, and determine that the accumulated area is M1. The area of ​​the reference frame corresponding to the current coding unit CU is M2. By calculating the ratio of M1 to M2, the area ratio Affine rate is determined to be S; thereafter, the Affine motion estimation skip decision of the current coding unit CU can be made according to the area ratio Affine rate = S.

[0076] In summary, by choosing different methods to calculate the proportion information, the influence of affine motion estimation can be intuitively reflected through the calculated proportion information, and then the affine motion estimation can be skipped by subsequent comparison thresholds to improve video encoding efficiency while ensuring video quality.

[0077] Step S204: When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, skip the affine motion estimation process of the encoding unit.

[0078] Specifically, after obtaining the proportion information corresponding to the affine coding unit as mentioned above, the proportion information can be compared with the preset proportion threshold; when the proportion information is less than the preset proportion threshold, it means that if the coding unit does not perform affine motion estimation but directly performs non-affine motion estimation processing, the impact on the video quality is relatively low. At the same time, if the coding unit has selected the affine merge mode, the affine motion estimation cannot be skipped. Therefore, after comparing the proportion information, it is determined that the coding unit has not selected the affine merge mode. Then, the affine motion estimation processing of the coding unit can be skipped, and the non-affine motion estimation processing can be performed on the coding unit, thereby ignoring the affine motion estimation calculation of the coding unit, thereby improving the video coding efficiency. It should be noted that any coding unit decides whether to skip the affine motion estimation, which can be decided by the video processing method provided in this embodiment.

[0079] Among them, the preset proportion threshold specifically refers to a set threshold for measuring whether it is necessary to ignore the affine motion estimation, which can be set according to actual needs, and this embodiment does not make any limitation here. Correspondingly, the affine merge mode not selected by the coding unit can be understood as a non-affine merge mode, which specifically refers to other coding modes except the affine merge mode, which belongs to an inter-frame prediction technology and can be widely used in video coding standards, such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). Correspondingly, the affine merge mode specifically refers to the Affine Merge mode, which is mainly used to process video blocks with complex motions (such as rotation, scaling, etc.) to improve coding efficiency and video quality.

[0080] That is to say, when the decision to skip affine motion estimation is made for a coding unit, if the coding unit corresponds to a high time domain layer, the proportion information is less than the preset proportion threshold, and the coding unit does not adopt the Affine Merge mode, the affine motion estimation part can be skipped for the coding unit, thereby improving the video coding efficiency while ensuring the video quality.

[0081] Continuing with the above example, on the basis of obtaining the area ratio Affine rate = S, the area ratio Affinerate = S can be compared with the threshold S1; if S is less than S1, and the current coding unit CU does not adopt the Affine Merge mode, it can be decided to skip the affine motion estimation, so that the encoder can use the non-Affine Merge mode to perform non-affine motion estimation processing on the coding unit CU, and so on. After completing the video encoding processing, the user can watch movie A through the video playback application X on the mobile phone.

[0082] In addition, in order to avoid affecting the video quality, if any condition is not met, affine motion estimation will still be required, that is, when the coding unit to be detected does not meet the affine motion estimation skip decision condition, or the proportion information is greater than or equal to the preset proportion threshold, or the coding unit selects the affine merge mode, non-affine motion estimation processing and affine motion estimation processing are performed on the coding unit.

[0083] That is to say, if any of the above conditions are not met, if the affine motion estimation processing is skipped for the coding unit, the visual quality of the video may be affected, so the coding unit can be subjected to non-affine motion estimation processing and affine motion estimation processing. When the encoder performs non-affine motion estimation processing and affine motion estimation processing on the coding unit, the non-affine motion estimation processing needs to determine the reference frame block that matches the coding unit in the reference frame of the coding unit, and then the position offset between the coding unit and the reference frame block can be calculated, that is, the motion vector, which is used to describe the displacement of the coding unit in the horizontal and vertical directions. Thereafter, according to the calculated motion vector, the reference frame block in the reference frame can be displaced to generate a prediction of the coding unit in the corresponding current frame. Then, the difference (residual) between the predicted image block and the actual image block is calculated, and the residual is encoded and transmitted.

[0084] When performing affine motion estimation, an affine transformation model can be used to describe the motion of a coding unit between consecutive frames. The affine transformation model can contain multiple parameters (such as translation, rotation angle, scaling factor, etc.), which need to be estimated by setting algorithms (such as least squares method, gradient descent method, etc.). During the estimation process, the encoder scans the neighborhood of the coding unit, finds the motion vector pointing to the same reference frame block, and creates the original sample position. Then, the target sample position is generated by offsetting the center point position of the current block and applying the affine transformation. Finally, the affine model parameters are calculated using all available original and target positions. On this basis, the reference frame block in the reference frame can be affine transformed according to the estimated affine model parameters to generate a prediction of the coding unit in the current frame. Similarly, the difference (residual) between the predicted image block and the actual image block is calculated, and the residual is encoded and transmitted.

[0085] In specific implementation, before the encoder processes the coding unit, it usually selects a prediction mode for the coding unit. In this embodiment, the prediction mode is selected as follows:

[0086] Determine a target prediction mode set corresponding to the coding unit; determine rate-distortion cost information corresponding to each prediction mode in the target prediction mode set by traversing the target prediction mode set; select a target prediction mode in the target prediction mode set according to the rate-distortion cost information corresponding to each prediction mode, and when the target prediction mode is not an affine merge mode, execute an affine motion estimation processing step of skipping the coding unit.

[0087] Specifically, the target prediction mode set specifically refers to a set of all prediction modes that the encoder can select for the coding unit. Correspondingly, the target prediction mode specifically refers to screening out the prediction mode with the minimum cost after calculating the rate-distortion cost information corresponding to each prediction mode.

[0088] Based on this, when the encoder selects a prediction mode for a coding unit, it can first determine the target prediction mode set corresponding to the coding unit; at this time, the target prediction mode set can be traversed to determine the rate-distortion cost information corresponding to each prediction mode in the target prediction mode set according to the traversal result; on this basis, considering that the rate-distortion cost information represents the size of the distortion cost of the prediction mode, the prediction mode with the smallest cost can be selected from the target prediction mode set according to the rate-distortion cost information corresponding to each prediction mode as the target prediction mode. Thereafter, if the target prediction mode is a non-affine merge mode, the processing mode of the coding unit can be determined to be a non-affine merge mode, and the affine motion estimation processing step of skipping the coding unit can be executed.

[0089] In practical applications, when selecting a prediction mode for a coding unit, the target bit rate and quantization parameter can be determined first, wherein the target bit rate can be preset by the encoder, and the quantization parameter can be calculated by the encoder based on the target bit rate. Secondly, the Lagrangian factor is calculated. In rate-distortion optimization, the key parameter λ between the coding distortion D and the bit rate R can be calculated by the Lagrangian algorithm. Thirdly, the encoder can traverse all the prediction modes available for the coding unit. At this time, for each prediction mode, the encoder can calculate the corresponding rate-distortion cost information in combination with the above parameters. Finally, the prediction mode with the smallest rate-distortion cost can be selected as the optimal prediction mode matching the coding unit. This allows the encoder to encode the coding unit according to this mode and update the status and parameters of the coding unit.

[0090] After the prediction mode is selected for the coding unit, the affine motion estimation decision can be made in combination with the last condition, so that the encoder can perform non-affine motion estimation processing on the coding unit, or perform non-affine motion estimation processing and affine motion estimation processing according to the decision result. Similarly, after the coding units contained in each video frame in the video segment are encoded, the encoded video segment can be used for subsequent services, such as pushing the encoded video to the client for users to watch.

[0091] The video processing method provided in this embodiment can adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency under the premise of ensuring video quality. When the encoding unit in the current encoding stage corresponds to the target time domain layer, the proportion information of the affine encoding unit of the reference frame of the encoding unit can be calculated, so as to reflect the influence range of the affine motion estimation in the video frame encoding stage through the proportion information of the affine encoding unit. When the proportion information is less than the preset proportion threshold and the encoding unit does not select the affine merge mode, it means that the influence is small. Even if the affine motion estimation is used for subsequent encoding processing, the improvement of the video quality is limited. Therefore, the affine motion estimation processing of the encoding unit can be skipped, and only the non-affine motion estimation processing can be performed, so as to effectively improve the video encoding efficiency under the premise of ensuring the visual quality of the video.

[0092] Corresponding to the above method embodiment, this specification also provides a video processing method applied in a live broadcast scenario. Figure 3 FIG. 1 shows a flow chart of a second video processing method provided by an embodiment of the present specification. Figure 3 As shown, the specific steps include:

[0093] Step S302, in response to the live viewing request, determining a target live video stream, and encoding the target live video stream.

[0094] Step S304, when the coding unit of the video frame of the target live video stream corresponds to the target time domain layer, calculate the proportion information of the affine coding unit of the reference frame of the coding unit.

[0095] Step S306: When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, skip the affine motion estimation process of the encoding unit.

[0096] Step S308, when the encoding of the target live video stream is completed, the encoded target live video stream is sent to the live viewing terminal.

[0097] Specifically, the method can be applied to a streaming media server in a live broadcast scenario, and is used to convert the received live video stream after the streaming media server receives the live video stream pushed by the live broadcast client, thereby converting the live video stream into a resolution or format that matches the viewing requirements of the live broadcast viewing terminal for viewing by the live broadcast viewing terminal. Accordingly, the live video stream is the video stream that the live broadcast data collected by the live broadcast client is pushed to the streaming media server. Accordingly, the live broadcast viewing terminal specifically refers to the client held by the user who watches the live broadcast content, which can be a mobile phone, computer, notebook or other device, and this embodiment does not make any limitation here.

[0098] Based on this, after receiving the live video stream, the streaming media server can determine the live video stream that the live viewing terminal needs to watch in response to the live viewing request of the live viewing terminal in order to adapt to different users, considering that different live viewing terminals have different viewing requests. After that, the live video stream can be encoded. During the encoding process, in order to ensure the visual quality and improve the encoding efficiency, the affine motion estimation skip decision can be made for the encoding units after the video frame is divided in the encoding stage.

[0099] On this basis, if the coding unit processed in the current coding stage corresponds to the target temporal layer, the proportion information of the affine coding unit of the reference frame of the coding unit can be calculated, so as to skip the affine motion estimation processing of the coding unit when the area information proportion is less than the preset proportion threshold and the processing mode of the coding unit is the non-affine merge mode.

[0100] In the encoding process of the live video stream, the above processing operation is performed on any encoding unit, which can skip unnecessary affine motion estimation, thereby improving the video encoding efficiency. When the encoding of the live video stream is completed, the encoded live video stream can be sent to the live viewing end for users of the live viewing end to watch the live content.

[0101] In addition, in a live broadcast scenario, after the live broadcast client (anchor end) captures the live broadcast screen content, since the original unprocessed video stream is not suitable for direct transmission in the network, the video processing method provided in this embodiment can also be used at this stage to encode and compress the video stream data to reduce the amount of data while ensuring the image quality.

[0102] For example, after the streaming media server collects the live data stream corresponding to live user A, in order to be able to send the video stream matching the client to the viewer client watching the live user A, it will encode the live video stream in response to the live viewing request submitted by the viewer. During the encoding process, the video frames in the live video stream will be divided into multiple coding units, and then at any encoding stage, it will detect whether the current coding unit belongs to the high temporal layer, the proportion of the affine area information of the affine coding unit in the corresponding reference frame relative to the reference frame, and whether the encoder selects the affine merging mode for the coding unit. If all the above three conditions are met, the affine motion estimation processing of the coding unit can be skipped, and only non-affine motion estimation processing is performed. By analogy, after the coding units contained in the video frames in the live video stream are processed, the video encoding efficiency can be improved under the premise of video quality. After the encoding is completed, the live video stream with the required format and resolution is pushed to the viewer client for viewing.

[0103] It should be noted that the video processing method provided in this embodiment is applied to a live broadcast scenario. For any contents not described in detail, please refer to the same or corresponding description contents in the above embodiments, and this embodiment will not be described in detail here.

[0104] In summary, in a live broadcast scenario, using multiple conditions to decide whether the coding unit needs affine motion estimation can improve the video coding efficiency while ensuring the quality of the live video stream, thereby effectively improving the effect of the audience client watching the live content.

[0105] Corresponding to the above method embodiment, this specification also provides a video processing method applied to a video on demand scenario. Figure 4 A flowchart of a third video processing method provided by an embodiment of this specification is shown. Figure 4 As shown, the specific steps include:

[0106] Step S402: In response to the video-on-demand request, determine a target video-on-demand, and encode the target video-on-demand.

[0107] Step S404: when the coding unit of the video frame of the target on-demand video corresponds to the target temporal layer, calculate the proportion information of the affine coding unit of the reference frame of the coding unit.

[0108] Step S406: When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, skip the affine motion estimation process of the encoding unit.

[0109] Step S408: When the encoding of the target video-on-demand is completed, the encoded target video-on-demand is sent to the video-on-demand client.

[0110] Specifically, the method can be applied to a video-on-demand server in a video-on-demand scenario, the server is used to store videos for users to watch on demand, and after receiving a video-on-demand request submitted by a video-on-demand client, read the video on demand from the video database, and perform encoding and decoding processing on the video, so as to send the video-on-demand client that meets the viewing needs of the user. Correspondingly, the video-on-demand client specifically refers to the client held by the user who submits the video-on-demand request, which can be a mobile phone, computer, notebook and other devices, and this embodiment does not make any limitation here.

[0111] Based on this, after determining the video requested by the on-demand client, considering that different users have different viewing requirements when ordering videos, such as resolution, format, etc. In order to adapt to different users, the on-demand video that they need to watch can be determined in response to the video on-demand request, and then the on-demand video can be encoded. During the encoding process, in order to ensure the visual quality and improve the encoding efficiency, the affine motion estimation skip decision can be made for the coding unit after the video frame is divided in the encoding stage.

[0112] On this basis, if the coding unit processed in the current coding stage corresponds to the target temporal layer, the proportion information of the affine coding unit of the reference frame of the coding unit can be calculated, so as to skip the affine motion estimation processing of the coding unit when the area information proportion is less than the preset proportion threshold and the processing mode of the coding unit is the non-affine merge mode.

[0113] In the encoding process of on-demand video, the above processing operation is performed on any encoding unit, which can skip unnecessary affine motion estimation, thereby improving video encoding efficiency. When the on-demand video encoding is completed, the encoded on-demand video can be sent to the on-demand client for the user of the on-demand client to watch the on-demand content.

[0114] For example, a video on demand platform provides users with on-demand services. After receiving a video on demand request submitted by user A for movie B, the video corresponding to movie B can be encoded in response to the video on demand request submitted by user A. During the encoding process, the video frames in the video will be divided into multiple encoding units, and then at any encoding stage, it will be detected whether the current encoding unit belongs to the high temporal layer, the proportion of the affine area information of the affine encoding unit in the corresponding reference frame relative to the reference frame, and whether the encoder selects the affine merge mode for the encoding unit. If all three conditions are met, the affine motion estimation processing of the encoding unit can be skipped, and only non-affine motion estimation processing is performed. By analogy, after the encoding units contained in the video frames in the video are processed, the video encoding efficiency can be improved under the premise of video quality. After the encoding is completed, the video content of movie B in the required format and resolution can be pushed to the terminal device of user A for user A to watch.

[0115] It should be noted that the video processing method provided in this embodiment is applied to a video on demand scenario. For any contents not described in detail, please refer to the same or corresponding description contents in the above embodiments, and this embodiment will not be described in detail here.

[0116] In summary, in the video on demand scenario, using multiple conditions to decide whether the coding unit needs affine motion estimation can improve the video encoding efficiency while ensuring the quality of the on-demand video, thereby effectively improving the effect of the on-demand client watching the on-demand content.

[0117] Corresponding to the above method embodiment, this specification also provides a video processing method applied in a video transcoding scenario. Figure 5 FIG. 4 shows a flowchart of a fourth video processing method provided by an embodiment of the present specification. Figure 5 As shown, the specific steps include:

[0118] Step S502: In response to the transcoding request, the video to be processed is loaded and the video to be processed is encoded.

[0119] Step S504, when the coding unit of the video frame of the to-be-processed video corresponds to the target temporal layer, calculate the proportion information of the affine coding unit of the reference frame of the coding unit.

[0120] Step S506: When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, skip the affine motion estimation process of the encoding unit.

[0121] Step S508: When the encoding of the video to be processed is completed, a target video is generated according to the encoded video to be processed.

[0122] Specifically, the method can be applied to a transcoding application in a video transcoding scenario, and the transcoding application can, after loading a video to be processed uploaded by a user, perform transcoding processing on the video according to the transcoding parameters set by the user, so as to feedback the transcoded target video to the user. Accordingly, the target video specifically refers to a video that matches the transcoding parameters set by the user after the video to be processed is transcoded.

[0123] Based on this, when the user uses the transcoding application to perform video transcoding, he can first respond to the transcoding request to load the video to be processed, and then encode the video to be processed. During the encoding process, in order to ensure that the encoding efficiency is improved under the premise of visual quality, the proportion information of the affine encoding unit of the reference frame of the encoding unit can be calculated. Thereby, when the area information proportion is less than the preset proportion threshold and the processing mode of the encoding unit is the non-affine merge mode, the affine motion estimation processing of the encoding unit is skipped. Thereby, when the area information proportion is less than the preset proportion threshold and the processing mode of the encoding unit is the non-affine merge mode, the non-affine motion estimation processing of the encoding unit is performed.

[0124] During the transcoding process, the above processing operation is performed for any coding unit, which can skip unnecessary affine motion estimation, thereby improving video coding efficiency. When the encoding of the video to be processed is completed, the target video can be obtained, and finally the target video is stored locally on the client for users to download and use.

[0125] For example, when a video editing user uses video editing software to process a video to be processed, the video to be processed can be loaded in response to the user's transcoding request. When encoding the video, the video editing application can divide the video frames in the video into multiple coding units, and then detect whether the current coding unit belongs to the high temporal layer, the proportion of the affine area information of the affine coding unit in its corresponding reference frame relative to the reference frame, and detect whether the encoder selects the affine merging mode for the coding unit. When all three conditions are met, the affine motion estimation processing of the coding unit can be skipped, and only non-affine motion estimation processing is performed. By analogy, after the coding units contained in the video frames in the video are processed, the video encoding efficiency can be improved under the premise of video quality. After the encoding is completed, the transcoded target video can be downloaded to the user client, so that the user can publish the edited video on the multimedia platform.

[0126] It should be noted that the video processing method provided in this embodiment is applied to the video transcoding scenario. For the contents not described in detail, please refer to the same or corresponding description contents in the above embodiments, and this embodiment will not be described in detail here.

[0127] In summary, in the video transcoding scenario, using multiple conditions to decide whether a coding unit needs affine motion estimation can improve the video coding efficiency while ensuring the quality of the transcoded video.

[0128] The following combination Figure 6 , taking the application of the video processing method provided in this specification in a live broadcast scenario as an example, the video processing method is further described. Figure 6 A processing flow chart of a video processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0129] Step S602, receiving the live video stream uploaded by the live broadcast end, and determining the coding unit to be detected corresponding to the live video stream at the current coding time.

[0130] Step S604: load the unit identification information corresponding to the coding unit to be detected, and detect whether the coding unit to be detected meets the affine motion estimation skip decision condition according to the unit identification information.

[0131] Step S606: If the conditions are met, determine that the time domain layer corresponding to the coding unit to be detected is the target time domain layer, and use the coding unit to be detected as the coding unit.

[0132] Step S608: traverse multiple initial coding units associated with the reference frame of the coding unit.

[0133] Step S610: Select an initial coding unit using an affine transformation model according to the traversal result as an affine coding unit associated with the coding unit.

[0134] Step S612: load the encoding size information corresponding to the affine encoding unit, and calculate the affine area information of the affine encoding unit according to the encoding size information.

[0135] Step S614, calculating the ratio of the affine area information to the area information of the reference frame.

[0136] Step S616: When the area information ratio is less than the preset ratio threshold and the coding unit does not select the affine merge mode, skip the affine motion estimation process of the coding unit.

[0137] Step S618, when the encoding of the live video stream is completed, the encoded live video stream is sent to the viewer client.

[0138] In summary, in order to be able to adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency while ensuring the video quality, the proportion information of the affine coding unit of the reference frame of the coding unit can be calculated when the coding unit in the current encoding stage corresponds to the target time domain layer, so as to reflect the influence range of the affine motion estimation in the video frame encoding stage through the proportion information of the affine coding unit. When the proportion information is less than the preset proportion threshold and the coding unit does not select the affine merge mode, it means that the influence is small. Even if affine motion estimation is used for subsequent encoding processing, the improvement of video quality is limited. Therefore, the affine motion estimation processing of the coding unit can be skipped, and only non-affine motion estimation processing can be performed to effectively improve the video encoding efficiency while ensuring the visual quality of the video.

[0139] Corresponding to the above method embodiment, this specification also provides a video processing device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of a video processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0140] The information calculation module 702 is configured to calculate the proportion information of the affine coding unit of the reference frame of the coding unit when the coding unit corresponds to the target temporal layer;

[0141] The skip processing module 704 is configured to skip the affine motion estimation processing of the encoding unit when the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode.

[0142] In an optional embodiment, the device further comprises:

[0143] The detection unit module is configured to determine the coding unit to be detected corresponding to the current encoding moment; load the unit identification information corresponding to the coding unit to be detected, and detect whether the coding unit to be detected meets the affine motion estimation skip decision condition according to the unit identification information; if so, determine that the time domain layer corresponding to the coding unit to be detected is the target time domain layer, and use the coding unit to be detected as the coding unit.

[0144] In an optional embodiment, the calculation information module 702 is further configured to:

[0145] A plurality of initial coding units in a reference frame of the coding unit are traversed; an initial coding unit for affine motion estimation is selected as an affine coding unit according to the traversal result, and proportion information of the affine coding unit is calculated.

[0146] In an optional embodiment, the device further comprises:

[0147] The frame segmentation module is configured to obtain a video clip, obtain a video frame sequence by performing frame segmentation processing on the video clip; divide the video frames contained in the video frame sequence, and determine at least two time domain layers according to the division result, wherein the video frames contained in each time domain layer constitute the video frame sequence; when encoding the video frames contained in each time domain layer, and when the time domain layer corresponding to the coding unit processed in the current encoding stage is the target time domain layer, execute the step of calculating the proportion information of the affine coding unit of the reference frame of the coding unit.

[0148] In an optional embodiment, the calculation information module 702 is further configured to:

[0149] Sum the affine area information of each affine coding unit to obtain local affine area information, determine the global area information of the reference frame, calculate the area ratio information of the local affine area information relative to the global area information, and use the area ratio information as the ratio information of the affine coding unit; or, determine the initial unit number of the multiple initial coding units and the affine unit number of the multiple affine coding units, calculate the number ratio information of the affine unit number relative to the initial unit number, and use the number ratio information as the ratio information of the affine coding unit.

[0150] In an optional embodiment, the device further comprises:

[0151] When the coding unit to be detected does not meet the affine motion estimation skip decision condition, or the proportion information is greater than or equal to the preset proportion threshold, or the coding unit selects the affine merge mode, non-affine motion estimation processing and affine motion estimation processing are performed on the coding unit.

[0152] In an optional embodiment, the device further comprises:

[0153] The mode selection module is configured to determine a target prediction mode set corresponding to the coding unit; determine rate-distortion cost information corresponding to each prediction mode in the target prediction mode set by traversing the target prediction mode set; select a target prediction mode in the target prediction mode set according to the rate-distortion cost information corresponding to each prediction mode, and execute an affine motion estimation processing step of skipping the coding unit when the target prediction mode is not an affine merge mode.

[0154] In an optional embodiment, the reference frame of the encoding unit is determined, including:

[0155] Determine multiple candidate reference frames, and determine a candidate coding area corresponding to the coding unit in each candidate reference frame; calculate the pixel similarity between the coding unit and each candidate coding area, and select a reference frame of the coding unit from the multiple candidate reference frames according to the pixel similarity calculation result.

[0156] The video processing device provided in this embodiment can adaptively decide whether to skip affine motion estimation in the video encoding stage, thereby improving the encoding efficiency under the premise of ensuring video quality. When the encoding unit in the current encoding stage corresponds to the target time domain layer, the proportion information of the affine encoding unit of the reference frame of the encoding unit can be calculated, so as to reflect the influence range of the affine motion estimation in the video frame encoding stage through the proportion information of the affine encoding unit. When the proportion information is less than the preset proportion threshold and the encoding unit does not select the affine merge mode, it means that the influence is small. Even if the affine motion estimation is used for subsequent encoding processing, the improvement of the video quality is limited. Therefore, the affine motion estimation processing of the encoding unit can be skipped, and only the non-affine motion estimation processing can be performed, so as to effectively improve the video encoding efficiency under the premise of ensuring the visual quality of the video.

[0157] The above is a schematic scheme of a video processing device of this embodiment. It should be noted that the technical scheme of the video processing device and the technical scheme of the above video processing method belong to the same concept, and the details not described in detail in the technical scheme of the video processing device can be referred to the description of the technical scheme of the above video processing method.

[0158] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0159] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0160] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0161] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0162] The processor 820 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned video processing method when executed by the processor.

[0163] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above video processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above video processing method.

[0164] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned video processing method when executed by a processor.

[0165] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above video processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above video processing method.

[0166] An embodiment of the present specification further provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned video processing method when executed by a processor.

[0167] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the above video processing method belong to the same concept, and the details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above video processing method.

[0168] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0170] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0171] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A video processing method, characterized in that include: When the coding unit corresponds to the target temporal layer, calculating the proportion information of the affine coding unit of the reference frame of the coding unit; When the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode, the affine motion estimation process of the encoding unit is skipped.

2. The video processing method according to claim 1, characterized in that: Before the step of calculating the proportion information of the affine coding unit of the reference frame of the coding unit in the case where the coding unit corresponds to the target temporal layer is performed, the step further includes: Determine the coding unit to be detected corresponding to the current coding moment; Loading unit identification information corresponding to the coding unit to be detected, and detecting whether the coding unit to be detected meets the affine motion estimation skip decision condition according to the unit identification information; If so, determine that the time domain layer corresponding to the coding unit to be detected is the target time domain layer, and use the coding unit to be detected as the coding unit.

3. The video processing method according to claim 1, characterized in that: The calculating the proportion information of the affine coding unit of the reference frame of the coding unit includes: Traversing a plurality of initial coding units in a reference frame of the coding unit; An initial coding unit for affine motion estimation is selected according to the traversal result as the affine coding unit, and the proportion information of the affine coding unit is calculated.

4. The video processing method according to claim 1, characterized in that: Before the step of calculating the proportion information of the affine coding unit of the reference frame of the coding unit in the case where the coding unit corresponds to the target temporal layer is performed, the step further includes: Acquire a video clip, and obtain a video frame sequence by performing frame processing on the video clip; Dividing the video frames included in the video frame sequence, and determining at least two time domain layers according to the division result, wherein the video frames included in each time domain layer constitute the video frame sequence; When encoding is performed on video frames contained in each time domain layer and the time domain layer corresponding to the coding unit processed in the current encoding stage is the target time domain layer, a step of calculating proportion information of the affine coding unit of the reference frame of the coding unit is performed.

5. The video processing method according to claim 3, characterized in that: In the case where there are multiple affine coding units, calculating the proportion information of the affine coding units includes: Sum the affine area information of each affine coding unit to obtain local affine area information, determine the global area information of the reference frame, calculate the area ratio information of the local affine area information relative to the global area information, and use the area ratio information as the ratio information of the affine coding unit; or Determine the initial unit quantity of the multiple initial coding units and the affine unit quantity of the multiple affine coding units, calculate the quantity ratio information of the affine unit quantity relative to the initial unit quantity, and use the quantity ratio information as the ratio information of the affine coding unit.

6. The video processing method according to claim 2, characterized in that: Also includes: When the coding unit to be detected does not meet the affine motion estimation skip decision condition, or the proportion information is greater than or equal to the preset proportion threshold, or the coding unit selects the affine merge mode, non-affine motion estimation processing and affine motion estimation processing are performed on the coding unit.

7. The video processing method according to any one of claims 1 to 5, characterized in that: Before the affine motion estimation processing step of skipping the coding unit is performed, the method further includes: Determining a target prediction mode set corresponding to the coding unit; By traversing the target prediction mode set, determining rate-distortion cost information corresponding to each prediction mode in the target prediction mode set; A target prediction mode is selected from the target prediction mode set according to rate-distortion cost information corresponding to each prediction mode, and when the target prediction mode is not an affine merge mode, an affine motion estimation processing step of skipping the coding unit is performed.

8. The video processing method according to any one of claims 1 to 5, characterized in that: The reference frame determination of the coding unit includes: Determine a plurality of candidate reference frames, and determine a candidate coding region corresponding to the coding unit in each candidate reference frame; Calculate the pixel similarity between the coding unit and each candidate coding region, and select a reference frame of the coding unit from the multiple candidate reference frames according to the pixel similarity calculation result.

9. A video processing device, characterized in that: include: A calculation information module, configured to calculate the proportion information of the affine coding unit of the reference frame of the coding unit when the coding unit corresponds to the target temporal layer; The skip processing module is configured to skip the affine motion estimation processing of the encoding unit when the proportion information is less than a preset proportion threshold and the encoding unit does not select the affine merge mode.

10. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images