Frame structure determination method and device, electronic equipment, storage medium and program product
By comparing the frame length to be encoded with the image group length, adaptively determining the frame structure decision scheme, the problem of slow decision-making speed of adaptive frame structure is solved, and video encoding speed and throughput are improved.
Patent Information
- Application Number
- CN202411571646.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-01-14
AI Technical Summary
Adaptive frame structure decision-making schemes require searching for multiple possible frame structures during video encoding, resulting in a reduced encoding speed.
By comparing the frame length to be encoded with the image group length, the frame structure decision scheme is adaptively determined, and different frame structure decision thresholds and frame structure search strategies are adopted to quickly determine the target frame structure.
It improves encoding speed, improves the encoder throughput, makes the frame decision scheme more flexible and has almost no impact on encoding performance.
Smart Images
Figure CN119316607A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video processing technology, and in particular to a frame structure determination method, a frame structure determination device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] During the video encoding process, the frame structure in the encoder determines the type of video frame (such as I / P / B frame), the temporal level of each video frame, available reference frames, etc. In related video reference software, video frames can be encoded using a fixed frame structure, such as in the order of IBBBBBBBPBBBBBBBP, that is, the position of I frame / P frame is fixed. Since B frames of fixed length are inserted between adjacent P frames, it is called a fixed frame structure.
[0003] However, subjective quality is likely to decline when video scenes switch or motion is intense. Therefore, an adaptive frame structure is often used to optimize subjective quality. That is, the position of the I / P frame is determined according to the video content. Since B frames of variable length are inserted between adjacent P frames, it is called an adaptive frame structure. Summary of the invention
[0004] The present disclosure provides a frame structure determination method, device, electronic device, computer-readable storage medium and computer program product to at least solve the problem in the related art that the adaptive frame structure decision solution needs to search and calculate multiple possible frame structures, which makes the whole process slow and leads to a reduction in encoding speed. The technical solution of the present disclosure is as follows:
[0005] According to a first aspect of an embodiment of the present disclosure, a frame structure determination method is provided, comprising: determining the length of a to-be-encoded frame of a to-be-encoded video, and obtaining the image group length of a preset image group used for video encoding; determining a scheme decision result based on a size comparison result between the to-be-encoded frame length and the image group length; determining a target frame structure of the to-be-encoded video based on the scheme decision result, and the target frame structure is used for video encoding.
[0006] In an exemplary embodiment of the present disclosure, the scheme decision result is determined based on the comparison result between the length of the frame to be encoded and the length of the image group, including: determining a frame structure decision threshold based on the image group length; when the length of the frame to be encoded is less than the frame structure decision threshold, determining the scheme decision result as a first decision result; when the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the image group length, determining the scheme decision result as a second decision result; when the length of the frame to be encoded is greater than the image group length, determining the scheme decision result as a third decision result.
[0007] In an exemplary embodiment of the present disclosure, determining the target frame structure of the video to be encoded according to the scheme decision result includes: when the scheme decision result is a first decision result, determining a first initial frame structure corresponding to the video to be encoded; determining a first encoding cost corresponding to a plurality of the first initial frame structures; and determining the target frame structure according to the first encoding cost.
[0008] In an exemplary embodiment of the present disclosure, determining the first initial frame structure corresponding to the video to be encoded includes: determining a frame length selection interval based on the length of the frame to be encoded and the length of the image group; determining a first basic frame structure based on the frame length selection interval; and performing a frame filling operation based on the first basic frame structure to obtain the first initial frame structure.
[0009] In an exemplary embodiment of the present disclosure, determining the target frame structure of the video to be encoded according to the scheme decision result includes: when the scheme decision result is the second decision result, determining the initial basic frame structure based on the length of the frame to be encoded; determining the number of frames of a specified type in the initial basic frame structure; when the number of frames of the specified type meets the frame structure specified condition, performing a frame structure search based on a first frame length interval to determine the target frame structure.
[0010] In an exemplary embodiment of the present disclosure, the frame structure search is performed based on the first frame length interval to determine the target frame structure, including: performing a frame structure search based on the length of the frame to be encoded and the first frame length interval to determine a second basic frame structure; performing a frame filling operation on the second basic frame structure to obtain a second initial frame structure; determining a second coding cost corresponding to each of the second initial frame structures, and determining the target frame structure based on the second coding cost.
[0011] In an exemplary embodiment of the present disclosure, determining the target frame structure of the video to be encoded according to the scheme decision result includes: when the scheme decision result is a third decision result, obtaining a second frame length interval, and determining a third basic frame structure according to the frame length to be encoded and the second frame length interval; performing a frame filling operation on the third basic frame structure to obtain a third initial frame structure; determining a third encoding cost corresponding to each of the third initial frame structures, and determining the target frame structure according to the third encoding cost.
[0012] According to the second aspect of the embodiments of the present disclosure, a frame structure determination device is provided, including: a length determination module, used to determine the length of a frame to be encoded of a video to be encoded, and obtain the image group length of a preset image group used for video encoding; a scheme decision module, used to determine a scheme decision result based on a comparison result between the length of the frame to be encoded and the image group length; a frame structure determination module, used to determine a target frame structure of the video to be encoded based on the scheme decision result, and the target frame structure is used for video encoding.
[0013] In an exemplary embodiment of the present disclosure, the scheme decision module includes a scheme decision unit, which is used to: determine a frame structure decision threshold according to the image group length; when the length of the frame to be encoded is less than the frame structure decision threshold, determine the scheme decision result as a first decision result; when the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the image group length, determine the scheme decision result as a second decision result; when the length of the frame to be encoded is greater than the image group length, determine the scheme decision result as a third decision result.
[0014] In an exemplary embodiment of the present disclosure, the frame structure determination module includes a first frame structure determination unit, which is used to: when the scheme decision result is the first decision result, determine the first initial frame structure corresponding to the video to be encoded; determine the first encoding costs corresponding to multiple first initial frame structures; and determine the target frame structure based on the first encoding cost.
[0015] In an exemplary embodiment of the present disclosure, the first frame structure determination unit includes a first frame structure determination subunit, which is used to: determine a frame length selection interval based on the length of the frame to be encoded and the image group length; determine a first basic frame structure based on the frame length selection interval; and perform a frame filling operation based on the first basic frame structure to obtain the first initial frame structure.
[0016] In an exemplary embodiment of the present disclosure, the frame structure determination module includes a second frame structure determination unit, including: when the scheme decision result is the second decision result, determining the initial basic frame structure based on the length of the frame to be encoded; determining the number of specified type frames in the initial basic frame structure; when the number of specified type frames meets the frame structure specified condition, performing a frame structure search based on the first frame length interval to determine the target frame structure.
[0017] In an exemplary embodiment of the present disclosure, the second frame structure determination unit includes a second frame structure determination subunit, which is used to: perform a frame structure search based on the interval between the frame length to be encoded and the first frame length to determine a second basic frame structure; perform a frame filling operation on the second basic frame structure to obtain a second initial frame structure; determine a second coding cost corresponding to each of the second initial frame structures, and determine the target frame structure based on the second coding cost.
[0018] In an exemplary embodiment of the present disclosure, the frame structure determination module includes a third frame structure determination unit, which is used to: when the scheme decision result is the third decision result, obtain the second frame length interval, and determine the third basic frame structure according to the frame length to be encoded and the second frame length interval; perform a frame filling operation on the third basic frame structure to obtain a third initial frame structure; determine the third coding cost corresponding to each of the third initial frame structures, and determine the target frame structure according to the third coding cost.
[0019] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement any one of the frame structure determination methods described above.
[0020] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the frame structure determination methods described above.
[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements any one of the frame structure determination methods described above when executed by a processor.
[0022] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: on the one hand, by comparing the length of the frame to be encoded with the length of the group of pictures, the frame structure decision scheme is adaptively determined according to different comparison results, making the frame decision scheme more flexible. On the other hand, the adaptive frame structure decision scheme can greatly improve the encoding speed while hardly affecting the encoding performance, thereby improving the throughput of the encoder.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0025] Figure 1 The figure is a flowchart of a method for determining a frame structure according to an exemplary embodiment.
[0026] Figure 2 It is an overall flow chart of an adaptive frame structure decision solution according to an exemplary embodiment.
[0027] Figure 3 The diagram is a schematic diagram showing an optimal subpath under a certain video frame length according to an exemplary embodiment.
[0028] Figure 4 The figure is a schematic diagram showing frame structure search based on frame length interval according to an exemplary embodiment.
[0029] Figure 5 The invention is a block diagram showing a device for determining a frame structure according to an exemplary embodiment.
[0030] Figure 6 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0033] In the process of video encoding, adaptive frame structures are often used to optimize subjective quality. For example, in a related scheme, a trellis algorithm is used to adaptively determine the position of a P frame, which requires searching and calculating multiple possible frame structures and selecting the optimal frame structure.
[0034] The adaptive frame structure decision scheme adopts a trellis-based path search method. The frame type of a video frame of length N needs to be determined by searching for the optimal structure of video frames of length 1 to N-1 in sequence, and each length of video frame needs to search for multiple possible frame structures with a maximum number of group of pictures (GOP) length. The whole process is slow, which greatly reduces the encoding speed.
[0035] Based on this, according to an embodiment of the present disclosure, a frame structure determination method, a frame structure determination device, an electronic device, a computer-readable storage medium, and a computer program product are proposed.
[0036] In this article, it is important to understand that the terms involved, such as GOP, are a group of continuous pictures, consisting of an I frame and several B / P frames, and are the basic units accessed by video image encoders and decoders. I frame, also known as key frame or intra-coded frame, is a complete image frame that exists independently of other frames. I frame can be decoded independently without relying on the information of other frames. P frame, that is, forward predictive coded frame or predictive frame (Predictive Frame), depends on the previous I frame or P frame to generate. B frame, bidirectional predictive interpolated coded frame or called bidirectional prediction frame (BidirectionalInterpolated Prediction Frame), needs to refer to the previous and next I frame or P frame to generate.
[0037] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.
[0038] First, the present disclosure provides a frame structure determination method, referring to Figure 1 , Figure 1 is a flowchart of a method for determining a frame structure according to an exemplary embodiment. Figure 1 As shown, the frame structure determination method can be used in a computer device, wherein the computer device described in the present disclosure may include mobile terminal devices such as mobile phones, tablet computers, laptop computers, PDAs, and fixed terminal devices such as desktop computers. This exemplary embodiment uses the method applied to a computer device as an example, and it can be understood that the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is implemented through the interaction between the computer device and the server. Specifically, the following steps are included.
[0039] In step S110, the length of the to-be-encoded frame of the to-be-encoded video is determined, and the length of the group of pictures of the preset group of pictures used for video encoding is obtained.
[0040] In an exemplary embodiment of the present disclosure, the video to be encoded may be a video to be encoded in a video encoding process, and the video to be encoded may be composed of multiple video frames to be encoded, and the video frames to be encoded may include I frames, P frames, and B frames. The length of the frames to be encoded may be the number of video frames to be encoded. The preset image group may be a preset image group, for example, the preset image group may be a GOP in video encoding, or a mini-GOP in video encoding. A mini-GOP is a group of images containing the minimum number of images from the bottom layer to the highest layer, and the size of the mini-GOP is related to the frame reference relationship and the number of time domain layers during video encoding. The image group length may be the number of image frames contained in the preset image group.
[0041] In the video encoding process, the length of the remaining video frames to be encoded can be determined as the length of the frames to be encoded. Since the frame structure decision scheme can be determined according to the frame structure that has been determined by encoding, and the length interval of the frame structure that has been determined by encoding is affected by the length of the preset picture group, the picture group length of the preset picture group used in the video encoding can be obtained.
[0042] In step S120, a scheme decision result is determined based on a comparison result between the length of the frame to be encoded and the length of the group of pictures.
[0043] In an exemplary embodiment of the present disclosure, the size comparison result may be a comparison result of the numerical values of the length of the frame to be encoded and the length of the group of pictures. The scheme decision result may be a classification result of determining the frame structure decision scheme adopted by the remaining frames to be encoded, and the scheme decision result may be determined according to different size comparison results.
[0044] The length of the frame to be encoded is compared with the length of the image group to determine the size comparison result between the two. The size comparison result between the two can reflect the different stages of video encoding. The scheme decision result is determined based on the size comparison result, that is, the corresponding frame structure decision scheme is matched for different encoding stages.
[0045] In step S130, according to the scheme decision result, the target frame structure of the video to be encoded is determined, and the target frame structure is used for video encoding.
[0046] In an exemplary embodiment of the present disclosure, the target frame structure may be a frame structure corresponding to a video frame to be encoded. The frame structure may reflect the arrangement of different video frames in a frame sequence.
[0047] After determining the matching scheme decision result, the target frame structure of the video to be encoded can be determined according to the scheme decision result, and the target frame structure can be used in subsequent video encoding.
[0048] According to the frame structure determination method in this example embodiment, on the one hand, by comparing the length of the frame to be encoded with the length of the group of pictures, the frame structure decision scheme is adaptively determined according to different comparison results, making the frame decision scheme more flexible. On the other hand, the adaptive frame structure decision scheme can greatly improve the encoding speed while hardly affecting the encoding performance, thereby improving the throughput of the encoder.
[0049] The frame structure determination method in this exemplary embodiment will be further described below.
[0050] In an exemplary embodiment of the present disclosure, for step S120, the scheme decision result is determined based on the comparison result between the length of the frame to be encoded and the length of the picture group, including: determining the frame structure decision threshold according to the length of the picture group; when the length of the frame to be encoded is less than the frame structure decision threshold, the scheme decision result is determined as the first decision result; when the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the length of the picture group, the scheme decision result is determined as the second decision result; when the length of the frame to be encoded is greater than the length of the picture group, the scheme decision result is determined as the third decision result.
[0051] The frame structure decision threshold may be a threshold of a frame structure decision scheme used to decide the length of the currently remaining frame to be encoded, and the frame structure decision threshold may be determined according to the length of the group of pictures.
[0052] In order to further optimize the speed of the video encoding process, the present invention configures a frame structure decision threshold, and sets a variety of different frame structure decision constraints based on the frame structure decision threshold. When the length of the frame to be encoded meets different constraints, the frame structure decision scheme under the constraint will be used to determine the frame structure.
[0053] Specifically, the frame structure decision threshold is first determined according to the length of the picture group. For example, the preset picture group in this embodiment can be a mini-GOP or a GOP. The length of the preset picture group, i.e., the picture group length, will be represented by GOP below. For example, the present disclosure uses GOP / 2 and GOP as segmentation points and configures two different frame structure decision thresholds.
[0054] Then, different frame structure decision constraints are set according to the frame structure decision threshold and the image group length. Figure 2 , Figure 2The figure is an overall flow chart of an adaptive frame structure decision scheme according to an exemplary embodiment. If the length of the frame to be encoded is less than the frame structure decision threshold, the first decision condition is met. Under this condition, it can be considered that the length of the remaining video frames to be encoded is short, or the encoding operation has reached the end of the sequence. At this time, the scheme decision result can be determined as the first decision result, and the first frame structure decision scheme will be used to determine the target frame structure in the future.
[0055] If the first decision condition is not met, the judgment continues. If the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the length of the image group, the second decision condition is met. Under this condition, although the length of the remaining video frames to be encoded is greater than the length of the frames to be encoded under the above constraints, it is still considered that the length of the video to be encoded itself is shorter. At this time, the scheme decision result can be determined as the second decision result, and the second frame structure decision scheme will be used to determine the target frame structure subsequently.
[0056] If the second decision condition is not met, the judgment is continued. If the length of the frame to be encoded is greater than the length of the image group, the third decision condition is met. Under this condition, it is considered that the length of the remaining video frame to be encoded is longer. At this time, the scheme decision result is determined as the third decision result, and the third frame structure decision scheme will be used to determine the target frame structure. By using the corresponding frame structure decision scheme for different frame lengths to be encoded, the flexibility of the frame structure decision can be further improved to increase the encoding speed.
[0057] It is easy for those skilled in the art to understand that different frame structure decision schemes are adopted according to different video frame lengths. The frame structure decision thresholds mentioned above use GOP / 2 and GOP as segmentation points. Other segmentation points may also be used in some other exemplary embodiments of the present disclosure, such as GOP / 3 or GOP / 4 as segmentation points, and the number of frame structure decision thresholds may be multiple, and the present disclosure does not make any special limitation on this.
[0058] In an exemplary embodiment of the present disclosure, for step S130, the target frame structure of the video to be encoded is determined according to the scheme decision result, including: when the scheme decision result is the first decision result, determining the first initial frame structure corresponding to the video to be encoded; determining the first encoding cost corresponding to multiple first initial frame structures; and determining the target frame structure according to the first encoding cost.
[0059] The first initial frame structure may be a plurality of frame structures determined according to an optimal frame structure whose length is less than the length of the frame to be encoded. The first encoding cost may be a cost generated by using the first initial frame structure for video encoding.
[0060] When the scheme decision result is the first decision result, that is, the length of the video frame to be encoded, that is, the length of the frame to be encoded N<=GOP / 2, the target frame structure can still be determined according to the frame structure decision scheme based on the trellis algorithm. Specifically, the first initial frame structure corresponding to the video to be encoded is determined according to the length of the frame to be encoded, and the first initial frame structure can be determined according to the optimal frame structure of the encoded frame that has been determined.
[0061] After obtaining multiple first initial frame structures, the first coding cost corresponding to each possible first initial frame structure is calculated, for example, the coding cost is determined according to the minimum Sum of Absolute Transformed Difference (SATD) corresponding to each first initial frame structure, and the first initial frame structure with the minimum first coding cost is used as the target frame structure. Under the first decision result, the frame structure decision scheme is adopted to quickly complete the frame structure determination operation.
[0062] In an exemplary embodiment of the present disclosure, a first initial frame structure corresponding to a video to be encoded is determined, including: determining a frame length selection interval according to the length of the frame to be encoded and the length of the image group; determining a first basic frame structure according to the frame length selection interval; and performing a frame filling operation based on the first basic frame structure to obtain a first initial frame structure.
[0063] The frame length selection interval may be a value interval corresponding to an optional frame length in a frame structure search space. The first basic frame structure may be an optimal frame structure of an encoded frame that can be used as a reference for a frame to be encoded, determined based on the frame length selection interval. The frame filling operation may be an operation of filling a video frame into the frame structure based on the first basic frame structure. The first initial frame structure may be all possible frame structures obtained after performing the frame filling operation.
[0064] The process of determining the first initial frame structure can be implemented based on the adaptive frame structure solution of trellis, that is, the frame structure of the video frame of the current length is determined based on the frame structure that has been determined previously. If the length of the video frame is N, the frame type of the first frame is specified as an I frame and the frame type of the last frame is a P frame in the encoding, then the frame type decision process for the video frame length of N is as follows:
[0065] When the video frame length is 1, the frame structure is I.
[0066] When the video frame length is 2, the frame structure is IP.
[0067] When the video frame length is 3, the frame structure is determined according to the frame structure of length 1 / 2, which may be IPP / IBP. The optimal frame structure of length 2 is selected from the above possible frame structures, and the selection criterion is the minimum coding cost, where the coding cost is the sum of the SATD costs of all video frames after the I frame. Here, it is assumed that the optimal frame structure is IBP.
[0068] When the video frame length is 4, the frame structure is determined according to the optimal frame structure of length 1 / 2 / 3, which may be IBBP / IPBP / IBPP. Here, it is assumed that the optimal frame structure is IBBP.
[0069] When the video frame length is 5, the frame structure is determined according to the optimal frame structure of length 1 / 2 / 3 / 4, which may be IBBBP / IPBBP / IBPBP / IBBPP. Here, it is assumed that the optimal frame structure is IBBBP.
[0070] When the video frame length is N, in the frame structure decision scheme, the frame length interval selectable by the frame structure decision can be configured first, that is, the frame length selection interval. The frame length selection interval can be determined according to the length of the frame to be encoded and the length of the group of pictures. For example, when the frame decision scheme is recursively implemented, the frame length selection interval can be determined as [N-GOP, N-1]. The first basic frame structure is determined according to the frame length selection interval. The first basic frame structure is determined according to the optimal frame structure with a length between (N-GOP) and (N-1), where GOP is the preset, longest mini-GOP length, such as 8 / 16 / 32 / 64.
[0071] Specifically, after the optimal frame structure of length X (i.e., the first basic frame structure) is determined, a frame filling operation is performed to fill the first basic frame structure with B frames until N video frames are reached (the frame type of the last video frame is a P frame), and the first initial frame structure is obtained, where X∈[N–GOP,N-1]. Then, the optimal frame structure of the video frame of length N is determined according to the SATD cost of the possible frame structures of this GOP, i.e., the target frame structure. Through the above frame structure determination scheme, all possible frame structures of the frame to be encoded can be determined according to the frame structure of length X, so as to subsequently determine the target frame structure.
[0072] In an exemplary embodiment of the present disclosure, for step S130, the target frame structure of the video to be encoded is determined according to the scheme decision result, including: when the scheme decision result is the second decision result, determining the initial basic frame structure based on the length of the frame to be encoded; determining the number of frames of a specified type in the initial basic frame structure; when the number of frames of the specified type meets the specified condition of the frame structure, performing a frame structure search based on the first frame length interval to determine the target frame structure.
[0073] The initial basic frame structure may be a basic true structure determined in the second frame decision scheme and available for use by the frame to be encoded, and the initial basic frame structure is an optimal frame structure under the length of some frames to be encoded. The number of frames of a specified type may be the number of video frames of a certain specific type, for example, the number of P frames in the frame structure. The frame structure specifying condition may be a predefined condition for determining whether the frame structure meets the corresponding characteristics. The frame interval length may be a search interval for the basic frame structure used when constructing the frame structure search space. The frame structure search may be a process of searching for a basic frame structure from encoded frames of various lengths.
[0074] When the scheme decision result is the second decision result, that is, when the length of the frame to be encoded N>GOP / 2 and N<=GOP, it is still necessary to first determine the basic frame structure corresponding to the frame to be encoded. The initial basic frame structure can be determined according to the length of the frame to be encoded. The initial basic frame structure of the length of the frame to be encoded N can still be determined according to the frame structure that has been determined. Specifically, the initial basic frame structure can also be determined by an adaptive frame structure scheme based on the trellis algorithm. For example, the optimal frame structure with a video frame length of N-1 is used as the initial basic frame structure.
[0075] After the initial basic frame structure is determined, the frame structure characteristics of the initial basic frame structure are analyzed to adopt a matching frame structure decision scheme according to different frame structure characteristics. For example, the number of specified type frames in the initial basic frame structure is determined. In this embodiment, the number of P frames in the initial basic frame structure can be determined, and then it is determined whether the number of P frames meets the frame structure specified condition, and the corresponding frame structure decision scheme is determined according to different judgment results.
[0076] Then, a target frame structure is constructed based on the first frame length interval. For example, the initial basic frame structure may be an optimal frame structure with a video frame length of N-1. If there is only one P frame in the optimal frame structure with a video frame length of N-1, then when the current video frame length is N, it is only necessary to determine the optimal path of the current length video frame based on some optimal subpaths, such as determining the target frame structure based on the frame filling interval length. According to the definition of the trellis algorithm, the optimal path may be determined by the position of the P frame in the frame structure, and the optimal path may be obtained by connecting the positions of all P frames in the frame structure. Reference Figure 3 , Figure 3 is a schematic diagram showing an optimal subpath under a certain video frame length according to an exemplary embodiment. For example, Figure 3 The optimal path of the frame structure in may be 4->8->12->16.
[0077] If the optimal frame structure with a video frame length of N-1 does not satisfy the condition of having only one P frame, the frame structure decision scheme based on the trellis algorithm is still used to determine the target frame structure. By analyzing the initial basic frame structure and determining the frame decision scheme, it is possible to prevent too many frames of the specified type from being inserted into the target frame structure, avoid excessive coding losses during the video coding process, and improve coding efficiency.
[0078] In an exemplary embodiment of the present disclosure, a frame structure search is performed based on a first frame length interval to determine a target frame structure, including: performing a frame structure search based on the length of a frame to be encoded and the first frame length interval to determine a second basic frame structure; performing a frame filling operation on the second basic frame structure to obtain a second initial frame structure; determining a second coding cost corresponding to each second initial frame structure, and determining the target frame structure based on the second coding cost.
[0079] When determining the target frame structure, the first frame length interval may be obtained first. For example, the first frame length interval may be determined to be 2. Then, when searching for the basic frame structure, the second basic frame structure may be obtained based on the frame structure with video frame lengths of N-1, N-3, N-5, etc. Then, a frame filling operation is performed on the second basic frame structure, such as filling consecutive B frames and the last P frame into the obtained second basic frame structure, and a possible frame structure of the video frame with a current frame length of N to be encoded is determined as the second initial frame structure.
[0080] After obtaining multiple second initial frame structures, the second coding cost corresponding to each second initial frame structure is determined, and the frame structure with the minimum coding cost is used as the optimal frame structure, that is, the target frame structure. The performance index used when determining the second coding cost can be the same as the performance index used for the first coding cost, and this disclosure will not be repeated. Performing a frame filling operation based on the frame filling interval length to determine a possible frame structure can reduce the frame structure search space, thereby increasing the encoding speed.
[0081] In an exemplary embodiment of the present disclosure, for step S130, the target frame structure of the video to be encoded is determined according to the scheme decision result, including: when the scheme decision result is the third decision result, the second frame length interval is obtained, and the third basic frame structure is determined according to the frame length to be encoded and the second frame length interval; a frame filling operation is performed on the third basic frame structure to obtain a third initial frame structure; a third encoding cost corresponding to each third initial frame structure is determined, and the target frame structure is determined according to the third encoding cost.
[0082] When the solution decision result is the third decision result, that is, when the length N of the frame to be encoded is greater than the GOP, only the frame structure of a partial length of the video frame needs to be determined as the basic frame structure. For example, the third basic frame structure is determined according to a pre-configured second frame length interval. For example, if the second frame padding interval length is determined to be 4, then when searching for the basic frame structure, the frame structures with video frame lengths of N, N - 4, N - 8, etc. can be used to obtain the third basic frame structure. Refer to Figure 4 , Figure 4 is a schematic diagram showing frame structure search based on frame length intervals according to an exemplary embodiment. Figure 4 shows the frame lengths that need to be used as the third basic frame structure when performing frame structure search according to a frame padding interval length of 4.
[0083] Then, a frame padding operation is performed on the third basic frame structure. For example, continuous B frames and the last P frame are filled into the obtained third basic frame structure to determine the possible frame structures of the video frame with the current length N of the frame to be encoded as the third initial frame structure. The process of determining the third initial frame structure includes the following steps: When determining the frame structure of a video frame with a length X (where X > GOP and X < N), based on the video frames for which the optimal paths have been determined, such as the frame structures with a video frame length interval of 4 (X - 4 / X - 8 / N – 12...), and then a certain number of B frames and the last P frame are filled to determine the possible frame structures of the video frame with the current length X. The above operation is recursively executed until the third initial frame structure with the length N of the frame to be encoded is determined.
[0084] After obtaining multiple third initial frame structures, the third coding cost corresponding to each third initial frame structure is determined, and the third initial frame structure with the minimum coding cost is used as the optimal frame structure, that is, the target frame structure. The performance metrics used when determining the third coding cost can be the same as those used for the first coding cost and the second coding cost, and the present disclosure will not elaborate on this.
[0085] Through the above frame structure decision scheme, in the case of a long frame to be encoded, only the optimal frame structures of a partial video length need to be decided, and when deciding the optimal frame structure of a certain length, only the partial optimal sub-structures are needed to determine the possible frame structures. In contrast, the related frame structure decision schemes need to search for the optimal frame structures one by one for each length, and when deciding the optimal frame structure of a certain length, it is determined one by one according to the GOP optimal sub-structures. Therefore, this scheme reduces the frame structure search space and speeds up the encoding speed.
[0086] It should be noted that in the above frame structure decision scheme, the frame structure of the current length is determined based on the partial length frame structure, and the length of the partial frame structure can be separated by 2, 4, or other values, which can be determined based on the cost-effectiveness of performance loss and speed improvement. In order to speed up the encoding speed without losing the encoding performance as much as possible, the present disclosure can configure the second frame filling interval length to be greater than or equal to the first frame filling interval length.
[0087] In summary, the frame structure determination method disclosed in the present invention determines the frame length to be encoded of the video to be encoded, obtains the image group length of the preset image group used for video encoding; determines the scheme decision result according to the size comparison result of the frame length to be encoded and the image group length; determines the target frame structure of the video to be encoded according to the scheme decision result, and the target frame structure is used for video encoding. On the one hand, by comparing the frame length to be encoded with the image group length, the frame structure decision scheme is adaptively determined according to different comparison results, making the frame decision scheme more flexible. On the other hand, the adaptive frame structure decision scheme can greatly improve the encoding speed while hardly affecting the encoding performance, thereby improving the throughput of the encoder. On the other hand, in the frame structure decision scheme, it is only necessary to determine the optimal frame structure based on the length of the partial video, and when deciding the optimal frame structure of a certain length, it is only necessary to determine the possible frame structure based on part of the optimal substructure, thereby reducing the frame structure search space.
[0088] Figure 5 is a block diagram of a frame structure determination device according to an exemplary embodiment. Figure 5 The frame structure determination device 500 includes: a length determination module 510, a scheme decision module 520 and a frame structure determination module 530.
[0089] Specifically, the length determination module 510 is used to determine the length of the frame to be encoded of the video to be encoded, and obtain the image group length of the preset image group used for video encoding; the scheme decision module 520 is used to determine the scheme decision result based on the comparison result between the length of the frame to be encoded and the image group length; the frame structure determination module 530 is used to determine the target frame structure of the video to be encoded based on the scheme decision result, and the target frame structure is used for video encoding.
[0090] In an exemplary embodiment of the present disclosure, the scheme decision module 520 includes a scheme decision unit, which is used to: determine a frame structure decision threshold according to the image group length; when the length of the frame to be encoded is less than the frame structure decision threshold, determine the scheme decision result as the first decision result; when the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the image group length, determine the scheme decision result as the second decision result; when the length of the frame to be encoded is greater than the image group length, determine the scheme decision result as the third decision result.
[0091] In an exemplary embodiment of the present disclosure, the frame structure determination module 530 includes a first frame structure determination unit, which is used to: when the scheme decision result is the first decision result, determine the first initial frame structure corresponding to the video to be encoded; determine the first encoding cost corresponding to multiple first initial frame structures; and determine the target frame structure based on the first encoding cost.
[0092] In an exemplary embodiment of the present disclosure, the first frame structure determination unit includes a first frame structure determination subunit, which is used to: determine a frame length selection interval based on the length of the frame to be encoded and the length of the image group; determine a first basic frame structure based on the frame length selection interval; perform a frame filling operation based on the first basic frame structure to obtain a first initial frame structure.
[0093] In an exemplary embodiment of the present disclosure, the frame structure determination module 530 includes a second frame structure determination unit, including: when the scheme decision result is the second decision result, determining the initial basic frame structure based on the length of the frame to be encoded; determining the number of specified type frames in the initial basic frame structure; when the number of specified type frames meets the frame structure specified condition, performing a frame structure search based on the first frame length interval to determine the target frame structure.
[0094] In an exemplary embodiment of the present disclosure, the second frame structure determination unit includes a second frame structure determination subunit, which is used to: perform a frame structure search based on the interval between the length of the frame to be encoded and the first frame length to determine the second basic frame structure; perform a frame filling operation on the second basic frame structure to obtain a second initial frame structure; determine the second coding cost corresponding to each second initial frame structure, and determine the target frame structure based on the second coding cost.
[0095] In an exemplary embodiment of the present disclosure, the frame structure determination module 530 includes a third frame structure determination unit, which is used to: when the scheme decision result is the third decision result, obtain the second frame length interval, and determine the third basic frame structure according to the frame length to be encoded and the second frame length interval; perform a frame filling operation on the third basic frame structure to obtain a third initial frame structure; determine the third coding cost corresponding to each third initial frame structure, and determine the target frame structure according to the third coding cost.
[0096] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0097] Reference below Figure 6 hereinafter describes an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0098] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.
[0099] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.
[0100] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .
[0101] The storage unit 620 may include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0102] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0103] The electronic device 600 may also communicate with one or more external devices 670 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0104] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above frame structure determination method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0105] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, any one of the frame structure determination methods described above is implemented.
[0106] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0107] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for determining a frame structure, characterized in that: include: Determine the frame length of the video to be encoded, and obtain the picture group length of the preset picture group used for video encoding; Determining a solution decision result according to a comparison result between the length of the frame to be encoded and the length of the group of pictures; According to the scheme decision result, a target frame structure of the video to be encoded is determined, and the target frame structure is used for video encoding.
2. The method according to claim 1, characterized in that The determining of a solution decision result according to a comparison result between the length of the frame to be encoded and the length of the group of pictures includes: Determining a frame structure decision threshold according to the length of the group of pictures; When the length of the frame to be encoded is less than the frame structure decision threshold, determining the scheme decision result as the first decision result; When the length of the frame to be encoded is greater than the frame structure decision threshold, and the length of the frame to be encoded is less than or equal to the length of the group of pictures, determining the scheme decision result as the second decision result; When the length of the frame to be encoded is greater than the length of the group of pictures, the scheme decision result is determined as the third decision result.
3. The method according to claim 1 or 2, characterized in that: Determining the target frame structure of the video to be encoded according to the decision result of the scheme includes: When the scheme decision result is a first decision result, determining a first initial frame structure corresponding to the video to be encoded; Determine first encoding costs corresponding to a plurality of the first initial frame structures; The target frame structure is determined according to the first encoding cost.
4. The method according to claim 3, characterized in that The determining a first initial frame structure corresponding to the video to be encoded includes: Determining a frame length selection interval according to the to-be-encoded frame length and the picture group length; Determine a first basic frame structure according to the frame length selection interval; A frame filling operation is performed based on the first basic frame structure to obtain the first initial frame structure.
5. The method according to claim 1 or 2, characterized in that: Determining the target frame structure of the video to be encoded according to the decision result of the scheme includes: When the scheme decision result is the second decision result, determining the initial basic frame structure based on the length of the frame to be encoded; Determining the number of frames of a specified type in the initial basic frame structure; When the number of frames of the specified type meets the frame structure specified condition, a frame structure search is performed based on the first frame length interval to determine the target frame structure.
6. The method according to claim 5, characterized in that The performing frame structure search based on the first frame length interval to determine the target frame structure includes: Perform frame structure search according to the interval between the to-be-encoded frame length and the first frame length to determine a second basic frame structure; Performing a frame filling operation on the second basic frame structure to obtain a second initial frame structure; Determine a second encoding cost corresponding to each of the second initial frame structures, and determine the target frame structure according to the second encoding cost.
7. The method according to claim 1 or 2, characterized in that: Determining the target frame structure of the video to be encoded according to the decision result of the scheme includes: When the scheme decision result is a third decision result, obtaining a second frame length interval, and determining a third basic frame structure according to the frame length to be encoded and the second frame length interval; Performing a frame filling operation on the third basic frame structure to obtain a third initial frame structure; A third encoding cost corresponding to each of the third initial frame structures is determined, and the target frame structure is determined according to the third encoding cost.
8. A frame structure determination device, characterized in that: include: The length determination module is used to determine the length of the to-be-encoded frame of the to-be-encoded video and obtain the length of the group of pictures of the preset group of pictures used for video encoding; A scheme decision module, used for determining a scheme decision result according to a comparison result between the length of the frame to be encoded and the length of the group of pictures; The frame structure determination module is used to determine the target frame structure of the video to be encoded according to the decision result of the scheme, and the target frame structure is used for video encoding.
9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the frame structure determination method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the frame structure determination method as described in any one of claims 1 to 7.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the frame structure determination method according to any one of claims 1 to 7 is implemented.