Frame type determination method, device, electronic device and storage medium

Abstract

The present disclosure relates to the field of video coding and decoding, and proposes a frame type determination method, device, electronic device and storage medium. The method comprises: calculating the coding information of each type of unknown video frames according to a preset reference relationship, counting the number of video frames corresponding to the calculated coding information, and when the coding information of X unknown types of video frames is calculated, determining the forward prediction coding frame from the X video frames according to the coding information of the X video frames, and determining the unknown type of video frames before the forward prediction coding frame in the X video frames as bidirectional prediction interpolation coding frames; outputting the type of the determined video frame, and re-executing the steps of counting the number of video frames corresponding to the calculated coding information and the subsequent steps. The frame type determination method of the embodiment of the present disclosure can reduce the calculation amount and delay of frame type determination and improve the efficiency of frame type determination.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding, and in particular to a frame type determination method, device, electronic device and storage medium. Background Art

[0002] The data volume of the video frame sequence is very large. In order to reduce transmission power consumption and storage cost, the video frame sequence is usually encoded before transmission or storage. In order to obtain a better encoding effect, it is necessary to first determine the type of each video frame in the video frame sequence. Different types of video frames can be encoded in different ways. Video frames can be divided into three types: intra coded picture, forward predictive coded frame, and bi-directional predictive interpolation coded frame.

[0003] Among them, intra-frame coded frames are also called I frames, forward predictive coded frames are also called P frames, and bidirectional predictive interpolation coded frames are also called B frames. I frames refer to video frames that have scene switching with the previous frame (the content of the frame changes significantly). After being encoded, I frames still retain complete data and can be decoded independently without referring to other video frames. Therefore, the encoding method of I frames is intra-frame coding. P frames need to refer to the data of the previous video frame to complete encoding, and also need to refer to the encoding result of the previous video frame when decoding, so the encoding method of P frames is forward predictive coding. B frames need to refer to the data of the previous video frame and the next video frame to complete encoding, and also need to refer to the encoding results of the previous video frame and the next video frame when decoding, so the encoding method of B frames is bidirectional predictive interpolation coding. Usually, I frames are determined first, and then P frames are determined from the remaining frames, and finally B frames are left. Obviously, the accuracy of determining P frames directly affects the accuracy of determining B frames, so the higher the accuracy of determining P frames, the better the video encoding effect.

[0004] The P frame determination method proposed in the prior art has the disadvantages of excessive computational complexity and delay. How to reduce the computational complexity and delay of frame type determination and improve the efficiency of frame type determination has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The frame type determination method of the embodiment of the present disclosure can reduce the calculation amount and delay of frame type determination and improve the efficiency of frame type determination.

[0006] According to one aspect of the present disclosure, a frame type determination method is provided, the method being used to determine the type of a video frame in a video frame sequence, the video frame sequence comprising a plurality of video frames of unknown types, each video frame comprising a plurality of coding blocks, the method comprising: calculating coding information of each unknown type of video frame according to a preset reference relationship, the coding information comprising a coding cost; counting the number of video frames corresponding to the coding information obtained by the calculation, and when the coding information of X video frames of unknown types is obtained by the calculation, determining a forward prediction coding frame from the X video frames according to the coding information of the X video frames, and determining a video frame of unknown type before the forward prediction coding frame among the X video frames as a bidirectional prediction interpolation coding frame, where X represents a preset first number and X is an integer greater than 1; outputting the determined type of the video frame, and re-executing the steps of counting the number of video frames corresponding to the coding information obtained by the calculation and subsequent steps.

[0007] In a possible implementation, the preset reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous video frame of the current video frame as the second reference frame, and the encoding information also includes the proportion of coding blocks using the intra-frame coding mode when referring to the first reference frame. The encoding information of each type of unknown video frame is calculated according to the preset reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of the current video frame referring to the first reference frame or the proportion of coding blocks using the intra-frame coding mode is calculated, and the second coding cost and third coding cost of the current video frame referring to the second reference frame are calculated.

[0008] In a possible implementation, the calculation of the first coding cost of the current video frame with reference to the first reference frame or the proportion of coding blocks using the intra-frame coding mode includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the first reference frame; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the first coding cost according to the sum of the coding costs of each coding block in the adopted coding mode, or obtaining the proportion according to the ratio of the number of coding blocks using the intra-frame coding mode to the total number of coding blocks of the current video frame.

[0009] In a possible implementation, the calculation of the second coding cost and the third coding cost of the current video frame with reference to the second reference frame includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the second reference frame; obtaining the second coding cost according to the sum of the coding costs of each coding block in the inter-frame coding mode; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the third coding cost according to the sum of the coding costs of each coding block in the adopted coding mode.

[0010] In a possible implementation, determining the forward predicted coded frame from the X video frames according to the coding information of the X video frames includes: determining Y candidate video frames from the X video frames according to the first coding cost or the ratio, 1≤Y≤X, and Y is an integer; and determining the forward predicted coded frame from the Y candidate video frames according to the second coding cost and the third coding cost.

[0011] In a possible implementation, the determining Y candidate video frames from the X video frames according to the first encoding cost or the ratio includes: when any one of the X video frames is used as the current video frame, under the condition that the first encoding cost of the current video frame is greater than a first threshold or the ratio corresponding to the current video frame is greater than a second threshold, determining the current video frame as a candidate video frame.

[0012] In a possible implementation, the forward predicted coding frame is determined from the Y candidate video frames based on the second coding cost and the third coding cost, including: when any one of the Y candidate video frames is used as the current video frame, the relative motion complexity and reference redundancy of the current video frame are calculated based on the second coding cost and the third coding cost of the current video frame, the relative motion complexity represents the relative motion degree of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy represents the influence of the coding block using the intra-frame coding mode on the coding cost of the current video frame; when the relative motion complexity of the current video frame is greater than a third threshold and the reference redundancy is greater than a fourth threshold, the current video frame is determined as a forward predicted coding frame.

[0013] In a possible implementation, the relative motion complexity of the current video frame is equal to the difference between the second coding cost of the current video frame and the adjacent video frame; the reference redundancy of the current video frame is equal to the difference between the second coding cost and the third coding cost of the current video frame.

[0014] According to another aspect of the present disclosure, a frame type determination device is provided, the device being used to determine the type of a video frame in a video frame sequence, the video frame sequence comprising a plurality of video frames of unknown type, each video frame comprising a plurality of coding blocks, the device comprising: a calculation module, being used to calculate the coding information of each unknown type of video frame according to a preset reference relationship, the coding information comprising a coding cost; a determination module, being used to count the number of video frames corresponding to the calculated coding information, and when the coding information of X video frames of unknown type is calculated, determining a forward prediction coding frame from the X video frames according to the coding information of the X video frames, and determining the unknown type of video frames before the forward prediction coding frame in the X video frames as a bidirectional prediction interpolation coding frame, where X represents a preset first number and X is an integer greater than 1; and an output module, being used to output the determined type of the video frame, and re-execute the steps of counting the number of video frames corresponding to the calculated coding information and subsequent steps.

[0015] In a possible implementation, the preset reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous video frame of the current video frame as the second reference frame, and the encoding information also includes the proportion of coding blocks using the intra-frame coding mode when referring to the first reference frame. The encoding information of each type of unknown video frame is calculated according to the preset reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of the current video frame referring to the first reference frame or the proportion of coding blocks using the intra-frame coding mode is calculated, and the second coding cost and third coding cost of the current video frame referring to the second reference frame are calculated.

[0016] In a possible implementation, the calculation of the first coding cost of the current video frame with reference to the first reference frame or the proportion of coding blocks using the intra-frame coding mode includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the first reference frame; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the first coding cost according to the sum of the coding costs of each coding block in the adopted coding mode, or obtaining the proportion according to the ratio of the number of coding blocks using the intra-frame coding mode to the total number of coding blocks of the current video frame.

[0017] In a possible implementation, the calculation of the second coding cost and the third coding cost of the current video frame with reference to the second reference frame includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the second reference frame; obtaining the second coding cost according to the sum of the coding costs of each coding block in the inter-frame coding mode; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the third coding cost according to the sum of the coding costs of each coding block in the adopted coding mode.

[0018] In a possible implementation, determining the forward predicted coded frame from the X video frames according to the coding information of the X video frames includes: determining Y candidate video frames from the X video frames according to the first coding cost or the ratio, 1≤Y≤X, and Y is an integer; and determining the forward predicted coded frame from the Y candidate video frames according to the second coding cost and the third coding cost.

[0019] In a possible implementation, the determining Y candidate video frames from the X video frames according to the first encoding cost or the ratio includes: when any one of the X video frames is used as the current video frame, under the condition that the first encoding cost of the current video frame is greater than a first threshold or the ratio corresponding to the current video frame is greater than a second threshold, determining the current video frame as a candidate video frame.

[0020] In a possible implementation, the forward predicted coding frame is determined from the Y candidate video frames based on the second coding cost and the third coding cost, including: when any one of the Y candidate video frames is used as the current video frame, the relative motion complexity and reference redundancy of the current video frame are calculated based on the second coding cost and the third coding cost of the current video frame, the relative motion complexity represents the relative motion degree of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy represents the influence of the coding block using the intra-frame coding mode on the coding cost of the current video frame; when the relative motion complexity of the current video frame is greater than a third threshold and the reference redundancy is greater than a fourth threshold, the current video frame is determined as a forward predicted coding frame.

[0021] In a possible implementation, the relative motion complexity of the current video frame is equal to the difference between the second coding cost of the current video frame and the adjacent video frame; the reference redundancy of the current video frame is equal to the difference between the second coding cost and the third coding cost of the current video frame.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0023] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0025] According to the frame type determination method of the embodiment of the present disclosure, according to a preset reference relationship, coding information of each type of unknown video frame is calculated, the coding information includes a coding cost, and the coding information of each frame is calculated only once, so as to avoid repeated calculation of the coding information, thereby reducing the amount of calculation and the delay caused by calculating the coding information; the number of video frames corresponding to the calculated coding information is counted, and when the coding information of X video frames of unknown type is calculated, a forward prediction coding frame is determined from the X video frames according to the coding information of the X video frames, and the video frame of unknown type before the forward prediction coding frame in the X video frames is determined as a bidirectional prediction interpolation coding frame Frame, that is, after obtaining X pieces of coding information, the frame type can be determined, so that the determination of the frame type and the calculation of the coding information can be carried out in parallel, further reducing the delay of the frame type determination; after the type of the video frame is determined, it is output, and the number of video frames corresponding to the coding information obtained by statistical calculation and the subsequent steps are re-executed, so that the output delay of the determined video frame type is smaller; and the video frame type can be determined once every time the calculated coding information of the video frame of unknown frame type reaches X, until the types of all video frames are determined, so the type of each video frame can be determined for a video frame sequence of any length. In summary, the frame type determination method of the disclosed embodiment can reduce the amount of calculation and delay of frame type determination and improve the efficiency of frame type determination.

[0026] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0028] Figure 1A schematic diagram showing the principle of a frame combination full search algorithm in the prior art.

[0029] Figure 2 An example of frame type combinations searched by the prior art frame combination full search algorithm is shown.

[0030] Figure 3 An example of a reference relationship used by a conventional frame combination full search algorithm when calculating coding costs is shown.

[0031] Figure 4 A schematic diagram showing the flow of a single-frame fast decision algorithm in the prior art.

[0032] Figure 5 An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0033] Figure 6 A schematic diagram showing the process of the frame type determination method according to an embodiment of the present disclosure.

[0034] Figure 7 A schematic diagram showing a video frame sequence according to an embodiment of the present disclosure.

[0035] Figure 8 A schematic diagram showing a low-latency reference relationship according to an embodiment of the present disclosure.

[0036] Fig. 9 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0037] Fig.10 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0041] The following introduces the P frame determination method proposed in the prior art.

[0042] A P frame determination method proposed in the prior art is a frame combination full search algorithm. Figure 1 A schematic diagram showing the principle of a frame combination full search algorithm in the prior art.

[0043] like Figure 1 As shown, the method starts from the most recently determined non-B frame (ie, P frame or I frame) and searches for all possible frame type combinations when the type of each video frame in the video frame sequence is a P frame and a B frame respectively. Figure 2 An example of a frame type combination searched by a frame combination full search algorithm of the prior art is shown. Figure 2 As shown, possible frame type combinations include BBBB, BBBP, BBPB, BBPP, etc. Then, the sum of the coding costs of each video frame under each frame type combination is calculated as the total coding cost of the video frame sequence under the frame type combination. Finally, the total coding costs under each frame type combination are compared, and the frame type combination with the smallest total coding cost is the optimal frame type combination, and the type of each video frame under this frame type combination is the final determined video frame type.

[0044] The frame combination full search algorithm considers the case where the B frame is also used as a reference frame, so it is more complicated when calculating the coding cost. The reference relationship usually adopted is that the P frame itself is used as a reference frame and refers to the previous non-B frame; the B frame between two adjacent non-B frames is used as a reference frame and refers to the previous non-B frame and the next non-B frame; other B frames are not used as reference frames and refer to the previous reference frame and the next reference frame.

[0045] Figure 3 An example of a reference relationship used by the frame combination full search algorithm in the prior art to calculate the coding cost is shown. Figure 3 As shown, assuming that a frame type combination of a video frame sequence is PBBBBBBPBP, the 1st frame, the 8th frame, and the 10th frame are all P frames and are all used as reference frames; the 1st frame refers to the previous non-B frame (not shown), the 8th frame refers to the 1st frame, and the 10th frame refers to the 8th frame; the 1st frame and the 8th frame are adjacent P frames, and the B frame between the two is the 4th frame (the 5th frame can also be selected, Figure 3 Frame 4 is taken as an example), so frame 4 is also used as a reference frame, and refers to frame 1 and frame 8; frame 8 and frame 10 are adjacent P frames, and the B frame between them is frame 9, so frame 9 is also used as a reference frame, and refers to frame 8 and frame 10; frame 2, frame 3, frame 5, frame 6, and frame 7 are all B frames that are not used as reference frames, and frame 2 and frame 3 both refer to frame 1 and frame 4, and frame 5, frame 6, and frame 7 all refer to frame 4 and frame 8.

[0046] The disadvantage of the frame combination full search algorithm is that there are too many frame type combinations that need to calculate the coding cost, and the calculation of the coding cost is very complicated, resulting in a large amount of calculation and a large delay in frame type judgment. Although the Viterbi algorithm is used to improve the frame combination full search algorithm, the number of frame type combinations that need to calculate the coding cost can be reduced and the traversal of all frame type combinations can be avoided, but there are still many frame type combinations to be traversed, and the calculation complexity of the coding cost has not been reduced, and the calculation amount and the delay of frame type judgment are still large.

[0047] In order to speed up the judgment of frame type, the prior art proposes a single-frame fast judgment algorithm. Figure 4 A schematic diagram showing the flow of a single-frame fast decision algorithm in the prior art.

[0048] like Figure 4 As shown, the method first presets a fixed queue length, which is usually equal to the depth of the module that implements the frame type judgment function, and then calculates the coding cost of the first two video frames after the most recently determined non-B frame when the non-B frame is coded with reference to the non-B frame and the frame type combinations are BP and PP respectively, and judges whether the two video frames include P frames according to the calculated coding cost. If it is determined that the two video frames do not include P frames, then for each subsequent video frame, only the inter-frame coding cost of the video frame itself as a P frame and the ratio of the coding blocks in the intra-frame coding mode are calculated, and the ratio is compared with the threshold value. According to the comparison result, it can be judged whether the current frame is a P frame. If it is determined that the two video frames include P frames, after updating the most recently determined non-B frame, the coding cost of the first two video frames after the most recently determined non-B frame when the non-B frame is coded with reference to the non-B frame and the frame type combinations are BP and PP respectively, and the subsequent steps are re-executed. Until the number of video frames of the determined type reaches the preset queue length, the types of the video frames of the determined type are output together.

[0049] This method still needs to calculate the coding cost of the same video frame under different frame type combinations from time to time, so there is a problem of repeated calculation of the coding cost and redundancy in the calculation. Calculating the coding cost of different frame type combinations is not conducive to hardware implementation and is difficult to accelerate using a hardware accelerator. This method will calculate the coding cost when the frame type is a B frame. The forward video frame and backward video frame need to be referenced when encoding the B frame. Therefore, the calculation delay of the coding cost of the B frame will be relatively large, increasing the delay in frame type judgment. The coding cost of a video frame is calculated only when the type of the video frame is judged, and it will not be output until the number of frame types equal to the queue length is judged. The queue length is usually fixed and has a large value, which further increases the delay in frame type judgment.

[0050] Furthermore, this method uses only one threshold to determine the frame type, and the determination result is too absolute, which may easily lead to misjudgment, resulting in excessive bit rate or poor encoding quality.

[0051] In summary, the P frame determination method proposed in the prior art has the disadvantages of excessive computational complexity and delay. How to reduce the computational complexity and delay of frame type determination and improve the efficiency of frame type determination has become a technical problem that needs to be urgently solved in this field.

[0052] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The frame type determination method of the embodiment of the present disclosure can reduce the calculation amount and delay of frame type determination and improve the efficiency of frame type determination.

[0053] Furthermore, the frame type determination method disclosed in the present invention can refer to only the forward video frame when calculating the coding cost, thereby reducing the computational complexity and the time cost of calculating the coding cost. The reference relationship used in calculating the coding cost disclosed in the present invention is convenient for hardware pipeline implementation, and is suitable for a frame type determination method combining software and hardware, further reducing the delay of frame type determination.

[0054] Furthermore, the frame type determination method disclosed in the present invention first determines the candidate video frames whose type may be P frames, and then determines the P frames from the candidate video frames. The two-stage frame type determination method makes the determination of the frame type more accurate, so that a lower bit rate and higher encoding quality can be guaranteed when encoding based on the determined frame type.

[0055] Figure 5 An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0056] like Figure 5 As shown, it is assumed that the video frame sequence stored on the first device needs to be transmitted to the second device. In this case, the first device can determine the I frame according to the prior art (for example, one I frame is determined for each video frame with a certain value interval, or the I frame is determined by detecting the scene switch of adjacent video frames), and then execute the frame type determination method of the embodiment of the present disclosure to determine the P frame and the B frame. The frame type determination method can be executed by a frame type determination device (not shown) of the software, and the determined frame type is also output by the frame type determination device. After the type of each video frame is determined, the encoding order of the video frame can be determined according to a preset reference relationship to ensure that when each video frame is encoded, the video frame referenced by the video frame has been encoded and the video frame referenced by the video frame has not yet started encoding.

[0057] The first device can encode each video frame according to the determined encoding order. When encoding each video frame, it is encoded according to the encoding method corresponding to the type of the video frame. For example, the encoding method of the I frame is intra-frame encoding, the encoding method of the P frame is forward prediction encoding, and the encoding method of the B frame is bidirectional prediction interpolation encoding. When the first device encodes each video frame, it can encode it only once to achieve higher encoding efficiency, or it can perform pre-coding and formal encoding, and adjust the formal encoding process according to the result of pre-coding to achieve better encoding effect. The embodiment of the present disclosure does not limit this. Intra-frame coding, forward prediction coding, and bidirectional prediction interpolation coding can all be implemented based on existing technologies, and adjusting the formal encoding process according to the result of pre-coding can also be implemented based on existing technologies. The specific encoding process will not be repeated here.

[0058] The first device can transmit the encoded code stream and the reference relationship, type and other information of each video frame to the second device. After receiving the code stream, the second device can decode it according to the reference relationship, type and other information of each video frame, thereby restoring each video frame, and then reorder the video frames according to the acquisition order of the video frames themselves (such as recording by the video frame numbers) to restore the video frame sequence.

[0059] Figure 6 A schematic diagram showing the process of the frame type determination method according to an embodiment of the present disclosure.

[0060] like Figure 6 As shown, the present disclosure proposes a frame type determination method, the method is used to determine the type of a video frame in a video frame sequence, the video frame sequence includes a plurality of video frames of unknown types, each video frame includes a plurality of coding blocks, the method includes:

[0061] Step S61, calculating encoding information of each type of unknown video frame according to a preset reference relationship, the encoding information including encoding cost;

[0062] Step S62, counting the number of video frames corresponding to the calculated encoding information, when the encoding information of X video frames of unknown type is calculated, determining a forward prediction encoding frame from the X video frames according to the encoding information of the X video frames, and determining the video frame of unknown type before the forward prediction encoding frame among the X video frames as a bidirectional prediction interpolation encoding frame, where X represents a preset first number and X is an integer greater than 1;

[0063] Step S63, outputting the determined type of the video frame, and re-executing the number of video frames corresponding to the encoding information obtained by statistical calculation and subsequent steps.

[0064] Figure 7 A schematic diagram showing a video frame sequence according to an embodiment of the present disclosure.

[0065] like Figure 7 As shown, it is assumed that video frames 11 to 19 are continuously captured video frames. Each video frame is numbered in the order of the capture time, and the smaller the number, the earlier the capture time of the video frame. Among them, video frame 11 and video frame 19 have been determined as I frames, and video frame 12 to video frame 18 can be regarded as a video frame sequence.

[0066] Each video frame may include multiple coding blocks, and each coding block may be coded using an intra-frame coding mode or an inter-frame coding mode. The coding modes used by different coding blocks in the same video frame may be the same or different.

[0067] In step S61, the coding information of each type of unknown video frame can be calculated according to the preset reference relationship, and the coding information includes the coding cost. The preset reference relationship determines which video frames each video frame refers to for coding when calculating the coding cost, and an example of the preset reference relationship is given later. The coding cost refers to the bit rate of the encoded video frame. There are many types of coding costs that can be calculated, and examples thereof are given later.

[0068] In addition to the coding cost, the coding information may also include more content, such as the proportion of coding blocks using the intra-frame coding mode described later. The embodiments of the present disclosure do not limit the specific content included in the coding information.

[0069] The encoding information of the video frames can be calculated according to the acquisition order of the video frames. Figure 6 In the example of , the coding information of video frames 12 to 18 can be calculated sequentially. The coding information of each video frame only needs to be calculated once, avoiding repeated calculations, and reducing the amount of calculation and delay.

[0070] In step S62, the number of video frames corresponding to the calculated encoding information may be counted. When encoding information of X video frames of unknown type is calculated, a forward prediction encoding frame is determined from the X video frames according to the encoding information of the X video frames, and a video frame of unknown type before the forward prediction encoding frame among the X video frames is determined as a bidirectional prediction interpolation encoding frame, where X represents a preset first number and is an integer greater than 1.

[0071] As long as step S61 is executed to calculate the coding information of X video frames of unknown frame types, step S62 can be executed to determine the video frame types. Therefore, after calculating the coding information of X video frames of unknown frame types, step S61 and step S62 can be performed in parallel, thereby speeding up the frame type determination.

[0072] X can be set according to the size of the group of pictures (Group Of Pictures, GOP) required by the application scenario. The group of pictures can be composed of continuous video frames encoded together, the I frame can be independently used as a group of pictures, and the P frame can form a group of pictures with multiple continuous forward B frames. If the nearest frame to the P frame is a non-B frame, the P frame can also be independently used as a group of pictures. A group of pictures can be encoded together. The first number can be greater than or equal to the maximum value W of the group of pictures size, and less than the maximum number of video frames that the frame type determination device can store simultaneously. The embodiment of the present disclosure does not limit the specific value of the first number.

[0073] by Figure 7 For example, assuming X=4, after calculating the coding information of video frames 12 to 15 (coding information of 4 video frames in total), the forward prediction coding frame (P frame) can be determined from video frames 12 to 15 according to the coding information of video frames 12 to 15. Assuming that video frame 14 is determined to be a forward prediction coding frame, video frame 12 and video frame 13 before video frame 14 are determined to be bidirectional prediction interpolation coding frames (B frames), and the execution of step S62 determines the types of three video frames. Assuming that video frame 12 is determined to be a forward prediction coding frame, since there are no other video frames before video frame 12, the execution of step S62 only determines the type of one video frame.

[0074] Next, step S63 may be executed to output the determined type of the video frame, and step S62 and subsequent steps may be re-executed. When step S62 is re-executed, the coding information of the video frame of unknown type among the X video frames when step S62 was previously executed is counted together with the coding information of other video frames calculated subsequently, and after the sum of the number of the calculated coding information of the video frames of unknown type reaches X again, the forward prediction coding frame is determined from the new X video frames based on the coding information of the new X video frames.

[0075] by Figure 7For example, assuming X=4, it is determined that video frame 14 is a P frame, and video frames 12 and 13 are B frames, and the types BBP of video frames 12 to 14 can be output. When step S62 is re-executed, the coding information of video frame 15 and the coding information of the video frame behind video frame 15 are statistically analyzed together, and after the coding information of video frames 15 to 18 (a total of 4 video frames) is calculated, the forward prediction coding frame can be determined from video frames 15 to 18 according to the coding information of video frames 15 to 18. Assuming that video frame 18 is determined to be a forward prediction coding frame, video frames 15 to 17 before video frame 18 are determined to be bidirectional prediction interpolation coding frames (B frames), and the types of four video frames are determined by executing step S62 this time. Then, step S63 can be re-executed to output the types BBBP of video frames 15 to 18.

[0076] By analogy, every time the calculated encoding information of a video frame of unknown frame type reaches X, the type of the video frame can be determined once, until the types of all the video frames are determined.

[0077] According to the frame type determination method of the embodiment of the present disclosure, according to a preset reference relationship, coding information of each type of unknown video frame is calculated, the coding information includes a coding cost, and the coding information of each frame is calculated only once, so as to avoid repeated calculation of the coding information, thereby reducing the amount of calculation and the delay caused by calculating the coding information; the number of video frames corresponding to the calculated coding information is counted, and when the coding information of X video frames of unknown type is calculated, a forward prediction coding frame is determined from the X video frames according to the coding information of the X video frames, and the video frame of unknown type before the forward prediction coding frame in the X video frames is determined as a bidirectional prediction interpolation coding frame Frame, that is, after obtaining X pieces of coding information, the frame type can be determined, so that the determination of the frame type and the calculation of the coding information can be carried out in parallel, further reducing the delay of the frame type determination; after the type of the video frame is determined, it is output, and the number of video frames corresponding to the coding information obtained by statistical calculation and the subsequent steps are re-executed, so that the output delay of the determined video frame type is smaller; and the video frame type can be determined once every time the calculated coding information of the video frame of unknown frame type reaches X, until the types of all video frames are determined, so the type of each video frame can be determined for a video frame sequence of any length. In summary, the frame type determination method of the disclosed embodiment can reduce the amount of calculation and delay of frame type determination and improve the efficiency of frame type determination.

[0078] The following introduces an example of a preset reference relationship used in an embodiment of the present disclosure, and an exemplary method for calculating encoding information of various types of unknown video frames according to the preset reference relationship.

[0079] In a possible implementation, the preset reference relationship may be a reference relationship used in calculating the coding cost in the prior art, an example of which can be found in Figure 3 and Figure 4 The relevant description will not be repeated here.

[0080] In order to further reduce computational complexity and latency, the present disclosure also proposes a preset low-latency reference relationship. Figure 8 A schematic diagram showing a low-latency reference relationship according to an embodiment of the present disclosure.

[0081] In a possible implementation, the preset reference relationship is that the closest intra-frame coded frame or forward prediction coded frame to the current video frame is used as the first reference frame, and the previous video frame of the current video frame is used as the second reference frame, and the coding information also includes the proportion of coding blocks using the intra-frame coding mode when referring to the first reference frame.

[0082] According to the preset reference relationship, the encoding information of each type of unknown video frame is calculated, including:

[0083] When each type of unknown video frame is used as the current video frame, the first coding cost of the current video frame referring to the first reference frame or the proportion of the coding block using the intra-frame coding mode is calculated, and the second coding cost and the third coding cost of the current video frame referring to the second reference frame are calculated.

[0084] For example, if Figure 8 As shown, the preset reference relationship can be that the closest intra-frame coded frame or forward predictive coded frame to the current video frame is used as the first reference frame, and the previous video frame of the current video frame is used as the second reference frame. Assume that video frames 11 to 19 are continuously captured video frames. Each video frame is numbered in the order of acquisition time, and the video frame with a smaller number is captured earlier. Among them, video frames 11 and 19 have been determined as I frames, and video frames 12 to 18 can be used as a video frame sequence. Among them, when video frame 12 is used as the current video frame, the first reference frame and the second reference frame can be the same and be video frame 11; when video frame 13 is used as the current video frame, the first reference frame can be video frame 11, and the second reference frame can be video frame 12.

[0085] The encoding information of the video frame may also include the proportion of the encoding blocks using the intra-frame encoding mode when the video frame refers to the first reference frame. When each video frame of unknown type is used as the current video frame, the first encoding cost of the current video frame referring to the first reference frame or the proportion of the encoding blocks using the intra-frame encoding mode may be calculated, and the second encoding cost and the third encoding cost of the current video frame referring to the second reference frame may be calculated.

[0086] The calculation method of the first coding cost and the calculation method of the third coding cost may be the same, and different from the calculation method of the second coding cost. If the first reference frame and the second reference frame of the current video frame are the same video frame, only the second coding cost and the third coding cost of the current video frame with reference to the second reference frame may be calculated, and optionally, the proportion of coding blocks using the intra-frame coding mode when the current video frame refers to the second reference frame may also be calculated.

[0087] Those skilled in the art should understand that, assuming that a P frame is determined at the first moment in step S62, when the coding information is calculated in step S61 between the first moment and the second moment in which the P frame is determined next time, the first reference frame may be changed to the P frame determined at the moment T1. In this way, the calculation accuracy of the coding information can be guaranteed.

[0088] By adopting a simpler low-latency reference relationship, the backward frame is not referenced when calculating the coding information, which facilitates hardware pipelining to accelerate the calculation of coding information and is suitable for a frame type determination method that combines software and hardware.

[0089] The first device may be provided with a queue for storing video frames. The queue is used to store video frames of unknown type, and the closest intra-frame coded frames or forward predicted coded frames of each type of unknown video frame. The queue depth may be equal to X+1. Each time the coding information of a video frame of unknown type is calculated, the video frame is stored in the queue. When the queue is full, it can be determined that the coding information of X video frames of unknown type is calculated. Each time an image group is determined, the B frame in the image group and the closest intra-frame coded frame or forward predicted coded frame of the image group are removed from the queue. In this way, the number of video frames corresponding to the calculated coding information can be counted.

[0090] Those skilled in the art should understand that there may be other ways to statistically calculate the number of video frames corresponding to the coding information obtained by calculation, such as setting a counter, and adding one to the value of the counter each time the coding information of an unknown type of video frame is calculated, and subtracting the size of the image group from the value of the counter each time a picture group is determined. As long as accurate statistics of the number of video frames corresponding to the calculated coding information can be achieved, the embodiments of the present disclosure do not limit the specific method of statistically calculating the number of video frames corresponding to the calculated coding information.

[0091] When calculating the coding cost of a video frame in the embodiment of the present disclosure, each coding block of the video frame has two coding modes to choose from, namely, an intra-frame coding mode and an inter-frame coding mode. When calculating different coding costs with reference to different video frames, the coding mode selection method of each coding block can be different.

[0092] An exemplary calculation method of the first coding cost and the ratio of coding blocks using the intra-frame coding mode is introduced below.

[0093] In a possible implementation, calculating the first coding cost of the current video frame referring to the first reference frame or the proportion of the coding blocks using the intra-frame coding mode includes:

[0094] Referring to the first reference frame, calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode;

[0095] For each coding block of the current video frame, a coding mode with a smaller coding cost is determined as the coding mode adopted by the coding block, and a first coding cost is obtained according to the sum of the coding costs of each coding block under the adopted coding mode, or a ratio is obtained according to the ratio of the number of coding blocks adopting the intra-frame coding mode to the total number of coding blocks of the current video frame.

[0096] For example, when calculating the first coding cost of the current video frame with reference to the first reference frame, for each coding block of the current video frame, the coding cost of each coding block in the inter-frame coding mode and the intra-frame coding mode can be calculated with reference to the first reference frame. If the coding cost of a certain coding block in the inter-frame coding mode is smaller, the inter-frame coding mode is determined as the coding mode adopted by the coding block. If the coding cost of a certain coding block in the intra-frame coding mode is smaller, the intra-frame coding mode is determined as the coding mode adopted by the coding block. By analogy, the coding mode adopted by all coding blocks of the video frame can be determined. The first coding cost of the video frame with reference to the first reference frame can be obtained according to the sum of the coding costs of each coding block in the adopted coding mode. The proportion of coding blocks that adopt the intra-frame coding mode when the video frame refers to the first reference frame can be obtained according to the ratio of the number of coding blocks that adopt the intra-frame coding mode to the total number of coding blocks of the current video frame.

[0097] Assume that the video frame includes coding block 1 to coding block 16. The coding cost of coding block 1 to coding block 4 in the inter-frame coding mode is smaller, and the coding cost of coding block 5 to coding block 16 in the intra-frame coding mode is smaller, then the inter-frame coding mode can be determined as the coding mode adopted by coding block 1 to coding block 4, and the intra-frame coding mode can be determined as the coding mode adopted by coding block 5 to coding block 16. The number of coding blocks adopting the intra-frame coding mode is 12, and the total number of coding blocks of the current video frame is 16, so the proportion of coding blocks adopting the intra-frame coding mode when the video frame refers to the first reference frame can be equal to 3 / 4 (12 / 16).

[0098] Those skilled in the art should understand that in practical applications, other methods may be used to calculate the first coding cost, such as referring to the first reference frame, calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode, and obtaining the first coding cost according to the sum of the coding costs of each coding block. As long as the first coding cost can represent the bit rate when the current video frame is encoded with reference to the first reference frame, the specific calculation method of the first coding cost is not limited in the embodiment of the present disclosure.

[0099] Combine the following Figure 7 An exemplary calculation method of the second coding cost and the third coding cost is introduced.

[0100] In a possible implementation, calculating a second encoding cost and a third encoding cost of a current video frame with reference to a second reference frame includes:

[0101] Referring to the second reference frame, calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode;

[0102] Obtaining a second coding cost according to the sum of coding costs of each coding block in the inter-frame coding mode;

[0103] For each coding block of the current video frame, a coding mode with a smaller coding cost is determined as the coding mode adopted by the coding block, and a third coding cost is obtained according to the sum of the coding costs of each coding block in the adopted coding mode.

[0104] For example, when calculating the second coding cost and the third coding cost of the current video frame with reference to the second reference frame, for each coding block of the current video frame, the second reference frame may be referred to to calculate the coding cost of each coding block in the inter-frame coding mode and the intra-frame coding mode. The second coding cost of the video frame with reference to the second reference frame may be obtained according to the sum of the coding costs of each coding block in the inter-frame coding mode.

[0105] If the coding cost of a certain coding block in the inter-frame coding mode is smaller, the inter-frame coding mode is determined as the coding mode adopted by the coding block. If the coding cost of a certain coding block in the intra-frame coding mode is smaller, the intra-frame coding mode is determined as the coding mode adopted by the coding block. By analogy, the coding modes adopted by all coding blocks of the video frame can be determined. The third coding cost of the video frame referring to the second reference frame can be obtained according to the sum of the coding costs of each coding block in the adopted coding mode.

[0106] Those skilled in the art should understand that, in practical applications, other methods may also be used to calculate the second coding cost and the third coding cost, as long as the second coding cost and the third coding cost can represent the bit rate when the current video frame is encoded with reference to the second reference frame, and the difference between the second coding cost and the third coding cost can represent the impact of the intra-frame coding mode on the bit rate of the current video frame, the embodiments of the present disclosure do not limit the specific calculation method of the second coding cost and the third coding cost.

[0107] Those skilled in the art should understand that the preset reference relationship may be set in other ways, as long as the encoding information of X video frames of unknown type can be calculated according to the preset reference relationship. The embodiments of the present disclosure do not limit the specific setting method of the preset reference relationship and the specific calculation method of the encoding information of each video frame.

[0108] The following describes an exemplary method for determining a forward prediction coding frame from X video frames according to an embodiment of the present disclosure.

[0109] In a possible implementation, determining a forward prediction coding frame from the X video frames according to coding information of the X video frames includes:

[0110] Determine Y candidate video frames from the X video frames according to the first encoding cost or ratio, 1≤Y≤X, and Y is an integer;

[0111] A forward prediction coding frame is determined from Y candidate video frames according to the second coding cost and the third coding cost.

[0112] For example, the frame type determination of the embodiment of the present disclosure can be implemented in two stages. In the first stage, Y candidate video frames from X video frames are screened out according to the first coding cost or ratio, and the possibility that the candidate video frame type is a P frame is relatively high. In the second stage, the forward prediction coding frame is determined from the Y candidate video frames according to the second coding cost and the third coding cost. Compared with the prior art solution of determining the frame type by a single detection, the embodiment of the present disclosure determines the type of the video frame in two stages with higher accuracy.

[0113] The following describes an exemplary method for determining Y candidate video frames from X video frames according to an embodiment of the present disclosure.

[0114] In a possible implementation, determining Y candidate video frames from X video frames according to the first encoding cost or ratio includes:

[0115] When any one of the X video frames is used as the current video frame, the current video frame is determined as a candidate video frame under the condition that a first encoding cost of the current video frame is greater than a first threshold or a corresponding ratio of the current video frame is greater than a second threshold.

[0116] For example, a first threshold and a second threshold may be preset, so that a first coding cost greater than the first threshold may indicate that the possibility of the video frame being a P frame is greater than the possibility of the video frame being a B frame, and a ratio greater than the second threshold may also indicate that the possibility of the video frame being a P frame is greater than the possibility of the video frame being a B frame. In this case, when any one of the X video frames is used as the current video frame, the first coding cost of the current video frame may be compared with the first threshold, or the ratio corresponding to the current video frame and the second threshold may be compared, and the current video frame may be determined as a candidate video frame under the condition that the first coding cost of the current video frame is greater than the first threshold or the ratio corresponding to the current video frame is greater than the second threshold.

[0117] The first threshold and the second threshold can be set according to the application scenario requirements, wherein the setting method of the first threshold is associated with the calculation method of the first coding cost, and can be a statistical empirical value inversely proportional to the number of video frames between the current video frame and the first reference frame. The second threshold can be between 0.2 and 1. The embodiments of the present disclosure do not limit the specific values ​​of the first threshold and the second threshold.

[0118] If the first encoding cost of each of the X video frames is less than or equal to the first threshold, or the corresponding ratio of each of the X video frames is less than or equal to the second threshold, the 1st to W-1th video frames of the X video frames may be determined as B frames, and the Wth video frame may be determined as a P frame according to the maximum value W of the image group size. For example, when X=W=8, the first 7 video frames may be determined as B frames, and the 8th video frame may be determined as a P frame.

[0119] The following describes an exemplary method for determining a forward prediction coding frame from Y candidate video frames according to an embodiment of the present disclosure.

[0120] In a possible implementation, determining a forward prediction coding frame from Y candidate video frames according to the second coding cost and the third coding cost includes:

[0121] When any one of the Y candidate video frames is used as the current video frame, the relative motion complexity and reference redundancy of the current video frame are calculated according to the second coding cost and the third coding cost of the current video frame, wherein the relative motion complexity indicates the relative motion degree of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy indicates the influence degree of the coding block using the intra-frame coding mode on the coding cost of the current video frame;

[0122] When the relative motion complexity of the current video frame is greater than a third threshold and the reference redundancy is greater than a fourth threshold, the current video frame is determined as a forward prediction coding frame.

[0123] For example, when any one of the Y candidate video frames is used as the current video frame, the relative motion complexity and reference redundancy of the current video frame can be calculated based on the second coding cost and the third coding cost of the current video frame, wherein the relative motion complexity indicates the relative motion degree of the coding block of the current video frame relative to the adjacent video frame. The reference redundancy indicates the influence of the coding block using the intra-frame coding mode on the coding cost of the current video frame. An exemplary method for calculating the relative motion complexity and reference redundancy based on the second coding cost and the third coding cost is given below.

[0124] A third threshold and a fourth threshold may be preset, and when the relative motion complexity of the current video frame is greater than the third threshold and the reference redundancy is greater than the fourth threshold, the current video frame is determined as a forward prediction coding frame. A relative motion complexity greater than the third threshold may indicate that the motion between the current video frame and adjacent video frames is violent or the brightness changes violently, so the current video frame is of high importance, and the coding effect of the current video frame as a P frame is obviously better than that of a B frame. A reference redundancy greater than the fourth threshold may indicate that the current video frame has more new content, and because the coding cost of the new content using the intra-frame coding mode is much less than the coding cost of the inter-frame coding mode, the coding effect of the current video frame as a P frame is obviously better than that of a B frame.

[0125] If there is only one video frame whose relative motion complexity is greater than the third threshold and whose reference redundancy is greater than the fourth threshold among the Y candidate video frames, then there is only one P frame determined, and the other video frames before the P frame among the X video frames are all video frames of unknown types. Therefore, the video frames before the P frame among the X video frames can be determined as B frames.

[0126] If there are two or more video frames whose relative motion complexity is greater than the third threshold and whose reference redundancy is greater than the fourth threshold among the Y candidate video frames, then there are also two or more P frames determined, and the video frames between each P frame and the previous non-B frame are all video frames of unknown type. All video frames of unknown type before each P frame can be determined as B frames.

[0127] If there is no video frame whose relative motion complexity is greater than the third threshold and whose reference redundancy is greater than the fourth threshold among the Y candidate video frames, the video frame satisfying any one of the conditions that the relative motion complexity is the largest and greater than the third threshold and the reference redundancy is the largest and greater than the fourth threshold can be determined as a P frame. If there is neither a video frame whose relative motion complexity is greater than the third threshold nor a video frame whose reference redundancy is greater than the fourth threshold, the 1st to W-1th video frames among the X video frames can be determined as B frames according to the maximum value W of the image group size, and the Wth video frame can be determined as a P frame.

[0128] The third threshold and the fourth threshold can be set according to the application scenario requirements, wherein the setting method of the third threshold is related to the second coding cost, and an exemplary setting method of the third threshold is given later. The fourth threshold can be an empirical value obtained through experimental data. The embodiments of the present disclosure do not limit the specific values ​​of the third threshold and the fourth threshold.

[0129] Since the forward prediction coding frame is jointly determined based on the relative motion complexity and the reference redundancy, the accuracy of determining the forward prediction coding frame can be greatly improved, thereby making the bit rate allocation more reasonable and obtaining a coding result with higher image quality.

[0130] The following describes an exemplary method for calculating relative motion complexity and reference redundancy in an embodiment of the present disclosure.

[0131] In a possible implementation, the relative motion complexity of the current video frame is equal to the difference between the second encoding costs of the current video frame and the adjacent video frame;

[0132] The reference redundancy of the current video frame is equal to the difference between the second encoding cost and the third encoding cost of the current video frame.

[0133] For example, the difference between the second encoding cost of the current video frame and the adjacent video frame can be used as the relative motion complexity of the current video frame. The adjacent video frame can select the previous video frame or the next video frame of the current video frame, and the present disclosure does not limit this. In this case, the greater the relative motion complexity of the current video frame, the more difficult it is to encode the current video frame with reference to other video frames, and the greater the probability of being a P frame. At this time, the third threshold can be set to 1 / k of the larger second encoding cost of the current video frame and the adjacent video frame, and k can be set by the user according to the application scenario requirements.

[0134] The difference between the second coding cost and the third coding cost of the current video frame can be used as the reference redundancy of the current video frame. The difference between the second coding cost and the third coding cost is caused by the coding block using the intra-frame coding mode. Therefore, the difference between the second coding cost and the third coding cost, that is, the reference redundancy, can characterize the impact of the intra-frame coding mode on the coding cost of the current video frame. In this case, the greater the reference redundancy, the more difficult it is for the current video frame to be encoded with reference to other video frames, and the greater the probability of being a P frame.

[0135] Those skilled in the art should understand that the relative motion complexity and reference redundancy may also be calculated in other ways, as long as the relative motion complexity can represent the relative motion of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy can represent the influence of the coding block using the intra-frame coding mode on the coding cost of the current video frame, the embodiments of the present disclosure do not limit the specific calculation method of the relative motion complexity and reference redundancy.

[0136] The present disclosure also provides a frame type determination device, Fig. 9 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0137] like Fig. 9 As shown, in a possible implementation manner, the device is used to determine the type of a video frame in a video frame sequence, the video frame sequence includes a plurality of video frames of unknown types, each video frame includes a plurality of coding blocks, and the device includes:

[0138] A calculation module 91, configured to calculate encoding information of each type of unknown video frame according to a preset reference relationship, wherein the encoding information includes encoding cost;

[0139] a determination module 92, configured to count the number of video frames corresponding to the calculated encoding information, and when the encoding information of X video frames of unknown type is calculated, determine a forward prediction encoding frame from the X video frames according to the encoding information of the X video frames, and determine a video frame of unknown type before the forward prediction encoding frame among the X video frames as a bidirectional prediction interpolation encoding frame, where X represents a preset first number and X is an integer greater than 1;

[0140] The output module 93 is used to output the determined type of the video frame, and re-execute the number of video frames corresponding to the encoding information obtained by statistical calculation and subsequent steps.

[0141] In a possible implementation, the preset reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous video frame of the current video frame as the second reference frame, and the encoding information also includes the proportion of coding blocks using the intra-frame coding mode when referring to the first reference frame. The encoding information of each type of unknown video frame is calculated according to the preset reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of the current video frame referring to the first reference frame or the proportion of coding blocks using the intra-frame coding mode is calculated, and the second coding cost and third coding cost of the current video frame referring to the second reference frame are calculated.

[0142] In a possible implementation, the calculation of the first coding cost of the current video frame with reference to the first reference frame or the proportion of coding blocks using the intra-frame coding mode includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the first reference frame; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the first coding cost according to the sum of the coding costs of each coding block in the adopted coding mode, or obtaining the proportion according to the ratio of the number of coding blocks using the intra-frame coding mode to the total number of coding blocks of the current video frame.

[0143] In a possible implementation, the calculation of the second coding cost and the third coding cost of the current video frame with reference to the second reference frame includes: calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode with reference to the second reference frame; obtaining the second coding cost according to the sum of the coding costs of each coding block in the inter-frame coding mode; for each coding block of the current video frame, determining the coding mode with a smaller coding cost as the coding mode adopted by the coding block, and obtaining the third coding cost according to the sum of the coding costs of each coding block in the adopted coding mode.

[0144] In a possible implementation, determining the forward predicted coded frame from the X video frames according to the coding information of the X video frames includes: determining Y candidate video frames from the X video frames according to the first coding cost or the ratio, 1≤Y≤X, and Y is an integer; and determining the forward predicted coded frame from the Y candidate video frames according to the second coding cost and the third coding cost.

[0145] In a possible implementation, the determining Y candidate video frames from the X video frames according to the first encoding cost or the ratio includes: when any one of the X video frames is used as the current video frame, under the condition that the first encoding cost of the current video frame is greater than a first threshold or the ratio corresponding to the current video frame is greater than a second threshold, determining the current video frame as a candidate video frame.

[0146] In a possible implementation, the forward predicted coding frame is determined from the Y candidate video frames based on the second coding cost and the third coding cost, including: when any one of the Y candidate video frames is used as the current video frame, the relative motion complexity and reference redundancy of the current video frame are calculated based on the second coding cost and the third coding cost of the current video frame, the relative motion complexity represents the relative motion degree of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy represents the influence of the coding block using the intra-frame coding mode on the coding cost of the current video frame; when the relative motion complexity of the current video frame is greater than a third threshold and the reference redundancy is greater than a fourth threshold, the current video frame is determined as a forward predicted coding frame.

[0147] In a possible implementation, the relative motion complexity of the current video frame is equal to the difference between the second coding cost of the current video frame and the adjacent video frame; the reference redundancy of the current video frame is equal to the difference between the second coding cost and the third coding cost of the current video frame.

[0148] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0149] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0150] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0151] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0152] Fig.10 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.10, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0153] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0154] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0155] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0156] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0157] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0158] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0159] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0160] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0161] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0162] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0163] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining a frame type, characterized in that: The method is used to determine the type of a video frame in a video frame sequence, wherein the video frame sequence includes a plurality of video frames of unknown types, each video frame includes a plurality of coding blocks, and the method includes: Calculating encoding information of each type of unknown video frame according to a preset reference relationship, wherein the encoding information includes encoding cost; Counting the number of video frames corresponding to the calculated encoding information, when the encoding information of X video frames of unknown type is calculated, determining a forward prediction encoding frame from the X video frames according to the encoding information of the X video frames, and determining a video frame of unknown type before the forward prediction encoding frame among the X video frames as a bidirectional prediction interpolation encoding frame, where X represents a preset first number and X is an integer greater than 1; Output the determined video frame type, and re-execute the number of video frames corresponding to the encoding information obtained by statistical calculation and subsequent steps.

2. The method according to claim 1, characterized in that The preset reference relationship is that the closest intra-frame coded frame or forward prediction coded frame to the current video frame is used as the first reference frame, and the previous video frame of the current video frame is used as the second reference frame, and the coding information also includes the proportion of coding blocks using the intra-frame coding mode when referring to the first reference frame. The step of calculating the encoding information of each type of unknown video frame according to the preset reference relationship includes: When each type of unknown video frame is used as the current video frame, the first coding cost of the current video frame referring to the first reference frame or the proportion of the coding block using the intra-frame coding mode is calculated, and the second coding cost and the third coding cost of the current video frame referring to the second reference frame are calculated.

3. The method according to claim 2, characterized in that The calculating the first coding cost of the current video frame referring to the first reference frame or the proportion of the coding blocks using the intra-frame coding mode includes: Referring to the first reference frame, calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode; For each coding block of the current video frame, a coding mode with a smaller coding cost is determined as the coding mode adopted by the coding block, and the first coding cost is obtained according to the sum of the coding costs of each coding block under the adopted coding mode, or the ratio is obtained according to the ratio of the number of coding blocks adopting the intra-frame coding mode to the total number of coding blocks of the current video frame.

4. The method according to claim 2 or 3, characterized in that: The calculating the second encoding cost and the third encoding cost of the current video frame with reference to the second reference frame comprises: Referring to the second reference frame, calculating the coding cost of each coding block of the current video frame in the inter-frame coding mode and the intra-frame coding mode; Obtaining the second coding cost according to the sum of coding costs of each coding block in the inter-frame coding mode; For each coding block of the current video frame, a coding mode with a smaller coding cost is determined as the coding mode adopted by the coding block, and the third coding cost is obtained according to the sum of the coding costs of each coding block in the adopted coding mode.

5. The method according to claim 2, characterized in that: The step of determining a forward prediction coding frame from the X video frames according to coding information of the X video frames comprises: Determine Y candidate video frames from the X video frames according to the first encoding cost or the ratio, 1≤Y≤X, and Y is an integer; The forward prediction coding frame is determined from the Y candidate video frames according to the second coding cost and the third coding cost.

6. The method according to claim 5, characterized in that The determining Y candidate video frames from the X video frames according to the first encoding cost or the ratio includes: When any one of the X video frames is used as the current video frame, the current video frame is determined as a candidate video frame under the condition that a first encoding cost of the current video frame is greater than a first threshold or a corresponding ratio of the current video frame is greater than a second threshold.

7. The method according to claim 5, characterized in that The step of determining the forward prediction coding frame from the Y candidate video frames according to the second coding cost and the third coding cost includes: When any one of the Y candidate video frames is used as the current video frame, a relative motion complexity and a reference redundancy of the current video frame are calculated according to the second coding cost and the third coding cost of the current video frame, wherein the relative motion complexity indicates the relative motion degree of the coding block of the current video frame relative to the adjacent video frame, and the reference redundancy indicates the influence degree of the coding block using the intra-frame coding mode on the coding cost of the current video frame; When the relative motion complexity of the current video frame is greater than a third threshold and the reference redundancy is greater than a fourth threshold, the current video frame is determined as a forward prediction coding frame.

8. The method according to claim 7, characterized in that The relative motion complexity of the current video frame is equal to the difference between the second encoding costs of the current video frame and the adjacent video frame; The reference redundancy of the current video frame is equal to the difference between the second encoding cost and the third encoding cost of the current video frame.

9. A frame type determination device, characterized in that: The apparatus is used to determine the type of a video frame in a video frame sequence, wherein the video frame sequence includes a plurality of video frames of unknown types, each video frame includes a plurality of coding blocks, and the apparatus includes: A calculation module, used to calculate the coding information of each type of unknown video frame according to a preset reference relationship, wherein the coding information includes a coding cost; a determination module, configured to count the number of video frames corresponding to the calculated encoding information, and when the encoding information of X video frames of unknown type is calculated, determine a forward prediction encoding frame from the X video frames according to the encoding information of the X video frames, and determine a video frame of unknown type before the forward prediction encoding frame among the X video frames as a bidirectional prediction interpolation encoding frame, where X represents a preset first number and X is an integer greater than 1; The output module is used to output the determined type of the video frame and re-execute the number of video frames corresponding to the encoding information obtained by statistical calculation and subsequent steps.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images