Frame type judgment method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of video coding and decoding, and provides a frame type judgment method and device, electronic equipment and a storage medium. The method comprises the following steps: calculating the coding cost of each type of unknown video frame according to a first reference relationship; taking X types of unknown video frames of which the coding cost is calculated, judging whether the video frames are I frames according to the coding cost of each type of unknown video frame in the taken video frames, and taking the I frames as a group when one I frame is judged; and judging whether the video frame is a P frame or not according to the coding cost of each type of unknown video frame in the extracted video frames, judging the video frame which has the acquisition time earlier than that of the P frame and has the unknown type in the extracted video frames as a B frame every time a P frame is judged, and taking the P frame and the B frame which are judged this time as a group. According to the method, the judgment efficiency of the type of the video frame can be improved, so that the coding efficiency of the video frame is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding, and in particular to a frame type determination method, device, electronic device and storage medium. Background Art

[0002] The data volume of video frame sequence is very large. In order to reduce transmission power consumption and storage cost, the video frame sequence is usually encoded before transmission or storage. In order to obtain better encoding effect, it is necessary to estimate the encoding cost of the video frame first, and then determine the grouping of video frames, the type of video frames in the group, and the block-level encoding information and quantization parameters of the video frame based on the encoding cost. Then, based on the determined encoding parameters, the video frame is formally encoded and output as a bitstream.

[0003] Most of the steps of the encoder model proposed in the prior art must be executed serially, and each step needs to process a large number of video frames, resulting in a large delay in each step. In addition, it is difficult to use hardware acceleration for the steps executed serially, which further reduces the encoding efficiency.

[0004] In summary, the existing video frame coding scheme has the disadvantage of low coding efficiency. How to improve the coding efficiency of video frames has become a technical problem to be solved urgently in this field. Summary of the invention

[0005] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The method can improve the determination efficiency of the type of video frames, thereby improving the coding efficiency of video frames.

[0006] According to one aspect of the present disclosure, a frame type determination method is provided, the method being used to group a video frame sequence and determine the type of video frames in each group, the video frame sequence comprising a plurality of video frames of unknown type, the method comprising: calculating the coding cost of each unknown type of video frame according to a preset first reference relationship; taking X unknown type of video frames for which the coding cost is calculated, and determining whether the video frame is an intra-frame coded frame according to the coding cost of each unknown type of video frame in the taken video frames, where X represents a first preset number and X is an integer greater than 1; each time an intra-frame coded frame is determined, the intra-frame coded frame determined this time is regarded as a group; based on the coding cost of each unknown type of video frame in the taken video frames, determining whether the video frame is a forward prediction coded frame; each time a forward prediction coded frame is determined, determining a video frame in the taken video frames whose acquisition time is earlier than that of the forward prediction coded frame and whose type is unknown as a bidirectional prediction interpolation coded frame, and treating the forward prediction coded frame and the bidirectional prediction interpolation coded frame determined this time as a group.

[0007] In a possible implementation, after the forward prediction coding frame and the bidirectional prediction interpolation coding frame judged this time are grouped, the method further includes: re-taking X unknown type video frames for which the coding costs are calculated, and when the number of unknown type video frames is less than X, after calculating the coding costs of all unknown type video frames, taking out all unknown type video frames; and executing the steps of judging whether the video frame is an intra-frame coded frame according to the coding cost of each unknown type video frame in the taken out video frames and subsequent steps until the type judgment of all video frames in the video frame sequence is completed.

[0008] In a possible implementation, the re-retrieved video frames include video frames whose types are still unknown at the current moment among the previously retrieved video frames.

[0009] In a possible implementation, the first reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous coded frame of the current video frame as the second reference frame. The coding costs of various types of unknown video frames are calculated based on the preset first reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of encoding the current video frame with reference to the first reference frame, the second coding cost of encoding with reference to the second reference frame, and the third coding cost of independent coding are calculated.

[0010] In a possible implementation, the method of judging whether a video frame is an intra-coded frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: judging that the current video frame is an intra-coded frame when a third coding cost of the current video frame is less than a product of a first value and a preset proportional coefficient, and the first value is equal to the smaller one of a first coding cost and a second coding cost of the current video frame.

[0011] In a possible implementation, the method of judging whether a video frame is a forward prediction coding frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: when the smaller of the first coding cost and the second coding cost of the current video frame is less than a preset first threshold, judging that the current video frame is a forward prediction coding frame.

[0012] In a possible implementation, the method further includes: determining a second reference relationship of the video frames in each group based on the number of video frames in the group; and reordering the video frames in the group based on the second reference relationship, and using the reordered order as the encoding order of the video frames in the group.

[0013] In a possible implementation, each video frame includes multiple coding blocks, and the method further includes: taking Y video frames that have been grouped and whose block-level coding information has not been calculated, and determining the block-level coding information of the video frame according to the type of each video frame, wherein the block-level coding information includes the coding method of each coding block and the coding cost of each coding block, and Y represents a second preset number, and Y is an integer greater than 1; reversely parsing the block-level coding information of the video frames in the most recently determined Z groups to determine the block-level quantization parameters of the video frames in the earliest determined group among the Z groups, wherein the block-level quantization parameters include the quantization parameters of each coding block, and Z represents a third preset number, and Z is an integer greater than 1; using the type of the video frames in the group, the block-level coding information and the block-level quantization parameters, to encode the video frames in the group to obtain the coding results of the video frames in the group; and re-executing the steps of taking the Y video frames that have been grouped and whose block-level coding information has not been calculated and subsequent steps.

[0014] In one possible implementation, the reverse parsing of the block-level coding information of the video frames in the most recently determined Z groups includes: parsing the block-level coding information of the video frames in the Z groups in an order opposite to the coding order of the video frames in the Z groups, and determining the importance of each coding block of each video frame in the Z groups; and determining the block-level quantization parameters of the video frames in the earliest determined group among the Z groups based on the importance of each coding block of each video frame in the Z groups.

[0015] In a possible implementation, the frame type determination method is executed by a frame type determination device, and X is less than the maximum number of video frames that the frame type determination device can store simultaneously.

[0016] According to another aspect of the present disclosure, a frame type determination device is provided, the device is used to group a video frame sequence and determine the type of video frames in each group, the video frame sequence includes a plurality of video frames of unknown type, the device includes: a calculation module, used to calculate the coding cost of each unknown type of video frame according to a preset first reference relationship; a first determination module, used to take X unknown type of video frames for which the coding cost is calculated, and determine whether the video frame is an intra-frame coded frame according to the coding cost of each unknown type of video frame in the taken video frames, where X represents a first preset number and X is an integer greater than 1; a first grouping module, used to treat the intra-frame coded frame determined this time as a group every time an intra-frame coded frame is determined; a second determination module, used to determine whether the video frame is a forward prediction coded frame according to the coding cost of each unknown type of video frame in the taken video frames; and a second grouping module, used to treat the video frame whose acquisition time is earlier than the forward prediction coded frame and whose type is unknown in the taken video frames as a bidirectional prediction interpolation coded frame every time a forward prediction coded frame is determined, and treat the forward prediction coded frame and the bidirectional prediction interpolation coded frame determined this time as a group.

[0017] In a possible implementation, the device further includes: a first calling module, configured to call the first judging module to re-take X unknown type video frames for which the coding costs are calculated after the second grouping module groups the forward prediction coding frames and the bidirectional prediction interpolation coding frames judged this time as a group; when the number of unknown type video frames is less than X, the first judging module takes out all unknown type video frames after the coding costs of all unknown type video frames are calculated; and calling the first judging module, the first grouping module, the second judging module, and the second grouping module to execute the steps of judging whether the video frame is an intra-frame coded frame based on the coding cost of each unknown type video frame in the taken out video frames and the subsequent steps until the type judgment of all video frames in the video frame sequence is completed.

[0018] In a possible implementation, the re-retrieved video frames include video frames whose types are still unknown at the current moment among the previously retrieved video frames.

[0019] In a possible implementation, the first reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous coded frame of the current video frame as the second reference frame. The coding costs of various types of unknown video frames are calculated based on the preset first reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of encoding the current video frame with reference to the first reference frame, the second coding cost of encoding with reference to the second reference frame, and the third coding cost of independent coding are calculated.

[0020] In a possible implementation, the method of judging whether a video frame is an intra-coded frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: judging that the current video frame is an intra-coded frame when a third coding cost of the current video frame is less than a product of a first value and a preset proportional coefficient, and the first value is equal to the smaller one of a first coding cost and a second coding cost of the current video frame.

[0021] In a possible implementation, the method of judging whether a video frame is a forward prediction coding frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: when the smaller of the first coding cost and the second coding cost of the current video frame is less than a preset first threshold, judging that the current video frame is a forward prediction coding frame.

[0022] In one possible implementation, the device also includes: a first determination module, used to determine a second reference relationship of the video frames in each group based on the number of video frames in the group; and a sorting module, used to reorder the video frames in the group based on the second reference relationship, and the reordered order is used as the encoding order of the video frames in the group.

[0023] In a possible implementation, each video frame includes a plurality of coding blocks, and the device further includes: a second determination module, configured to take Y video frames that have been grouped and for which block-level coding information has not been calculated, and determine the block-level coding information of the video frame according to the type of each video frame, wherein the block-level coding information includes the coding mode of each coding block and the coding cost of each coding block, and Y represents a second preset number, which is an integer greater than 1; a third determination module, configured to reversely parse the block-level coding information of the video frames in the most recently determined Z groups, and determine the block-level quantization parameters of the video frames in the earliest determined group among the Z groups, wherein the block-level quantization parameters include the quantization parameters of each coding block, and Z represents a third preset number, which is an integer greater than 1; an encoding module, configured to encode the video frames in the group using the type of the video frames in the group, the block-level coding information and the block-level quantization parameters, and obtain the encoding result of the video frames in the group; and a second calling module, configured to call the second determination module to re-execute the steps of taking the Y video frames that have been grouped and for which the block-level coding information has not been calculated and the subsequent steps.

[0024] In a possible implementation, the reverse parsing of the block-level coding information of the video frames in the most recently determined Z groups includes: parsing the block-level coding information of the video frames in the Z groups in an order opposite to the coding order of the video frames in the Z groups to determine the importance of each coding block of each video frame in the Z groups; and determining the block-level quantization parameter of the video frame in the earliest determined group of the Z groups according to the importance of each coding block of each video frame in the Z groups. In a possible implementation, the frame type determination method is performed by a frame type determination device, and X is less than the maximum number of video frames stored simultaneously by the frame type determination device.

[0025] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0026] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0027] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0028] According to the frame type determination method of the embodiment of the present disclosure, for a video frame sequence including a plurality of video frames of unknown types, the coding cost of each unknown type of video frame is calculated according to a preset first reference relationship, thereby completing the calculation of the coding cost; X unknown type of video frames for which the coding cost is calculated are taken, and according to the coding cost of each unknown type of video frame in the taken video frames, whether the video frame is an intra-frame coded frame is determined, that is, after the coding cost of the X unknown type of video frames is obtained, the determination of the I frame can be started, so that the determination of the frame type and the calculation of the coding cost can be carried out in parallel, thereby reducing the delay of the frame type determination; after the X video frames have been taken out and the frame type determination of the X video frames has been started, when the coding cost of the new X video frames is calculated , the frame type judgment of the new X video frames can also be carried out in parallel with the frame type judgment of the previous X video frames, further reducing the delay of the frame type judgment; each time an intra-frame coding frame is judged, the intra-frame coding frame judged this time is taken as a group, and the group including the intra-frame coding frame can be determined; according to the coding cost of each unknown type of video frame in the extracted video frame, it is judged whether the video frame is a forward prediction coding frame, and each time a forward prediction coding frame is judged, the video frame in the extracted video frame whose acquisition time is earlier than the forward prediction coding frame and whose type is unknown is judged as a bidirectional prediction interpolation coding frame, and the forward prediction coding frame judged this time and the bidirectional prediction interpolation coding frame are taken as a group, which can realize the judgment of P frame and B frame, and determine the group including P frame. In this case, the judgment of the type of each video frame and the grouping of the video frame sequence can be completed for a video frame sequence of any length. Since the parallel degree of the steps is improved and the delay of the frame type judgment is reduced, the frame type judgment method of the embodiment of the present disclosure can improve the judgment efficiency of the type of the video frame, thereby improving the coding efficiency of the video frame.

[0029] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0031] Figure 1 A schematic diagram showing the workflow of a prediction module of the prior art.

[0032] Figure 2 A schematic diagram showing a fixed reference relationship 2 is shown.

[0033] Figure 3 A schematic diagram showing a fixed reference relationship 3 is shown.

[0034] Figure 4An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0035] Figure 5 A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0036] Figure 6 A schematic diagram showing a video frame sequence according to an embodiment of the present disclosure.

[0037] Figure 7 A schematic diagram showing a first reference relationship according to an embodiment of the present disclosure.

[0038] Figure 8 A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0039] Fig. 9 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0040] Fig.10 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0042] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0043] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0044] The data volume of the video frame sequence is very large. In order to reduce the transmission power consumption and storage cost, the video frame sequence is usually encoded before transmission or storage. In order to obtain a better encoding effect, it is necessary to first estimate the encoding cost of the video frame, and then determine the grouping of the video frame and the type of the video frame in the group based on the encoding cost, and further determine the block-level encoding information and quantization parameters of the video frame according to the grouping and type. The encoding parameters used in the formal encoding, and then formally encode the video frame based on the determined encoding parameters to output the code stream. The video frame encoder model of the prior art completes the above-mentioned steps of encoding cost calculation, video frame type judgment and grouping, and encoding parameter determination through the software prediction module (Lookahead). Among them, when determining the video frame grouping, the prediction module will first determine the type of each video frame input to the prediction module according to the encoding cost. The video frame may include three types, namely, intra-coded frame (Intra Coded Picture), forward predicted coded frame (Predicted Frame) and bi-directional predictive interpolated coded frame (Bi-directional Predictive Frame). Among them, the intra-coded frame is also called I frame, the forward predicted coded frame is also called P frame, and the bi-directional predictive interpolated coded frame is also called B frame. According to the type of the video frames, the prediction module may group the video frame sequence, thereby obtaining a group and type information of the video frames in the group.

[0045] Figure 1 A schematic diagram showing the workflow of a prediction module of the prior art.

[0046] The prediction module may include a queue for storing video frames, and the depth D of the queue may be greater than a preset maximum number M of video frames in a group.

[0047] like Figure 1 As shown, video frames of unknown types are successively input into the prediction module. When the queue is full, that is, the queue stores D video frames, the prediction module starts to calculate the encoding cost of the D video frames in the queue as I frames. Among them, the prediction module first pre-groups the video frames in the queue so that the number of video frames in each group is equal to M. Then, for each group of video frames, the video frames are pre-encoded using a fixed reference relationship 1, and the encoding cost of each video frame included therein is calculated. After the encoding cost corresponding to the group of video frames is calculated, it is determined whether each video frame included in the group of video frames is an I frame based on the calculated encoding cost. The reference relationship used when pre-encoding the video frames determines which video frames are referenced for encoding each video frame.

[0048] After all D video frames are judged as I frames, the fixed reference relationship 2 or the fixed reference relationship 3 is used to pre-encode the video frames, and the encoding cost of each video frame whose type has not been determined among the D video frames is calculated as a P frame or a B frame. After all the encoding costs of each video frame whose type has not been determined are calculated, it is judged whether each video frame whose type has not been determined is a P frame according to the calculated encoding costs. Video frames that are neither I frames nor P frames can be determined as B frames.

[0049] Fixed reference relationship 2 or fixed reference relationship 3 allows the backward video frame to be used as a reference frame for the current video frame. Fixed reference relationship 2 is used when B frames are not allowed to be used as reference frames, and fixed reference relationship 3 is used when B frames are allowed to be used as reference frames. Figure 2 A schematic diagram showing a fixed reference relationship 2 is shown. Figure 3 A schematic diagram showing a fixed reference relationship 3 is shown.

[0050] like Figure 2 As shown, if B frames are not allowed to be used as reference frames, when calculating the coding cost of the current video frame as a B frame, the nearest non-B frame in the previous direction and the nearest non-B frame in the next direction are used as reference frames; when calculating the coding cost of the current video frame as a P frame, the nearest non-B frame in the previous direction is used as a reference frame.

[0051] like Figure 3 As shown, a B frame between two adjacent non-B frames is allowed to be used as a reference frame (Bref frame) for other B frames. When calculating the coding cost of the current video frame as a B frame, the nearest P frame or I frame or Bref frame in the forward direction and the nearest P frame or I frame or Bref frame in the backward direction are used as reference frames. When calculating the coding cost of the current video frame as a P frame, the nearest non-B frame in the forward direction is used as a reference frame.

[0052] Next, the prediction module can determine the encoding order of the D video frames according to the determined D video frame types and the fixed reference relationship 2, and calculate the block-level encoding information of the D video frames according to the encoding order, including the encoding method of each encoding block, the encoding cost of each encoding block, etc. Then, the block-level quantization parameters of the video frames are calculated according to the block-level encoding information of the video frames, including the quantization parameters of each encoding block. After the block-level encoding information and block-level quantization parameters of the D video frames are determined, the video frames are formally grouped according to the type of the video frames, wherein the I frame is treated as a group alone, the P frame and the forward continuous multiple B frames are treated as a group, and if the video frame closest to the P frame is another P frame or an I frame, the P frame is also treated as a group alone.

[0053] After each group is obtained, the video frames in the group are reordered, and the video frames included in the group as well as information such as type, block-level coding information, block-level quantization parameters, etc. are output to the formal encoding module (not shown) in the reordered order. The formal encoding module can formally encode the video frames according to the received information.

[0054] After each group including the video frames and information such as type, block-level coding information, and block-level quantization parameters is output, the video frames included in the group are deleted from the queue, and the types of the remaining video frames in the queue are reset to unknown types. When the queue is full again, the steps of calculating the encoding cost of the D video frames in the queue as the I frame and subsequent steps are re-executed.

[0055] This scheme has the following disadvantages: First, before outputting the video frames and types, block-level coding information, block-level quantization parameters and other information included in a group, it is necessary to calculate the coding cost of the entire queue and perform I frame detection, P frame detection, block-level coding information calculation, and block-level quantization parameter calculation. These are all completed in series, so the amount of calculation and delay are very large; second, each time the video frames and types, block-level coding information, block-level quantization parameters and other information included in a group are output, the type of the video frame in the queue is reset once. After the reset, these reset video frames need to re-complete the steps of calculating the coding cost of the video frame, I frame detection, P frame detection and other steps. Therefore, the coding cost of the same video frame may be repeatedly calculated, further increasing the amount of calculation and delay; third, the backward video frame is referenced when calculating the coding cost, so the reference relationship is more complicated, increasing the calculation complexity and further introducing a greater delay; fourth, the coding cost calculation and I frame detection, P frame detection, block-level coding information calculation, and block-level quantization parameter calculation are completed in series, which is only applicable to software coding scenarios, not hardware coding, and it is difficult to use hardware accelerated coding, so the coding efficiency is difficult to improve. In summary, the existing video frame coding scheme has the disadvantages of high coding cost and low coding efficiency. How to reduce the coding cost of video frames and improve the coding efficiency of video frames has become a technical problem to be solved in this field.

[0056] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The method can improve the determination efficiency of the type of video frames, thereby improving the coding efficiency of video frames.

[0057] Furthermore, the method decouples the steps originally executed serially by the prediction module, so that the calculation of the coding cost can be completed in parallel with the I frame detection and the P frame detection, and the calculation of the coding cost, the I frame detection and the P frame detection can also be completed in parallel with the block-level coding information calculation and the block-level quantization parameter calculation. It is suitable for hardware encoding and can further improve the encoding efficiency.

[0058] Furthermore, the method uses only the forward video frame as a reference frame during encoding, which can reduce the computational complexity of the encoding cost. And only one reference relationship is needed to calculate the encoding cost, which reduces the number of times the encoding cost is calculated, and can reduce the computational cost of the encoding cost and improve the computational efficiency of the encoding cost.

[0059] Figure 4 An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0060] like Figure 4 As shown, it is assumed that the video frame sequence stored on the first device needs to be transmitted to the second device. In this case, the first device can use the frame type determination method disclosed in the present invention to determine the type of the video frame, complete the grouping of the video frame, calculate the block-level coding information and block-level quantization parameters of each video frame, and then determine the coding order of the video frame according to the preset reference relationship to ensure that when each video frame is encoded, the video frame referenced by the video frame has been encoded and the video frame referenced by the video frame has not yet started to be encoded.

[0061] Each group can be used as a group of pictures (Group Of Pictures, GOP), an I frame can be used as a group of pictures independently, and a P frame can be used as a group of pictures with multiple consecutive B frames in the forward direction. If the nearest frame in the forward direction of a P frame is a non-B frame, the P frame can also be used as a group of pictures independently. A group of pictures can be encoded together, and the size of the group of pictures can be equal to the number of video frames included in the group of pictures.

[0062] The first device may encode the video frame according to the determined encoding order, using the type of the video frame, block-level encoding information, and block-level quantization parameters. The first device may transmit the encoded code stream and information such as the reference relationship and type of each video frame to the second device. After receiving the code stream, the second device may decode the code stream according to the reference relationship and type of each video frame, thereby restoring each video frame, and then reorder the video frames according to the acquisition order of the video frames themselves (such as recorded by the video frame numbers) to restore the video frame sequence.

[0063] Figure 5 A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0064] like Figure 5 As shown, in a possible implementation, the present disclosure proposes a frame type determination method, the method is used to group a video frame sequence and determine the type of the video frame in each group, the video frame sequence includes a plurality of video frames of unknown types, the method includes:

[0065] Step S51, calculating the encoding cost of each type of unknown video frame according to a preset first reference relationship;

[0066] Step S52, taking X unknown-type video frames for which coding costs are calculated, and judging whether the video frame is an intra-frame coded frame according to the coding cost of each unknown-type video frame among the taken video frames;

[0067] Step S53, each time an intra-coded frame is determined, the intra-coded frames determined this time are grouped as a group, where X represents a first preset number and X is an integer greater than 1;

[0068] Step S54, judging whether the video frame is a forward prediction coding frame according to the coding cost of each unknown type of the extracted video frames;

[0069] Step S55, every time a forward prediction coding frame is determined, the video frames whose acquisition time is earlier than the forward prediction coding frame and whose type is unknown are determined as bidirectional prediction interpolation coding frames, and the forward prediction coding frames and bidirectional prediction interpolation coding frames determined this time are grouped as one.

[0070] For example, in step S51, the encoding cost of each type of unknown video frame can be calculated according to the preset first reference relationship. The preset first reference relationship determines which video frames each video frame refers to for encoding when calculating the encoding cost, and an example of the preset first reference relationship is given later. The encoding cost refers to the bit rate of the encoded video frame. There are many types of encoding costs that can be calculated, and examples thereof are given later.

[0071] The coding costs of the video frames may be calculated serially according to the acquisition order of the video frames, or the coding costs of multiple video frames may be calculated in parallel. The embodiment of the present disclosure does not limit the calculation order of the coding costs of the video frames.

[0072] Figure 6 A schematic diagram showing a video frame sequence according to an embodiment of the present disclosure.

[0073] like Figure 6 As shown, it is assumed that the video frame sequence includes video frames 11 to 19 of unknown types that are continuously acquired. The nearest non-B frame in the video frame sequence is video frame 10, and video frame 10 is an I frame. Each video frame is numbered in the order of acquisition time, and the smaller the number, the earlier the acquisition time of the video frame.

[0074] The number of video frames for which the coding cost has been calculated may be recorded. When the number of video frames for which the coding cost has been calculated reaches X, step S52 may be executed to take X video frames of unknown type for which the coding cost has been calculated, and determine whether the video frame is an intra-frame coded frame according to the coding cost of each unknown video frame in the taken video frames.

[0075] The X video frames may be X video frames collected continuously, or may be X video frames collected discontinuously, and the present disclosure does not impose any limitation on this.

[0076] As long as the number of video frames whose coding costs are calculated by executing step S51 reaches X, step S52 can be started to determine whether the video frame type is a P frame. Therefore, after the number of video frames whose coding costs are calculated reaches X, step S51 and step S52 can be performed in parallel, thereby speeding up the frame type determination. X represents a first preset number, and X can be less than the total number of video frames included in the video frame sequence. X can be an integer greater than 1, and examples of its value method are given later.

[0077] by Figure 6 For example, assuming X=4, the 4 video frames taken out are video frames 11 to 14. According to the coding cost of each video frame in video frames 11 to 14, it can be determined whether the video frame is an I frame. The video frame that is not determined to be an I frame may be a P frame or a B frame, so the type is still unknown. The I frame determination of each video frame can be performed according to the acquisition order of each video frame taken out. An exemplary determination method of an I frame is given later.

[0078] Each time an I frame is determined, step S53 may be executed to treat the I frame determined this time as a group. Figure 6 For example, assuming that it is determined that video frames 11 to 13 are not I frames, and video frame 14 is determined to be an I frame, then video frame 14 can be regarded as a group alone. Video frames 11 to 13 are still video frames of unknown type.

[0079] After completing the I frame determination of the extracted video frame, step S54 may be executed to determine whether the video frame is a P frame according to the encoding cost of each unknown type of the extracted video frame. It should be noted that step S54 may be executed after step S53, and the video frame that has been determined as an I frame in step S53 is no longer a video frame of unknown type when step S54 is executed. Therefore, when step S54 is executed, the video frame that has been determined as an I frame in step S53 does not need to participate in the P frame determination process.

[0080] by Figure 6 For example, assuming X=4, the extracted video frames are video frames 11 to 14, and when executing step S53, video frame 14 is determined to be an I frame. When executing step S54, only video frames 11 to 13 are video frames of unknown type among the extracted video frames. According to the encoding cost of each video frame in video frames 11 to 13, it can be determined whether the video frame is a P frame. An exemplary determination method of a P frame is given later.

[0081] The P frame judgment of each video frame can be performed according to the acquisition order of the video frames. When a P frame is judged, the P frame judgment of other video frames can be stopped, and step S55 is executed to judge the video frames whose acquisition time is earlier than the P frame and whose type is unknown among the extracted video frames as B frames, and the P frames and B frames judged this time are grouped as one. If when the P frame is judged, there is no video frame whose acquisition time is earlier than the P frame and whose type is unknown among the extracted video frames, then when step S55 is executed, the P frames judged this time can be separately grouped as one.

[0082] by Figure 6 For example, assuming X=4, the video frames taken out are video frames 11 to 14, and video frame 14 has been determined to be an I frame. Assuming that video frame 11 is not determined to be a P frame, and video frame 12 is determined to be a P frame, then video frame 11 can be determined to be a B frame, and video frame 11 and video frame 12 can be grouped together, and whether video frame 13 is a P frame is no longer determined. Assuming that video frame 11 is determined to be a P frame, video frame 11 can be used as a separate group, and whether video frame 12 and video frame 13 are P frames is no longer determined.

[0083] Furthermore, steps S53-S55 and step S51 may also be performed in parallel. After X video frames have been taken out and the frame type determination of the X video frames has begun, when the encoding costs of new X video frames are calculated, the frame type determination of the new X video frames and the frame type determination of the previous X video frames may also be performed in parallel, thereby further reducing the delay of the frame type determination.

[0084] According to the frame type determination method of the embodiment of the present disclosure, for a video frame sequence including a plurality of video frames of unknown types, the coding cost of each unknown type of video frame is calculated according to a preset first reference relationship, thereby completing the calculation of the coding cost; X unknown type of video frames for which the coding cost is calculated are taken, and according to the coding cost of each unknown type of video frame in the taken video frames, whether the video frame is an intra-frame coded frame is determined, that is, after the coding cost of the X unknown type of video frames is obtained, the determination of the I frame can be started, so that the determination of the frame type and the calculation of the coding cost can be carried out in parallel, thereby reducing the delay of the frame type determination; after the X video frames have been taken out and the frame type determination of the X video frames has been started, when the coding cost of the new X video frames is calculated , the frame type judgment of the new X video frames can also be carried out in parallel with the frame type judgment of the previous X video frames, further reducing the delay of the frame type judgment; each time an intra-frame coding frame is judged, the intra-frame coding frame judged this time is taken as a group, and the group including the intra-frame coding frame can be determined; according to the coding cost of each unknown type of video frame in the extracted video frame, it is judged whether the video frame is a forward prediction coding frame, and each time a forward prediction coding frame is judged, the video frame in the extracted video frame whose acquisition time is earlier than the forward prediction coding frame and whose type is unknown is judged as a bidirectional prediction interpolation coding frame, and the forward prediction coding frame judged this time and the bidirectional prediction interpolation coding frame are taken as a group, which can realize the judgment of P frame and B frame, and determine the group including P frame. In this case, the judgment of the type of video frame and the grouping of video frame sequence can be completed for video frame sequence of any length. Since the parallel degree of the steps is improved and the delay of frame type judgment is reduced, the frame type judgment method of the embodiment of the present disclosure can improve the judgment efficiency of the type of video frame, thereby improving the coding efficiency of video frame.

[0085] In a possible implementation, after the forward prediction coding frame and the bidirectional prediction interpolation coding frame determined this time are grouped as one, the method further includes:

[0086] Re-take X video frames of unknown type for which the encoding costs are calculated. When the number of video frames of unknown type is less than X, calculate the encoding costs of all video frames of unknown type and then take out all video frames of unknown type.

[0087] The steps of determining whether the video frame is an intra-frame coded frame and the subsequent steps are performed according to the coding cost of each unknown type of the extracted video frames, until the types of all the video frames in the video frame sequence are determined.

[0088] For example, after determining a group including a P frame, the frame type determination of the video frame extracted this time is completed, and X video frames of unknown type whose coding costs are calculated can be re-obtained, and the coding cost of each unknown video frame in the extracted video frames is determined to determine whether the video frame is an intra-frame coding frame and the subsequent steps (i.e., re-execute steps S52-S55). Wherein, when the number of video frames of unknown type is less than X, after calculating the coding costs of all video frames of unknown type, all video frames of unknown type can be extracted.

[0089] In this way, for a video frame sequence of any length, frame type determination and grouping of all video frames in the video frame sequence can be completed.

[0090] In a possible implementation, the re-retrieved video frames include video frames whose types are still unknown at the current moment among the previously retrieved video frames.

[0091] For example, the video frames retrieved by executing step S52 again may include video frames of undetermined types among the X video frames retrieved when step S52 was executed last time, and other video frames of unknown types whose encoding costs are subsequently calculated. Wherein, when there are S video frames of undetermined types among the X video frames retrieved when step S52 was executed last time (S is a positive integer less than X), there may be XS video frames of unknown types whose encoding costs are subsequently calculated.

[0092] by Figure 6 For example, assuming that X=4, the video frames taken out in the previous step S52 are video frames 11 to 14, and when the frame type judgment is performed on video frames 11 to 14, video frame 14 is judged as an I frame, video frame 11 is judged as a B frame, and video frame 12 is judged as a P frame. When step S52 is performed again, video frame 13 is still a video frame of unknown type, so the video frames taken out in step S52 may include video frame 13. When video frames 15 to 17 have completed the encoding cost and have not been taken out for frame type judgment, the video frames taken out in step S52 may also include video frames 15 to 17. That is, the video frames taken out again include video frame 13 and video frames 15 to 17.

[0093] It can be seen that according to the frame type determination method disclosed in the present invention, for a video frame whose coding cost has been calculated but whose type has not been determined, even if it participates in multiple frame type determinations, it is not necessary to calculate the coding cost again (such as video frame 13), and the frame type determination can be performed again directly based on the calculated coding cost. Therefore, the coding cost of each video frame only needs to be calculated once, avoiding repeated calculations, and reducing the amount of calculation and delay.

[0094] Those skilled in the art should understand that the re-retrieved video frames may also be completely different from the previously retrieved video frames. The present disclosure does not restrict whether the re-retrieved video frames and the previously retrieved video frames include repeated video frames, as long as all video frames in the video frame sequence complete frame type determination and grouping.

[0095] In a possible implementation, the frame type determination method is executed by a frame type determination device, and X is less than the maximum number of video frames that the frame type determination device simultaneously stores.

[0096] For example, the frame type determination method may be executed by a frame type determination device, which may be implemented by pure software or by a combination of software and hardware, and the present disclosure does not limit this. The frame type determination device may be provided with a queue for storing video frames, and the maximum number of video frames stored simultaneously by the frame type determination device may be equal to the depth D' of the queue. The value of X may be set to a value less than D'.

[0097] Furthermore, X may be set according to the image group size required by the application scenario. For example, X may be greater than or equal to the maximum image group size M'. The specific value of X is not limited in the embodiment of the present disclosure.

[0098] In this way, the starting method of the step of determining the video frame type in the present disclosure is made more flexible.

[0099] The following introduces an example of a preset first reference relationship used in an embodiment of the present disclosure, and an exemplary method for calculating encoding costs of various types of unknown video frames according to the preset first reference relationship. Figure 7 A schematic diagram showing a first reference relationship according to an embodiment of the present disclosure.

[0100] In a possible implementation, the first reference relationship is to use the closest intra-frame coded frame or forward prediction coded frame to the current video frame as the first reference frame, and use the previous coded frame to the current video frame as the second reference frame.

[0101] According to the preset first reference relationship, the encoding cost of each type of unknown video frame is calculated, including:

[0102] When each unknown type video frame is used as the current video frame, a first coding cost of encoding the current video frame with reference to the first reference frame, a second coding cost of encoding the current video frame with reference to the second reference frame, and a third coding cost of independent coding are calculated.

[0103] For example, if Figure 7As shown, the first reference relationship can be that the closest intra-frame coded frame or forward predictive coded frame to the current video frame is used as the first reference frame, and the previous coded frame of the current video frame is used as the second reference frame. Assume that video frames 11 to 19 are continuously captured video frames of unknown type, and the type of video frame 10 is known and is an I frame. Each video frame is numbered in the order of acquisition time, and the video frame with a smaller number has an earlier acquisition time. Among them, when video frame 11 is used as the current video frame, the first reference frame and the second reference frame can be the same and be video frame 10; when video frame 12 is used as the current video frame, the first reference frame can be video frame 10, and the second reference frame can be video frame 11.

[0104] When each unknown type video frame is used as the current video frame, a first encoding cost of the current video frame referring to the first reference frame, a second encoding cost referring to the second reference frame, and a third encoding cost of independent encoding may be calculated.

[0105] Each video frame may include multiple coding blocks. The first coding cost of the current video frame is calculated with reference to the first reference frame. The coding cost of each coding block of the current video frame in the inter-frame coding mode is calculated with reference to the first reference frame, and the first coding cost is obtained according to the sum of the coding costs of each coding block in the inter-frame coding mode. The second coding cost of the current video frame is calculated with reference to the second reference frame. The coding cost of each coding block of the current video frame in the inter-frame coding mode is calculated with reference to the second reference frame, and the second coding cost is obtained according to the sum of the coding costs of each coding block in the inter-frame coding mode. The third coding cost of independent coding of the current video frame is calculated. The coding cost of each coding block of the current video frame in the intra-frame coding mode is calculated, and the third coding cost is obtained according to the sum of the coding costs of each coding block in the intra-frame coding mode.

[0106] The first coding cost, the second coding cost, and the third coding cost can be calculated by hardware, and the hardware can realize parallel calculation of the coding costs of multiple video frames, which greatly improves the calculation efficiency of the coding costs and reduces delays.

[0107] Those skilled in the art should understand that as long as the first reference frame is referred to when calculating the first coding cost, the second reference frame is referred to when calculating the second coding cost, and other video frames are not referred to when calculating the third coding cost, and the first coding cost and the second coding cost are the coding costs using the inter-frame coding mode, and the third coding cost is the coding cost using the intra-frame coding mode, the embodiments of the present disclosure do not limit the specific calculation method of the first coding cost, the second coding cost, and the third coding cost.

[0108] The calculation of the first coding cost, the second coding cost, and the third coding cost can be implemented based on the existing technology, and the present disclosure will not elaborate on this.

[0109] When the coding cost is calculated according to the first reference relationship, no reference is made to the backward frame, which is convenient for hardware pipelining to accelerate the coding cost calculation and is suitable for a frame type determination method combining software and hardware.

[0110] The following describes an exemplary implementation method of determining an I frame according to the calculated coding cost according to an embodiment of the present disclosure.

[0111] In a possible implementation, judging whether a video frame is an intra-frame coded frame according to the coding cost of each unknown type of the extracted video frames includes:

[0112] When the third coding cost of the current video frame is less than the product of the first value and the preset proportional coefficient, the current video frame is determined to be an intra-coded frame, and the first value is equal to the smaller one of the first coding cost and the second coding cost of the current video frame.

[0113] For example, the smaller the coding cost of a video frame, the better the coding effect. Therefore, the smaller one of the first coding cost and the second coding cost of the current video frame, that is, the first value, can represent the coding cost of the current video frame as a P frame and only using the inter-frame coding mode. However, when encoding a P frame, the coding block can use the inter-frame coding mode or the intra-frame coding mode. Therefore, a proportional coefficient for representing the impact of the intra-frame coding mode on the coding cost can be preset, and the product of the first value and the proportional coefficient is used to obtain the minimum coding cost of the current video frame as a P frame.

[0114] The third coding cost of the current video frame represents the minimum coding cost of the current video frame as an I frame encoding. By comparing the third coding cost and the product of the first value and the preset proportional coefficient, when the third coding cost is smaller, it can be determined that the effect of encoding the current video frame as an I frame is higher, and thus the current video frame can be determined as an I frame.

[0115] The proportionality coefficient may be set according to the application scenario requirements, and the embodiments of the present disclosure do not limit the specific value of the proportionality coefficient.

[0116] Those skilled in the art should understand that other methods may be used to determine whether a video frame is an I frame based on the coding cost, as long as the minimum coding cost for encoding the current video frame as a P frame and the minimum coding cost for encoding the current video frame as an I frame can be determined and compared. The embodiments of the present disclosure do not limit the specific implementation method of using the coding cost to determine whether a video frame is an I frame.

[0117] The following describes an exemplary implementation method of determining a P frame according to the calculated coding cost according to an embodiment of the present disclosure.

[0118] In a possible implementation, judging whether a video frame is a forward prediction coding frame according to the coding cost of each unknown type of video frame in the retrieved video frames includes:

[0119] When the smaller one of the first encoding cost and the second encoding cost of the current video frame is less than a preset first threshold, the current video frame is determined to be a forward prediction encoding frame.

[0120] For example, the judgment of whether the type of a video frame is a P frame is performed after the judgment of whether the type of the video frame is an I frame. A video frame that has been judged as an I frame will not participate in the judgment of a P frame. Therefore, a video frame that needs to be judged as a P frame may actually only be a P frame or a B frame. Therefore, when judging a P frame, there is no need to use the third coding cost that is only related to the intra-frame coding mode.

[0121] Exemplarily, a smaller one of the first encoding cost and the second encoding cost of the current video frame may be selected, and it is determined whether the smaller encoding cost is less than a preset first threshold. If the smaller encoding cost is less than the first threshold, it can be considered that the encoding cost of the current video frame as a P frame encoding meets the requirements, and the current video frame can be determined to be a P frame; if the smaller encoding cost is greater than or equal to the first threshold, it can be considered that the encoding cost of the current video frame as a P frame encoding is too large and needs to be encoded as a B frame, so it can be determined that the current video frame is not a P frame.

[0122] The first threshold may be set according to application scenario requirements, and the embodiment of the present disclosure does not limit the specific value of the first threshold.

[0123] Those skilled in the art should understand that other methods may be used to determine whether a video frame is a P frame based on the coding cost, as long as the coding cost can be used to determine whether the coding cost of encoding the current video frame as a P frame meets the requirements. The embodiments of the present disclosure do not limit the specific implementation method of using the coding cost to determine whether a video frame is a P frame.

[0124] After a group is obtained, the coding order of each video frame in the group can be further determined, and the video frames in the group can be reordered according to the coding order. An exemplary method for reordering video frames in a group is described below.

[0125] In a possible implementation, according to the number of video frames in each group, a second reference relationship of the video frames in the group is determined;

[0126] The video frames in the group are reordered according to the second reference relationship, and the reordered order is used as the encoding order of the video frames in the group.

[0127] For example, after obtaining a group, the number of video frames included in the group can be determined first. The mapping between the number of video frames in the group and the reference relationship can be preset, and when the number of video frames is different, the reference relationship obtained by mapping can be different. Each time a group is obtained, the reference relationship corresponding to the group can be determined according to the number of video frames included in the group, and the video frames in the group can be reordered according to the determined reference relationship.

[0128] For example, when the number of video frames included in a group is 6, the corresponding reference relationship can be: the 2nd, 4th, and 6th video frames in the group are reference frames. When the number of video frames included in a group is 9, the corresponding reference relationship can be: the 3rd, 6th, and 9th video frames in the group are reference frames. When reordering, the order of reference frames is earlier than the order of non-reference frames.

[0129] For a packet including only I frames or P frames, since it originally includes only one video frame, there is no need to reorder. To distinguish between a packet including only I frames and a packet including only P frames, an I frame identifier may be added to the packet including only I frames.

[0130] In this case, when encoding the video frame group, the actual encoding order can be adapted to the number of video frames included in the video frame group, so that the encoding effect is better.

[0131] Optionally, the frame type determination method of the embodiment of the present disclosure can be executed to further determine the encoding parameters of the video frames in the group, such as the block-level encoding information and block-level quantization parameters of the video frames, and then encode the video frames. The following is an exemplary method of determining the block-level encoding information and block-level quantization parameters of the video frames disclosed in the present disclosure. Figure 8 A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0132] like Figure 8 As shown, in a possible implementation, each video frame includes multiple coding blocks, and the method further includes:

[0133] Step S81, taking Y video frames that have completed grouping and for which block-level coding information has not yet been calculated, and determining block-level coding information of each video frame according to the type of the video frame, the block-level coding information including the coding mode of each coding block and the coding cost of each coding block, where Y represents a second preset number and is an integer greater than 1;

[0134] Step S82, reversely parsing the block-level coding information of the video frames in the most recently determined Z groups, and determining the block-level quantization parameters of the video frames in the earliest determined group among the Z groups, where the block-level quantization parameters include quantization parameters of each coding block, and Z represents a third preset number, and Z is an integer greater than 1;

[0135] Step S83, encoding the video frames in the group using the type, block-level coding information and block-level quantization parameters of the video frames in the group, to obtain coding results of the video frames in the group;

[0136] Re-execute the steps of obtaining Y video frames for which grouping has been completed and block-level coding information has not yet been calculated and the subsequent steps.

[0137] For example, the total number of video frames included in the determined grouping can be counted. When the total number of video frames included in the determined grouping reaches Y, the block-level coding information of Y video frames can be determined, that is, step S81 is executed to take Y video frames that have completed grouping and have not yet calculated the block-level coding information, and determine the block-level coding information of each video frame according to the type of each video frame. The block-level coding information may at least include the coding method of each coding block and the coding cost of each coding block. Optionally, it may also include information related to the coding process, such as the adjustment method of adaptive quantization. The embodiment of the present disclosure does not limit the specific content included in the block-level coding information. Determining the block-level coding information of a video frame according to the type of the video frame can be implemented based on the existing technology and will not be repeated here.

[0138] Y represents the second preset number, which can be set according to the application scenario requirements, for example, it can be equal to the depth D' of the storage queue. The specific value of Y is not limited in the embodiment of the present disclosure.

[0139] After determining the block-level coding information of the video frames in the Z groups, step S82 can be executed to reversely parse the block-level coding information of the video frames in the most recently determined Z groups, and determine the block-level quantization parameters of the video frames in the earliest determined group among the Z groups. The block-level quantization parameters of the video frame may include the quantization parameters of each coding block in the video frame. The reverse parsing can be implemented based on the existing technology, and the details of the reverse parsing are given later. Z represents a preset third number, and Z can be an integer greater than 1. The embodiment of the present disclosure does not limit the specific value of Z.

[0140] When determining the block-level coding information and block-level quantization parameters of the video frames, the video frames also need to be encoded. Optionally, the reordering of the video frames in the group can be completed before determining the block-level coding information of the video frames. In this case, when determining the block-level coding information and block-level quantization parameters of the video frames, the video frames can be encoded according to the reordered coding order, thereby improving the efficiency of determining the block-level coding information and block-level quantization parameters.

[0141] After determining the block-level quantization parameter of a group, step S83 may be executed to encode the video frames in the group using the type of video frames in the group, block-level coding information and block-level quantization parameter to obtain the coding result of the video frames in the group. At the same time, the steps of obtaining Y video frames for which grouping has been completed and block-level coding information has not yet been calculated (step S81) and subsequent steps may be re-executed.

[0142] For example, when Z=3, assuming that the three groups whose block-level coding information is determined most recently are groups 1 to 3, the block-level coding information of groups 1 to 3 can be reversely parsed, and the block-level quantization parameters of the video frames in group 1 can be determined based on the parsing results. Re-execute the steps of taking Y video frames that have been grouped but whose block-level coding information has not yet been calculated, and the block-level coding information of the video frames in group 4 is determined. The three groups whose block-level coding information is determined most recently are changed to groups 2 to 4, and the block-level coding information of groups 2 to 4 can be reversely parsed, and the block-level quantization parameters of the video frames in group 2 can be determined based on the parsing results. Similarly, after the quantization parameters of each group are determined, the steps of taking Y video frames that have been grouped but whose block-level coding information has not yet been calculated and the steps thereafter are executed once, and the block-level coding information and block-level quantization parameters of the video frames in all groups can be determined.

[0143] In this case, the determination of block-level coding information and block-level quantization parameters can be decoupled from the judgment of video frame types and the grouping of video frames and can be completed in parallel, greatly improving the coding efficiency.

[0144] An exemplary method of reverse parsing to determine the block-level quantization parameters of a video frame is described below.

[0145] In a possible implementation, reverse parsing of the block-level coding information of the video frames in the most recently determined Z groups includes:

[0146] Parsing block-level coding information of the video frames in the Z groups in an order opposite to the coding order of the video frames in the Z groups, and determining the importance of each coding block of each video frame in the Z groups;

[0147] According to the importance of each coding block of each video frame in the Z groups, a block-level quantization parameter of a video frame in an earliest determined group among the Z groups is determined.

[0148] For example, the coding order of the video frames in the Z groups has been determined based on the second reference relationship. During reverse parsing, the block-level coding information of the video frames in the Z groups can be parsed in the order opposite to the coding order of the video frames in the Z groups to determine the importance of each coding block of each video frame in the Z groups. For the coding blocks of the I frame, they can be temporarily marked as coding blocks of the P frame. After the quantization parameters of the coding blocks are determined, they can be re-marked as coding blocks of the I frame so that the reverse parsing is also applicable to the coding blocks of the I frame. Calculating the importance of the coding blocks and calculating the block-level quantization parameters of the video frames based on the importance of the coding blocks can be implemented based on the existing technology and will not be repeated here.

[0149] In this way, reverse parsing can be completed to determine the block-level quantization parameters of the video frame. Those skilled in the art should understand that reverse parsing can be implemented using other existing technical methods, and the embodiments of the present disclosure do not limit the specific method of reverse parsing.

[0150] Compared with the existing scheme, the present disclosure decouples the grouping function (including video frame type judgment and grouping) in the prediction module from the coding parameter calculation function (including block-level coding information calculation and block-level quantization parameter calculation), that is, it is no longer necessary to wait until the coding parameter calculation of the video frame in a group is completed before starting to determine the next group, but after determining a group, the next group can be determined, and the coding parameters of the video frame in the group determined this time can be calculated at the same time. If a certain number of video frames are required to start the coding parameter calculation function once, the number of video frames that have completed grouping and have not calculated the block-level coding information can be counted to monitor whether the conditions for starting the coding parameter calculation function once are met; the increase in the number of video frames required to start the coding parameter calculation function once only affects the time consumption of coding parameter calculation, and does not affect the time consumption of video frame judgment and grouping, making the overall function of video frame coding more flexible; the number of video frames for which the coding cost needs to be calculated when starting the grouping function once is reduced from the depth of the video frame storage queue to the maximum number of video frames included in the group, so that the interval between starting the two grouping functions is greatly shortened, and the coding cost required to be calculated when starting the grouping function once is also greatly reduced, thereby reducing the coding delay and improving the coding efficiency.

[0151] I frame judgment and P frame judgment use the same, low-latency reference relationship (first reference relationship), and there is no need to constantly change the reference relationship to repeatedly calculate the encoding cost of the same video frame, which can greatly reduce the calculation complexity and amount of the encoding cost, that is, it simplifies the time-consuming encoding cost calculation process; and this reference relationship does not need to refer to the backward video frame, and is more suitable for hardware implementation.

[0152] Fig. 9 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0153] like Fig. 9As shown, the present disclosure further provides a frame type determination device, the device is used to group a video frame sequence and determine the type of the video frame in each group, the video frame sequence includes a plurality of video frames of unknown types, the device includes:

[0154] A calculation module 91, configured to calculate the encoding cost of each type of unknown video frame according to a preset first reference relationship;

[0155] A first determination module 92 is configured to take X unknown-type video frames for which coding costs are calculated, and determine whether the video frame is an intra-frame coding frame according to the coding cost of each unknown-type video frame among the taken video frames, where X represents a first preset number and is an integer greater than 1;

[0156] A first grouping module 93, configured to group each intra-coded frame determined as a group;

[0157] A second determination module 94 is used to determine whether the video frame is a forward prediction coding frame according to the coding cost of each unknown type of the extracted video frame;

[0158] The second grouping module 95 is used to determine, every time a forward prediction coding frame is determined, a video frame whose acquisition time is earlier than the forward prediction coding frame and whose type is unknown among the extracted video frames is determined as a bidirectional prediction interpolation coding frame, and the forward prediction coding frame and the bidirectional prediction interpolation coding frame determined this time are grouped as one.

[0159] In a possible implementation, the device further includes: a first calling module, configured to call the first judging module to re-take X unknown type video frames for which the coding costs are calculated after the second grouping module groups the forward prediction coding frames and the bidirectional prediction interpolation coding frames judged this time as a group; when the number of unknown type video frames is less than X, the first judging module takes out all unknown type video frames after the coding costs of all unknown type video frames are calculated; and calling the first judging module, the first grouping module, the second judging module, and the second grouping module to execute the steps of judging whether the video frame is an intra-frame coded frame based on the coding cost of each unknown type video frame in the taken out video frames and the subsequent steps until the type judgment of all video frames in the video frame sequence is completed.

[0160] In a possible implementation, the re-retrieved video frames include video frames whose types are still unknown at the current moment among the previously retrieved video frames.

[0161] In a possible implementation, the first reference relationship is to use the intra-frame coded frame or forward prediction coded frame closest to the current video frame as the first reference frame, and the previous coded frame of the current video frame as the second reference frame. The coding costs of various types of unknown video frames are calculated based on the preset first reference relationship, including: when each type of unknown video frame is used as the current video frame, the first coding cost of encoding the current video frame with reference to the first reference frame, the second coding cost of encoding with reference to the second reference frame, and the third coding cost of independent coding are calculated.

[0162] In a possible implementation, the method of judging whether a video frame is an intra-coded frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: judging that the current video frame is an intra-coded frame when a third coding cost of the current video frame is less than a product of a first value and a preset proportional coefficient, and the first value is equal to the smaller one of a first coding cost and a second coding cost of the current video frame.

[0163] In a possible implementation, the method of judging whether a video frame is a forward prediction coding frame based on the coding cost of each type of unknown video frame in the retrieved video frames includes: when the smaller of the first coding cost and the second coding cost of the current video frame is less than a preset first threshold, judging that the current video frame is a forward prediction coding frame.

[0164] In one possible implementation, the device also includes: a first determination module, used to determine a second reference relationship of the video frames in each group based on the number of video frames in the group; and a sorting module, used to reorder the video frames in the group based on the second reference relationship, and the reordered order is used as the encoding order of the video frames in the group.

[0165] In a possible implementation, each video frame includes a plurality of coding blocks, and the device further includes: a second determination module, configured to take Y video frames that have been grouped and for which block-level coding information has not been calculated, and determine the block-level coding information of the video frame according to the type of each video frame, wherein the block-level coding information includes the coding mode of each coding block and the coding cost of each coding block, and Y represents a second preset number, which is an integer greater than 1; a third determination module, configured to reversely parse the block-level coding information of the video frames in the most recently determined Z groups, and determine the block-level quantization parameters of the video frames in the earliest determined group among the Z groups, wherein the block-level quantization parameters include the quantization parameters of each coding block, and Z represents a third preset number, which is an integer greater than 1; an encoding module, configured to encode the video frames in the group using the type of the video frames in the group, the block-level coding information and the block-level quantization parameters, and obtain the encoding result of the video frames in the group; and a second calling module, configured to call the second determination module to re-execute the steps of taking the Y video frames that have been grouped and for which the block-level coding information has not been calculated and the subsequent steps.

[0166] In one possible implementation, the reverse parsing of the block-level coding information of the video frames in the most recently determined Z groups includes: parsing the block-level coding information of the video frames in the Z groups in an order opposite to the coding order of the video frames in the Z groups, and determining the importance of each coding block of each video frame in the Z groups; and determining the block-level quantization parameters of the video frames in the earliest determined group among the Z groups based on the importance of each coding block of each video frame in the Z groups.

[0167] In a possible implementation, the frame type determination method is executed by a frame type determination device, and X is less than the maximum number of video frames that the frame type determination device can store simultaneously.

[0168] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0169] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0170] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0171] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0172] Fig.10 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.10 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0173] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0174] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0175] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0176] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0177] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0178] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0179] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0180] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0181] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0182] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0183] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A frame type determination method, characterized in that: The method is used to group a video frame sequence and determine the type of video frames in each group, wherein the video frame sequence includes a plurality of video frames of unknown types, and the method includes: Calculating the encoding cost of each type of unknown video frame according to a preset first reference relationship; Taking X video frames of unknown type for which coding costs are calculated, and judging whether the video frame is an intra-frame coding frame according to the coding cost of each video frame of unknown type among the taken video frames, where X represents a first preset number and X is an integer greater than 1; Each time an intra-coded frame is determined, the intra-coded frame determined this time is regarded as a group; According to the coding cost of each unknown type of video frame in the extracted video frames, determining whether the video frame is a forward prediction coding frame; Each time a forward prediction coding frame is determined, a video frame whose acquisition time is earlier than that of the forward prediction coding frame and whose type is unknown is determined as a bidirectional prediction interpolation coding frame, and the forward prediction coding frame and the bidirectional prediction interpolation coding frame determined this time are grouped as one.

2. The method according to claim 1, characterized in that After the forward prediction coding frame and the bidirectional prediction interpolation coding frame determined this time are grouped as one, the method further comprises: Re-take X video frames of unknown type for which the encoding costs are calculated. When the number of video frames of unknown type is less than X, calculate the encoding costs of all video frames of unknown type and then take out all video frames of unknown type. The steps of judging whether the video frame is an intra-frame coded frame according to the coding cost of each unknown type of the extracted video frames and subsequent steps are performed until the types of all the video frames in the video frame sequence are judged.

3. The method according to claim 2, characterized in that The re-fetched video frames include video frames whose types are still unknown at the current moment among the previously fetched video frames.

4. The method according to claim 1, characterized in that The first reference relationship is that the closest intra-frame coded frame or forward prediction coded frame to the current video frame is used as the first reference frame, and the previous coded frame to the current video frame is used as the second reference frame. The step of calculating the encoding cost of each type of unknown video frame according to the preset first reference relationship includes: When each unknown type video frame is used as the current video frame, a first encoding cost of encoding the current video frame with reference to the first reference frame, a second encoding cost of encoding the current video frame with reference to the second reference frame, and a third encoding cost of independent encoding are calculated.

5. The method according to claim 4, characterized in that The step of judging whether the video frame is an intra-frame coded frame according to the coding cost of each unknown type of the extracted video frames includes: When the third coding cost of the current video frame is less than the product of the first value and the preset proportional coefficient, the current video frame is judged to be an intra-frame coding frame, and the first value is equal to the smaller one of the first coding cost and the second coding cost of the current video frame.

6. The method according to claim 4, characterized in that The step of judging whether the video frame is a forward prediction coding frame according to the coding cost of each unknown type of the extracted video frames includes: When the smaller one of the first encoding cost and the second encoding cost of the current video frame is less than a preset first threshold, the current video frame is determined to be a forward prediction encoding frame.

7. The method according to claim 1, characterized in that The method further comprises: Determining a second reference relationship of the video frames in each group according to the number of the video frames in the group; The video frames in the group are reordered according to the second reference relationship, and the reordered order is used as the encoding order of the video frames in the group.

8. The method according to claim 7, characterized in that Each video frame includes a plurality of coding blocks, and the method further includes: Taking Y video frames for which grouping has been completed and block-level coding information has not been calculated, and determining block-level coding information of each video frame according to the type of the video frame, wherein the block-level coding information includes a coding mode of each coding block and a coding cost of each coding block, where Y represents a second preset number and is an integer greater than 1; Reversely parse the block-level coding information of the video frames in the most recently determined Z groups, and determine the block-level quantization parameters of the video frames in the earliest determined group among the Z groups, where the block-level quantization parameters include quantization parameters of each coding block, and Z represents a third preset number, and Z is an integer greater than 1; Encoding the video frames in the group using the type, block-level coding information, and block-level quantization parameters of the video frames in the group to obtain coding results of the video frames in the group; Re-execute the steps of obtaining Y video frames for which grouping has been completed and block-level coding information has not yet been calculated and the subsequent steps.

9. The method according to claim 8, characterized in that The reverse parsing of the block-level coding information of the video frames in the most recently determined Z groups includes: Parsing the block-level coding information of the video frames in the Z groups in an order opposite to the coding order of the video frames in the Z groups to determine the importance of each coding block of each video frame in the Z groups; According to the importance of each coding block of each video frame in the Z groups, a block-level quantization parameter of a video frame in an earliest determined group among the Z groups is determined.

10. The method according to claim 1, characterized in that The frame type determination method is performed by a frame type determination device, and X is less than the maximum number of video frames simultaneously stored by the frame type determination device.

11. A frame type determination device, characterized in that: The device is used to group a video frame sequence and determine the type of video frames in each group, the video frame sequence includes a plurality of video frames of unknown types, and the device includes: A calculation module, used for calculating the encoding cost of each type of unknown video frame according to a preset first reference relationship; A first judgment module is used to take X unknown-type video frames whose coding costs are calculated, and judge whether the video frame is an intra-frame coding frame according to the coding cost of each unknown-type video frame among the taken video frames, where X represents a first preset number and is an integer greater than 1; A first grouping module, configured to group each intra-coded frame determined as a group; A second judgment module is used to judge whether the video frame is a forward prediction coding frame according to the coding cost of each unknown type of the extracted video frame; The second grouping module is used to determine, every time a forward prediction coding frame is determined, a video frame whose acquisition time is earlier than the forward prediction coding frame and whose type is unknown among the extracted video frames is determined as a bidirectional prediction interpolation coding frame, and the forward prediction coding frame and the bidirectional prediction interpolation coding frame determined this time are grouped as one.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 10 when executing the instructions stored in the memory.

13. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images