Video bit rate control method, device and electronic device

Abstract

Embodiments of the present invention relate to the field of Internet technology, and in particular, to a method, device, and electronic device for video rate control. The video rate control method comprises: determining m associated frames associated with a target frame from encoded frames; inputting the encoding residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the encoding residual information of the m associated frames; wherein the quantization parameter offset is used for encoding the target frame. In the embodiment of the present invention, the compression of the video by rate control is effectively realized through the RNN prediction model.

Description

[Technical field]

[0001] The embodiments of the present invention relate to the field of Internet technology, and in particular to a video bit rate control method, device and electronic device. [Background technology]

[0002] In the process of video compression, the bit rate control needs to be achieved by adjusting the quantization parameters of each macroblock of the video frame. In order to obtain higher video quality, a better bit rate control algorithm is needed.

[0003] However, in the process of video compression, if the method of directly inputting the original data of the video frame is used to implement bit rate control, it will seriously increase the bandwidth and delay. Therefore, how to implement bit rate control to compress the video has become an urgent problem to be solved. [Summary of the invention]

[0004] The embodiments of the present invention provide a method, device and electronic device for video bit rate control. The embodiments of the present invention can use coding residual information or other information or a combination of multiple groups of information except coding residual information to obtain the quantization parameter or quantization parameter offset of each macroblock of the target frame, and only need to change the recurrent neural network (RNN) training model; similarly, the coded macroblock coding information of the target frame can be used to predict the quantization parameter or quantization parameter offset of the target macroblock of the target frame, and only need to change the RNN prediction model. Through the recurrent neural network RNN ​​prediction model, the compression of the video by bit rate control is effectively achieved.

[0005] In a first aspect, an embodiment of the present invention provides a video bit rate control method, including:

[0006] Determine m associated frames associated with the target frame from the encoded frames;

[0007] Inputting the coding residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the coding residual information of the m associated frames;

[0008] The quantization parameter offset is used for encoding the target frame.

[0009] In one possible implementation manner, before determining m associated frames associated with the target frame from the encoded frames, the method further includes:

[0010] If the current i-th frame image is the first X predicted frames of the same image group GOP, the i-th frame image is encoded and the encoding residual information of the i-th frame image is saved, where X is a preset known value;

[0011] If the current i-th frame image is a predicted frame after the first X predicted frames of the same GOP, the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, where i is 1, 2...Y, Y is the number of image frames contained in the GOP, and k is 1, 2...Y-1.

[0012] In one possible implementation, if the i-th frame image is determined as the target frame, after obtaining the quantization parameter offset of each macroblock of the target frame according to the RNN prediction model, the method further includes:

[0013] Encoding using a quantization parameter offset for each macroblock of the target frame;

[0014] The coding residual information of the m associated frames is updated.

[0015] In one possible implementation, if the kth future frame after the i-th frame image is determined as the target frame, it includes:

[0016] Encoding the i-th frame image to obtain encoding residual information of the i-th frame image;

[0017] Determine m associated frames associated with the kth future frame from the encoded frames including the i-th frame image;

[0018] After the RNN prediction model obtains the quantization parameter offset of the kth future frame according to the coding residual information of the m associated frames, the method further includes: when encoding is performed on the future frame, using the quantization parameter offset of the future frame for encoding.

[0019] In one possible implementation, inputting the encoded residual information of the m associated frames into a recurrent neural network (RNN) prediction model includes:

[0020] Determining a target macroblock in the target frame;

[0021] Determine, from the m associated frames, associated macroblocks corresponding to the target macroblock;

[0022] The coding residual information of the associated macroblock is input into the RNN prediction model, and the RNN prediction model outputs the quantization parameter offset of the target macroblock according to the coding residual information of the associated macroblock.

[0023] In one possible implementation manner, determining the associated macroblocks corresponding to the target macroblock from the m associated frames respectively includes:

[0024] Determine respectively from the m associated frames the corresponding macroblocks at the same position as the target macroblock, or determine the corresponding macroblock according to the position corresponding to the motion vector, and the adjacent macroblocks distributed around the selected macroblock in each associated frame;

[0025] The associated macroblock of the target macroblock is determined according to the selected macroblock and the corresponding adjacent macroblocks in the m associated frames.

[0026] In one possible implementation, determining neighboring macroblocks distributed around the selected macroblock from each associated frame includes:

[0027] A filling macroblock is arranged around the boundary of each associated frame, and the coding residual information of the filling macroblock is determined according to the coding residual information of the boundary macroblock adjacent to the filling macroblock;

[0028] When the selected macroblock is a boundary macroblock of each of the associated frames, the adjacent macroblocks of the selected macroblock include adjacent filling macroblocks.

[0029] One possible implementation method also includes:

[0030] Determine a training data set, the training data set comprising: encoding information of a plurality of image frames, the encoding information of each image frame comprising: encoding residual information and a target quantization parameter offset, wherein the target quantization parameter offset is determined by an MB-tree algorithm;

[0031] The RNN prediction model is trained using the training data set so that the RNN prediction model establishes prediction rules for the encoding residual information of the input multiple image frames and the quantization parameter offset of the output target frame or future frame.

[0032] In a second aspect, an embodiment of the present invention provides a video bit rate control device, including:

[0033] A determination module, used for determining m associated frames associated with the target frame from the encoded frames;

[0034] The output module is used to input the coding residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the coding residual information of the m associated frames.

[0035] One possible implementation method further includes: a first processing module, wherein the first processing module is used to encode the i-th frame image and save the encoding residual information of the i-th frame image if the current i-th frame image is the first X predicted frames of the same image group GOP, where X is a preset known value; if the current i-th frame image is a predicted frame after the first X predicted frames of the same GOP, the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, where i is 1, 2...Y, Y is the number of image frames included in the GOP, and k is 1, 2...Y-1.

[0036] One possible implementation manner further includes: a second processing module, the second processing module is used to encode using the quantization parameter offset of each macroblock of the target frame; and update the encoding residual information of the m associated frames.

[0037] In one possible implementation, the first processing module is specifically used to encode the i-th frame image to obtain the encoding residual information of the i-th frame image; determine m associated frames associated with the k-th future frame from the encoded frames containing the i-th frame image; after the RNN prediction model obtains the quantization parameter offset of the k-th future frame based on the encoding residual information of the m associated frames, it also includes when the encoding is executed on the future frame, using the quantization parameter offset of each macroblock of the future frame for encoding.

[0038] In one possible implementation, the output module is specifically used to determine the target macroblock in the target frame; determine the associated macroblocks corresponding to the target macroblock from the m associated frames respectively; input the coding residual information of the associated macroblock into the RNN prediction model, and the RNN prediction model outputs the quantization parameter offset of the target macroblock according to the coding residual information of the associated macroblock.

[0039] In one possible implementation, the output module is further specifically used to determine the corresponding macroblocks with the same position as the target macroblock from the m associated frames, or determine the corresponding macroblock according to the position corresponding to the motion vector, as well as the adjacent macroblocks distributed around the selected macroblock in each associated frame; determine the associated macroblock of the target macroblock according to the selected macroblock and the corresponding adjacent macroblocks in the m associated frames.

[0040] In one possible implementation, the output module is further specifically used to set a filling macroblock around the boundary of each of the associated frames, and the coding residual information of the filling macroblock is determined based on the coding residual information of the boundary macroblock adjacent to the filling macroblock; when the selected macroblock is a boundary macroblock of each of the associated frames, the adjacent macroblocks of the selected macroblock include the filling macroblocks that are adjacent in position.

[0041] One possible implementation method also includes: a pre-training module, used to determine a training data set, the training data set includes: coding information of several image frames, the coding information of each image frame includes: coding residual information and a target quantization parameter offset, wherein the target quantization parameter offset is determined by an MB-tree algorithm; using the training data set to train the RNN prediction model so that the RNN prediction model establishes a prediction rule for the coding residual information of multiple input image frames and the quantization parameter offset of the output target frame or future frame.

[0042] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0043] at least one processor; and

[0044] at least one memory in communication with the processor, wherein:

[0045] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method provided by the first aspect.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method provided in the first aspect.

[0047] It should be understood that the second to fourth aspects of this specification are consistent with the technical solutions of the first aspect of this specification, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

Brief Description of the Drawings

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a video bit rate control method provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of another video bit rate control method provided by an embodiment of the present invention;

[0051] Figure 3 (a) to (d) are schematic diagrams of determining coding residual information of associated frame macroblocks at non-image boundaries provided by an embodiment of the present invention;

[0052] Figure 4(a) to (d) are schematic diagrams for determining the coding residual information of associated frame macroblocks at the image boundary provided by an embodiment of the present invention;

[0053] Figure 5 A structural diagram of an RNN prediction model provided by an embodiment of the present invention;

[0054] Figure 6 A flowchart of a RNN prediction model pre-training provided by an embodiment of the present invention;

[0055] Figure 7 A video encoding flow chart for predicting a quantization parameter offset provided by an embodiment of the present invention;

[0056] Figure 8 Another video encoding flow chart for predicting quantization parameter offset provided by an embodiment of the present invention;

[0057] Fig. 9 A schematic diagram of the structure of a video bit rate control device provided by an embodiment of the present invention;

[0058] Fig.10 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. [Specific implementation method]

[0059] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0060] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0061] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit this specification. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0062] Figure 1 A flow chart of a video bit rate control method provided by an embodiment of the present invention is as follows Figure 1 As shown, Figure 1 As shown, the method includes:

[0063] Step 101: Determine m associated frames associated with a target frame from encoded frames.

[0064] In some embodiments, the target frame may be encoded by the quantization parameter offset of the target frame. First, more than one associated frame of the target frame should be selected, and the associated frame is an encoded video frame.

[0065] Step 102: input the coding residual information of the m associated frames into a recurrent neural network (RNN) prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the coding residual information of the m associated frames.

[0066] In some embodiments, the quantization parameter offset of the target frame can be directly predicted, and the coded residual information of the entire associated frame can be directly input into the RNN prediction model to obtain the quantization parameter offset of the entire target frame. Another method is to predict the quantization parameter offset of each macroblock of the target frame. First, confirm the corresponding control macroblock of each macroblock in the target frame in each associated frame, and then input the coded residual information of each control macroblock in the associated frame and its reference macroblock into the RNN prediction model, and the quantization parameter offset of each macroblock in the target frame can be obtained.

[0067] When obtaining the coded residual information of each comparison macroblock and its reference macroblock in the associated frame, it is necessary to perform the following steps: Figure 2 The method shown, Figure 2 A flowchart of another video bit rate control method provided by an embodiment of the present invention, the method comprising:

[0068] Step 201, determining a target macroblock in the target frame.

[0069] Step 202: Determine associated macroblocks corresponding to the target macroblock from the m associated frames respectively.

[0070] Step 203: input the coding residual information of the associated macroblock into the RNN prediction model, and the RNN prediction model outputs the quantization parameter offset of the target macroblock according to the coding residual information of the associated macroblock.

[0071] Specifically, when determining the associated macroblock corresponding to the target macroblock, the corresponding macroblocks with the same position as the target macroblock should be determined from the m associated frames respectively, or the corresponding macroblock should be determined according to the position corresponding to the motion vector, as well as the adjacent macroblocks distributed around the corresponding selected macroblock in each associated frame; based on the selected macroblock and the corresponding adjacent macroblocks in the m associated frames, the associated macroblock of the target macroblock is determined.

[0072] like Figure 3 As shown in (a) to (d), Figure 3 (a) to (d) are schematic diagrams of determining coding residual information of associated frame macroblocks at non-image boundaries provided by an embodiment of the present invention. Figure 3(a) shows a target frame, and a macroblock with a checkered pattern in the target frame is determined as a target macroblock. Figure 3 (b)~ Figure 3 (d) shows three associated frames related to the target frame. The macroblocks with checkered patterns in the three associated frames are reference macroblocks with the same positions as the target macroblocks. Figure 3 (b)~ Figure 3 The gray macroblocks in (d) are the adjacent macroblocks surrounding the reference macroblock. The nine-grid macroblocks composed of the gray adjacent macroblocks and the reference macroblocks are determined as the associated macroblocks of the target macroblock. The selection of the associated macroblocks is not limited to the nine-grid form, and the coding residual information of the associated macroblocks is input into the RNN prediction model. The output of the RNN prediction model is the quantization parameter offset of the target macroblock.

[0073] Specifically, adjacent macroblocks distributed around the selected macroblock are determined from each associated frame, including: padding macroblocks are arranged around the boundary of each of the associated frames, and the coding residual information of the padding macroblocks is determined according to the coding residual information of the boundary macroblocks adjacent to the padding macroblocks; when the selected macroblock is a boundary macroblock of each of the associated frames, the adjacent macroblocks of the selected macroblock include the padding macroblocks that are adjacent in position.

[0074] like Figure 4 As shown in (a) to (d), Figure 4 (a) to (d) are structural diagrams of obtaining coding residual information of associated frame macroblocks at the image boundary provided by the embodiment of the present invention. Figure 4 (a) with a striped pattern, the border macroblock should be Figure 4 In (b) to (d), the target frame's associated frame boundary is filled with dotted macroblocks, and the residual information of each dotted macroblock is the same as the coded residual information of its adjacent macroblocks. Figure 3 , Figure 4 The obtained associated frame macroblock coding residual information may also be determined by using the position corresponding to the motion vector. After the coding residual information of the macroblock is determined, the coding residual information of the macroblock with stripe pattern and the dotted line macroblock surrounding the macroblock in each associated frame can be input into the RNN prediction model to obtain the quantization parameter offset of the target macroblock. The structure of the RNN prediction model is as follows: Figure 5 As shown, Figure 5 A structural diagram of an RNN prediction model provided in an embodiment of the present invention. Figure 5 First, the coded residual information of the associated macroblock in the associated frame is input into the RNN prediction model. In this figure, the coded residual information of the associated macroblock of the associated frame related to the current frame is input to Figure 3As shown in the figure, the RNN prediction model processes the coded residual information of the associated macroblocks of the three associated frames, and then passes the output data through the fully connected layers (FC) to obtain the quantization parameter offset QPoffset of the target macroblock in the target frame.

[0075] In some embodiments, before using the RNN prediction model, the RNN prediction model needs to be trained first. Figure 6 As shown, Figure 6 A flowchart of a RNN prediction model pre-training provided by an embodiment of the present invention. The above method includes:

[0076] Step 301, determining a training data set, the training data set includes: coding information of a number of image frames, the coding information of each image frame includes: coding residual information and a target quantization parameter offset, wherein the target quantization parameter offset is determined by an MB-tree algorithm.

[0077] Step 302: train the RNN prediction model using the training data set so that the RNN prediction model establishes prediction rules for the coding residual information of the input multiple image frames and the quantization parameter offset of the output target frame or future frame.

[0078] When training the RNN prediction model, the macroblock tree (Mb-tree) algorithm should be used to track the propagate_cost of the future frame macroblock's dependence on the current frame macroblock, preprocess the future frame to obtain the coding residual information of each macroblock of the future frame, and then calculate the quantization parameter offset of the corresponding macroblock of the current frame according to the corresponding formula. First, according to the variables intra_cost, inter_cost and propagate_in of the future macroblock. The propagate_fraction is obtained by 1-inter_cost / int_cost. The propagate_fraction is the information fraction of the future macroblock's dependence on the target frame macroblock. Then, the information propagate_amount propagated from the target macroblock to the future frame is obtained by (intra_cost+propagate_in)*propagate_fraction, and the propagate_amount is distributed to the corresponding macroblock of the target frame and added to the propagate_cost of the corresponding macroblock. Finally, the quantization parameter offset of the target macroblock is output.

[0079] The formula for calculating the quantization parameter offset is as follows:

[0080] -strength*log2(1+propagate_cost / intra_cost)=QPoffset

[0081] Among them, strength is the empirical factor, and QPoffset is the quantization parameter offset.

[0082] After obtaining the quantization parameter offset, the quantization parameter offset can be used as the training target of the RNN prediction model to train the model. First, select the associated frame from the encoded frame before the target frame. The associated frame can be three video frames adjacent to the target frame. The training process of the RNN prediction model can also be as follows: Figure 5 As shown, in Figure 5 First, the coded residual information of three related frames is input into the RNN prediction model, and the back propagation algorithm is used to calculate the error function of the quantization parameter offset. The network parameters are adjusted and updated after multiple iterations until the error function reaches a reasonable range. After adjusting the network parameters of the RNN prediction model, a trained RNN prediction model can be obtained, and the model can be applied to the prediction of the quantization parameter offset of each macroblock of the video frame.

[0083] Before determining m associated frames associated with the target frame from the encoded frames, the method further includes: if the current i-th frame image is the first X predicted frames of the same image group GOP, encoding the i-th frame image and saving the encoding residual information of the i-th frame image, where X is a preset known value.

[0084] The above method of using the RNN prediction model to predict the quantization parameter offset can be applied to Figure 7 In the video encoding process shown in Figure 7 A video encoding flow chart for predicting a quantization parameter offset is provided in an embodiment of the present invention. Figure 7 The process shown is a coding process for predicting the quantization offset parameters of the current frame when the target frame is the current frame. Figure 7 In the encoding process, the encoding process includes:

[0085] Step 01, start encoding the i-th frame.

[0086] Encode video frames within a Group of Pictures (GOP).

[0087] Step 02, determine whether it is a key frame I frame.

[0088] When it is determined that the i-th frame is a key frame I frame, proceed to step 03; otherwise, proceed to step 04.

[0089] Step 03, encode directly.

[0090] When it is determined that the i-th frame is a key frame I frame, the i-th frame is directly encoded and step 10 is performed.

[0091] Step 04, determine whether it is the first three P frames in the GOP.

[0092] When it is determined that the i-th frame is not a key frame I frame, and when it is determined that the i-th frame is the first three predicted frames P frames in the GOP, proceed to step 05; otherwise, proceed to step 07.

[0093] Step 05, encode directly.

[0094] When it is determined that the i-th frame is one of the first three predicted P frames in the GOP, such as P0, P1 and P2, the i-th frame is directly encoded and step 06 is performed.

[0095] Step 06, save the coding residual information of each macroblock in the frame.

[0096] This step is used to predict the QPoffset of the target frame through the RNN prediction model based on the saved macroblock coding residual information of the associated frame.

[0097] Step 07, obtain the QPoffset of each macroblock in the current frame through RNN.

[0098] When it is determined that the i-th frame is not one of the first three predicted P frames in the GOP, each associated frame in the encoded frame is selected, and the encoded residual information of each associated frame is input into the RNN prediction model to obtain the QPoffset of each macroblock in the current frame.

[0099] Step 08, encoding (QPoffset).

[0100] After obtaining the QPoffset of the current frame through the RNN prediction model, the current frame will be encoded through the QPoffset and step 09 will be performed.

[0101] Step 09: Update the coding residual information of each macroblock of the associated frame.

[0102] The purpose of this step is to continue to use the latest coded residual information when encoding the next frame.

[0103] Step 10, i++.

[0104] After step 03, step 06 and step 09 are completed, it is necessary to jump to step 10 to encode the next frame.

[0105] There is also a situation: if the current i-th frame image is a predicted frame after the first X predicted frames of the same GOP, the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, the value of i is 1, 2...Y, Y is the number of image frames contained in the GOP, and the value of k is 1, 2...Y-1.

[0106] Specifically, if the kth future frame after the i-th frame image is determined as the target frame, then: the i-th frame image is encoded to obtain the encoding residual information of the i-th frame image; m associated frames associated with the k-th future frame are determined from the encoded frames containing the i-th frame image; after the RNN prediction model obtains the quantization parameter offset of the k-th future frame according to the encoding residual information of the m associated frames, the method also includes: when the encoding is executed to the future frame, the quantization parameter offset of each macroblock of the future frame is used for encoding.

[0107] like Figure 8 As shown, Figure 8 Another video encoding flow chart for predicting a quantization parameter offset is provided in an embodiment of the present invention. Figure 8 The process shown is a coding process for predicting the quantization offset parameters of future frames, and the coding process includes:

[0108] Step 001, start encoding the i-th frame.

[0109] Encode the video frames within a GOP.

[0110] Step 002, determine whether it is a key frame I frame.

[0111] When it is determined that the i-th frame is a key frame I frame, proceed to step 003; otherwise, proceed to step 004.

[0112] Step 003, direct encoding.

[0113] When the i-th frame is determined to be a key frame I frame, the i-th frame is directly encoded.

[0114] Step 004, determine whether it is the first three P frames in the GOP.

[0115] When it is determined that the i-th frame is not a key frame I frame, and when it is determined that the i-th frame is the first three predicted frames P frames in the GOP, step 005 is performed; otherwise, step 007 is performed.

[0116] Step 005, direct encoding.

[0117] When it is determined that the i-th frame is the first three predicted P frames in the GOP, such as P0, P1 and P2, the i-th frame is directly encoded and step 006 is performed.

[0118] Step 006, save the coding residual information of each macroblock in the frame.

[0119] This step is used to predict the QPoffset of the target frame through the RNN prediction model based on the saved macroblock coding residual information of the associated frame.

[0120] Step 007, determine whether it is the fourth P frame in the GOP.

[0121] When it is determined that the i-th frame is the fourth predicted frame P frame in the GOP, proceed to step 008; otherwise, proceed to step 012.

[0122] Step 008, direct encoding.

[0123] When the i-th frame is the fourth predicted frame P3 in the GOP, it is necessary to directly encode the frame first and perform step 009.

[0124] Step 009, obtaining the QPoffset of each macroblock in the future frame through RNN.

[0125] The first three associated frames P0, P1 and P2 are selected, and the macroblock residuals of the three associated frames are input into the trained RNN prediction model. The RNN prediction model is used to output the QPoffset of each macroblock of the future frame, i.e., the fifth predicted frame P4, and step 010 is performed.

[0126] Step 010, Last_Qpoffset=Qpoffset.

[0127] Among them, Last_Qpoffset is the QPoffset of the fifth predicted frame P4, and then step 011 is performed.

[0128] Step 011, updating the coding residual information of each macroblock of the associated frame.

[0129] After Last_Qpoffset is obtained, the coding residual information of each macroblock of the associated frames P0, P1 and P2 is automatically updated at the same time. The purpose of updating the coding residual information of each macroblock of each associated frame is to use the latest coding residual information for the next frame.

[0130] Step 012, encode (Last_Qpoffset).

[0131] When it is determined that the i-th frame is not the fourth predicted frame P frame in the GOP, for example, the fifth predicted frame P4, it can be encoded using the previously obtained Last_Qpoffset and enter step 013.

[0132] Step 013, obtain the QPoffset of each macroblock in the future frame through RNN.

[0133] Select associated frames P1, P2 and P3, and input the coding residual information of each macroblock of the three associated frames into the RNN prediction model to obtain the QPoffset of the sixth prediction frame P5 and perform step 014.

[0134] Step 014, Last_qpoffset = QPoffset.

[0135] Wherein Last_qpoffset is the QPoffset of the sixth predicted frame P5, and then step 015 is performed.

[0136] Step 015, updating the coding residual information of each macroblock of the associated frame.

[0137] After obtaining Last_qpoffset, the coding residual information of each macroblock of P1, P2 and P3 of the associated frame is automatically updated at the same time.

[0138] Step 016, i++.

[0139] After step 003, step 006, step 011 and step 015 are completed, it is necessary to jump to step 10 to encode the next frame.

[0140] Fig. 9 A schematic diagram of the structure of a video bit rate control device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the gesture recognition device 400 may include: a determination module 401 and an output module 402 .

[0141] The determination module 401 is used to determine m associated frames associated with the target frame from the encoded frames.

[0142] The output module 402 is used to input the coding residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the coding residual information of the m associated frames.

[0143] In the above embodiment of the present invention, optionally, it also includes: a first processing module, wherein the first processing module is used to encode the i-th frame image and save the encoding residual information of the i-th frame image if the current i-th frame image is the first X predicted frames of the same image group GOP, and X is a preset known value; if the current i-th frame image is a predicted frame after the first X predicted frames of the same GOP, the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, the value of i is 1, 2...Y, Y is the number of image frames included in the GOP, and the value of k is 1, 2...Y-1.

[0144] In the above embodiment of the present invention, optionally, a second processing module is further included, and the second processing module is used to encode using the quantization parameter offset of each macroblock of the target frame; and update the encoding residual information of the m associated frames.

[0145] In the above embodiment of the present invention, optionally, the first processing module is specifically used to encode the i-th frame image to obtain the encoding residual information of the i-th frame image; determine m associated frames associated with the k-th future frame from the encoded frames containing the i-th frame image; after the RNN prediction model obtains the quantization parameter offset of the k-th future frame according to the encoding residual information of the m associated frames, it also includes when the encoding is executed to the future frame, using the quantization parameter offset of the future frame for encoding.

[0146] In the above embodiment of the present invention, optionally, the output module is specifically used to determine the target macroblock in the target frame; determine the associated macroblocks corresponding to the target macroblock from the m associated frames respectively; input the coding residual information of the associated macroblock into the RNN prediction model, and the RNN prediction model outputs the quantization parameter offset of the target macroblock according to the coding residual information of the associated macroblock. In the above embodiment of the present invention, optionally, the providing module is specifically used to provide the target language data to the user in voice or text.

[0147] In the above embodiment of the present invention, optionally, the output module is further specifically used to determine the corresponding macroblocks with the same position as the target macroblock from the m associated frames, or determine the corresponding macroblock according to the position corresponding to the motion vector, as well as the adjacent macroblocks distributed around the selected macroblock in each associated frame; determine the associated macroblock of the target macroblock according to the selected macroblock and the corresponding adjacent macroblocks in the m associated frames.

[0148] In the above embodiment of the present invention, optionally, the output module is further specifically used to set a filling macroblock around the boundary of each of the associated frames, and the coding residual information of the filling macroblock is determined according to the coding residual information of the boundary macroblock adjacent to the filling macroblock; when the selected macroblock is a boundary macroblock of each of the associated frames, the adjacent macroblocks of the selected macroblock include the filling macroblocks located adjacently.

[0149] In the above embodiment of the present invention, optionally, a pre-training module is also included, which is used to determine a training data set, wherein the training data set includes: coding information of a number of image frames, the coding information of each image frame includes: coding residual information and a target quantization parameter offset, wherein the target quantization parameter offset is determined by an MB-tree algorithm; and the RNN prediction model is trained using the training data set so that the RNN prediction model establishes prediction rules for the coding residual information of multiple input image frames and the quantization parameter offset of the output target frame or future frame.

[0150] Fig.10 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention.

[0151] like Fig.10 As shown, the electronic device may include at least one processor; and at least one memory connected to the processor in communication, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the instructions of this specification. Figure 1 , Figure 2 and Figure 6 The illustrated embodiment provides a video bit rate control method.

[0152] The electronic device may be a device capable of performing gesture recognition with a user, such as a cloud server, and the specific form of the electronic device is not limited in the embodiments of this specification. It is understandable that the electronic device here is the machine mentioned in the method embodiment.

[0153] Fig.10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0154] like Fig.10 As shown, the electronic device is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, memory 430, and a communication bus 440 connecting different system components (including memory 430 and processing unit 410).

[0155] The communication bus 440 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0156] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0157] The memory 430 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0158] A program / utility having a set (at least one) of program modules may be stored in memory 430, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0159] The processor 410 executes various functional applications and data processing by running the programs stored in the memory 430, such as implementing the present invention. Figure 1 , Figure 2 and Figure 6 The illustrated embodiment provides a video bit rate control method.

[0160] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the present specification. Figure 1 , Figure 2 and Figure 6 The illustrated embodiment provides a video bit rate control method.

[0161] The above-mentioned computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ReadOnly Memory; hereinafter referred to as: ROM), an erasable programmable read-only memory (Erasable Programmable ReadOnly Memory; hereinafter referred to as: EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0162] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0164] Computer program code for performing the operations of this specification may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0167] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this specification, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0168] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of this specification includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of this specification belong.

[0169] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0170] It should be noted that the terminals involved in the embodiments of this specification may include but are not limited to personal computers (Personal Computer; hereinafter referred to as: PC), personal digital assistants (Personal Digital Assistant; hereinafter referred to as: PDA), wireless handheld devices, tablet computers (Tablet Computer), mobile phones, MP3 players, MP4 players, etc.

[0171] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0173] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk and other media that can store program codes.

[0174] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A video bit rate control method, characterized in that: include: Determine m associated frames associated with the target frame from the encoded frames; Inputting the coding residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, and the RNN prediction model outputs the quantization parameter offset of each macroblock of the target frame according to the coding residual information of the m associated frames; Wherein, the quantization parameter offset is used for encoding the target frame; Before determining m associated frames associated with the target frame from the encoded frames, the method further includes: If the current i-th frame image is a predicted frame after the first X predicted frames of the same GOP, the i-th frame image or the k-th future frame after the i-th frame image is determined as the target frame, where i is 1, 2...Y, Y is the number of image frames contained in the GOP, and k is 1, 2...Y-1.

2. The method according to claim 1, characterized in that: Before determining m associated frames associated with the target frame from the encoded frames, the method further includes: If the current i-th frame image is the first X predicted frames of the same group of images GOP, the i-th frame image is encoded, and the encoding residual information of the i-th frame image is saved, where X is a preset known value.

3. The method according to claim 2, characterized in that If the i-th frame image is determined as the target frame, after obtaining the quantization parameter offset of each macroblock of the target frame according to the RNN prediction model, the method further includes: Encoding using the quantization parameter offset of each macroblock of the target frame; The coding residual information of the m associated frames is updated.

4. The method according to claim 2, characterized in that: If the kth future frame after the i-th frame image is determined as the target frame, then: Encoding the i-th frame image to obtain encoding residual information of the i-th frame image; Determine m associated frames associated with the kth future frame from the encoded frames including the i-th frame image; After the RNN prediction model obtains the quantization parameter offset of the kth future frame based on the coding residual information of the m associated frames, the method further includes: when the encoding is executed to the future frame, encoding is performed using the quantization parameter offset of each macroblock of the future frame.

5. The method according to any one of claims 1 to 4, characterized in that: Inputting the encoded residual information of the m associated frames into a recurrent neural network RNN ​​prediction model, comprising: Determining a target macroblock in the target frame; Determine, from the m associated frames, associated macroblocks corresponding to the target macroblock; The coding residual information of the associated macroblock is input into the RNN prediction model, and the RNN prediction model outputs the quantization parameter offset of the target macroblock according to the coding residual information of the associated macroblock.

6. The method according to claim 5, characterized in that Determining the associated macroblocks corresponding to the target macroblock from the m associated frames respectively includes: Determine respectively from the m associated frames the corresponding macroblocks at the same position as the target macroblock, or determine the corresponding macroblock according to the position corresponding to the motion vector, and the adjacent macroblocks distributed around the selected macroblock in each associated frame; The associated macroblock of the target macroblock is determined according to the selected macroblock and the corresponding adjacent macroblocks in the m associated frames.

7. The method according to claim 6, characterized in that Determine neighboring macroblocks distributed around the selected macroblock from each associated frame, including: A filling macroblock is arranged around the boundary of each associated frame, and the coding residual information of the filling macroblock is determined according to the coding residual information of the boundary macroblock adjacent to the filling macroblock; When the selected macroblock is a boundary macroblock of each of the associated frames, the adjacent macroblocks of the selected macroblock include adjacent filling macroblocks.

8. The method according to claim 1, characterized in that: The method further comprises: Determine a training data set, the training data set comprising: encoding information of a plurality of image frames, the encoding information of each image frame comprising: encoding residual information and a target quantization parameter offset, wherein the target quantization parameter offset is determined by an MB-tree algorithm; The RNN prediction model is trained using the training data set so that the RNN prediction model establishes prediction rules for the encoding residual information of the input multiple image frames and the quantization parameter offset of the output target frame or future frame.

9. An electronic device, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method according to any one of claims 1 to 8.

Images