Coding system, method, chip, and electronic device

Abstract

The present disclosure relates to the field of video coding and decoding, and proposes a coding system, method, chip, and electronic device. The coding system includes a thread scheduling layer, a firmware layer, and a hardware layer. The thread scheduling layer and the firmware layer use software resources, and the hardware layer includes hardware resources. The thread scheduling layer is used to apply for hardware layer resources through the firmware layer to determine the type of video frames in a video frame sequence, group the video frame sequence according to the type, perform first-stage coding and second-stage coding on each group of video frames, and output the coding results of the second-stage coding as the code stream corresponding to the group of video frames. Among them, the coding results of any group of video frames in the first stage coding are used to determine the coding parameters used by the group of video frames in the second stage coding, and the reference relationships used by the video frames included in the same group of video frames in the first and second coding stages are the same. The coding system can improve the degree of parallelism between the steps of two-stage coding, coding efficiency, and coding quality.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding, and in particular to a coding system, method, chip, and electronic device. Background Art

[0002] The amount of data in a video frame sequence is very large. In order to reduce transmission power consumption and storage costs, the video frame sequence is usually encoded before transmission or storage. In order to obtain a better encoding effect, it is necessary to first determine the type of each video frame in the video frame sequence. The type of video frame determines whether the video frame needs to refer to other video frames when encoding and what type of video frame to refer to. Video frames can be divided into three types: intra coded picture, forward predictive coded frame, and bidirectional predictive interpolated coded frame. Then the video frame sequence is grouped according to the interval of the intra coded frame, and different groups of pictures (Group Of Pictures, GOP) are encoded separately.

[0003] For a video frame sequence to be encoded, if the GOP structure, frame type, quantization parameters of each frame, and even the type of each coding block of each image frame, the coding information of each coding block, and the quantization parameters of each coding block can be reasonably set, the most efficient encoding result and the best encoding quality will be obtained. Two-pass encoding can achieve this.

[0004] Two-stage encoding is also called two-pass encoding. The first stage of encoding generates necessary encoding parameters such as frame type, block-level encoding information, quantization parameters, etc. The second stage of encoding encodes the video frame according to the encoding parameters. Existing two-stage encoding schemes all use software encoding. Limited by the performance of the central processing unit (CPU) running the software, the complexity of the first stage encoding cannot be too high, and the first stage encoding process is forced to be simplified. Therefore, the accuracy of the encoding parameters determined by the first stage encoding is low; and the encoding efficiency of software encoding is low. In order to improve the encoding efficiency, the first stage encoding and the second stage encoding need to use different reference relationships, resulting in the encoding parameters determined by the first stage encoding. The referenceability of the second stage encoding is reduced. When the accuracy and referenceability of the used encoding parameters are relatively low, the effect of the second stage encoding is difficult to achieve the best, which reduces the encoding quality. Summary of the invention

[0005] In view of this, the present disclosure proposes a coding system, method, chip, and electronic device. The coding system of the present disclosure is divided into multiple levels, and uses hardware resources and software resources to jointly complete the two-stage coding of video frames, which can improve the coding quality of the second stage coding while ensuring high coding efficiency.

[0006] According to one aspect of the present disclosure, a coding system is provided, which is used to implement two-stage coding of video frames, including a thread scheduling layer, a firmware layer and a hardware layer, wherein the thread scheduling layer and the firmware layer use software resources, and the hardware layer includes hardware resources. The thread scheduling layer is used to, in response to receiving a video frame sequence, apply for the resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage coding and second-stage coding on each group of video frames, and output the coding results of the second-stage coding as a code stream corresponding to the group of video frames; wherein the coding results of any group of video frames in the first-stage coding are used to determine the coding parameters used for the group of video frames in the second-stage coding, and the reference relationships used by the video frames included in the same group of video frames in the first coding stage and the second coding stage are the same.

[0007] In a possible implementation, the thread scheduling layer includes a first thread, a second thread, a third thread, and a fourth thread, the hardware layer includes multiple resources, the encoding parameters include block-level encoding information and quantization parameters, the first thread is used to, in response to receiving a video frame sequence, apply for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, and output the type of each group of video frames to the second thread; the second thread is used to, in response to receiving the type of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, the encoding results of the first-stage encoding are used to determine the block-level encoding information of the group of video frames, and the block-level encoding information of the group of video frames is output to the second thread. and types thereof to the third thread; the third thread is used for, in response to receiving the block-level coding information and types of any group of video frames, applying for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, wherein the parsing results are used to determine the quantization parameters of the earliest received group of video frames among the X groups of video frames, and outputting the quantization parameters, block-level coding information and types of the group of video frames to the fourth thread, where X represents a first preset number and X is an integer greater than 1; the fourth thread is used for, in response to receiving the quantization parameters, block-level coding information and types of any group of video frames, applying for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames, and outputting the coding results of the second-stage coding as the bitstream corresponding to the group of video frames.

[0008] In a possible implementation, the types of the video frames include intra-frame coded frames, forward prediction coded frames, and bidirectional prediction interpolation coded frames. The first thread is specifically used to, when the type of the current video frame is an intra-frame coded frame, treat the current video frame as a group of video frames alone; when the type of the current video frame is a forward prediction coded frame, treat the current video frame, the current video frame and the video frames between the current video frame and the nearest forward intra-frame coded frame or forward prediction coded frame as a group of video frames.

[0009] In a possible implementation, the block-level coding information of each video frame indicates the coding method, coding cost, and quantization adjustment method of each coding block of the video frame, and the quantization parameter of each video frame indicates the quantization parameter of each coding block of the video frame.

[0010] In a possible implementation, the first thread is further used to, after grouping the video frame sequence according to the type, determine a reference relationship corresponding to the group of video frames according to the number of video frames included in any group of video frames, and reorder the group of video frames according to the determined reference relationship; the second thread is specifically used to, in response to receiving the type of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform a first stage encoding on the group of video frames in the reordered order.

[0011] In one possible implementation, the second thread is specifically used to, in response to receiving a type of any group of video frames, determine a reference relationship corresponding to the group of video frames according to the number of types of the group of video frames; reorder the video frames included in the group of video frames according to the determined reference relationship, and apply for resources of the hardware layer through the firmware layer to perform a first stage encoding on the group of video frames in the reordered order.

[0012] In a possible implementation, the third thread is specifically configured to, in response to receiving the block-level coding information and types of the X groups of video frames, apply for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the X groups of video frames in an order opposite to the reordered order; and the fourth thread is specifically configured to, in response to receiving quantization parameters, block-level coding information, and types of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames in the reordered order.

[0013] In a possible implementation, the firmware layer includes a first firmware corresponding to the first thread, and the first thread is specifically used to output the video frame sequence to the first firmware in response to receiving the video frame sequence; the first firmware is used to apply for resources of the hardware layer to encode the at least one video frame in response to receiving at least one video frame, and judge the type of the video frame in real time according to the encoding result of each video frame and output it to the first thread, or, when the encoding results of Y video frames are obtained, judge the types of the Y video frames according to the encoding results of the Y video frames and output them to the first thread, where Y represents a second preset number, and Y is an integer greater than 1.

[0014] In one possible implementation, the firmware layer includes a second firmware corresponding to the second thread, and the second thread is specifically used to, in response to receiving the type of any group of video frames, output the group of video frames and the type of the group of video frames to the second firmware; the second firmware is used to, in response to receiving at least one video frame and the type of the at least one video frame, apply for resources of the hardware layer to perform a first-stage encoding on the at least one video frame, determine the block-level encoding information of the video frame based on the encoding result of the first-stage encoding of each video frame, and output it to the second thread.

[0015] In a possible implementation, the firmware layer includes a third firmware corresponding to the third thread, and the third thread is specifically used to, in response to receiving block-level coding information and types of any group of video frames, output the block-level coding information and types of the most recently received X groups of video frames and the X groups of video frames to the third firmware; the third firmware is used to, in response to receiving the block-level coding information and types of the X groups of video frames and the X groups of video frames, apply for resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames, determine quantization parameters of a group of video frames received earliest among the X groups of video frames according to the parsing results, and output them to the third thread.

[0016] In a possible implementation, the firmware layer includes a fourth firmware corresponding to the fourth thread, and the fourth thread is specifically used to, in response to receiving quantization parameters, block-level coding information and types of any group of video frames, output the group of video frames and the quantization parameters, block-level coding information, and types of the group of video frames to the fourth firmware; the fourth firmware is used to, in response to receiving at least one video frame and the quantization parameters, block-level coding information, and type of the at least one video frame, apply for resources of the hardware layer to perform second-stage encoding on the at least one video frame, and output the encoding results of the second-stage encoding of each video frame to the fourth thread.

[0017] In a possible implementation, when any thread applies for resources of the hardware layer through the firmware layer, the application priority of resources with a high load level is lower than the application priority of resources with a low load level.

[0018] According to another aspect of the present disclosure, a coding method is provided, which is applied to a coding system, wherein the system is used to implement two-stage coding of video frames, and includes a thread scheduling layer, a firmware layer and a hardware layer, wherein the thread scheduling layer and the firmware layer use software resources, and the hardware layer includes hardware resources. The method includes: in response to receiving a video frame sequence, the thread scheduling layer applies for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage coding and second-stage coding on each group of video frames, and output the coding results of the second-stage coding as a code stream corresponding to the group of video frames; wherein the coding results of any group of video frames encoded in the first stage are used to determine the coding parameters used for the group of video frames encoded in the second stage, and the reference relationships used by the video frames included in the same group of video frames in the first coding stage and the second coding stage are the same.

[0019] In a possible implementation, the thread scheduling layer includes a first thread, a second thread, a third thread, and a fourth thread, the hardware layer includes multiple resources, the encoding parameters include block-level encoding information and quantization parameters, and the thread scheduling layer, in response to receiving a video frame sequence, applies for the resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding results of the second stage encoding as the code stream corresponding to the group of video frames, including: the first thread, in response to receiving a video frame sequence, applies for the resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, and output the type of each group of video frames to the second thread; the second thread, in response to receiving the type of any group of video frames, applies for the resources of the hardware layer through the firmware layer The hardware layer uses resources of the hardware layer to perform first-stage encoding on the group of video frames, the encoding results of the first-stage encoding are used to determine block-level encoding information of the group of video frames, and the block-level encoding information and types of the group of video frames are output to the third thread; in response to receiving the block-level encoding information and types of any group of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level encoding information of the most recently received X groups of video frames, the parsing results are used to determine the quantization parameters of the earliest received group of video frames among the X groups of video frames, and the quantization parameters, block-level encoding information and types of the group of video frames are output to the fourth thread, where X represents a first preset number and X is an integer greater than 1; in response to receiving the quantization parameters, block-level encoding information and types of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames, and outputs the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames.

[0020] In a possible implementation, the types of the video frames include intra-frame coded frames, forward prediction coded frames, and bidirectional prediction interpolation coded frames, and the video frame sequence is grouped according to the types, including: when the type of the current video frame is an intra-frame coded frame, the current video frame is separately regarded as a group of video frames; when the type of the current video frame is a forward prediction coded frame, the current video frame, the current video frame and the video frames between the current video frame and the nearest forward intra-frame coded frame or forward prediction coded frame are regarded as a group of video frames.

[0021] In a possible implementation, the block-level coding information of each video frame indicates the coding method, coding cost, and quantization adjustment method of each coding block of the video frame, and the quantization parameter of each video frame indicates the quantization parameter of each coding block of the video frame.

[0022] In a possible implementation, the method also includes: after the first thread groups the video frame sequence according to the type, according to the number of video frames included in any group of video frames, determining a reference relationship corresponding to the group of video frames, and reordering the group of video frames according to the determined reference relationship; in response to receiving the type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: in response to receiving the type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames in the reordered order.

[0023] In one possible implementation, in response to receiving a type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: in response to receiving a type of any group of video frames, the second thread determines a reference relationship corresponding to the group of video frames according to the number of types of the group of video frames; reorders the video frames included in the group of video frames according to the determined reference relationship, and applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames in the reordered order.

[0024] In a possible implementation, in response to receiving the block-level coding information and type of any group of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, including: in response to receiving the block-level coding information and type of the X groups of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the X groups of video frames in an order opposite to the reordered order; in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames, including: in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames in an order after the reordering.

[0025] In a possible implementation, the firmware layer includes a first firmware corresponding to the first thread, and the first thread, in response to receiving a video frame sequence, applies for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, including: the first thread, in response to receiving the video frame sequence, outputs the video frame sequence to the first firmware; the first firmware, in response to receiving at least one video frame, applies for resources of the hardware layer to encode the at least one video frame, determines the type of the video frame in real time according to the encoding result of each video frame and outputs it to the first thread, or, when the encoding results of Y video frames are obtained, determines the types of the Y video frames according to the encoding results of the Y video frames and outputs them to the first thread, where Y represents a second preset number, and Y is an integer greater than 1.

[0026] In one possible implementation, the firmware layer includes a second firmware corresponding to the second thread, and the second thread, in response to receiving the type of any group of video frames, applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: the second thread, in response to receiving the type of any group of video frames, outputs the group of video frames and the type of the group of video frames to the second firmware; the second firmware, in response to receiving at least one video frame and the type of the at least one video frame, applies for resources of the hardware layer to perform a first-stage encoding on the at least one video frame, and determines the block-level encoding information of the video frame according to the encoding result of the first-stage encoding of each video frame and outputs it to the second thread.

[0027] In a possible implementation, the firmware layer includes a third firmware corresponding to the third thread, and the third thread, in response to receiving the block-level coding information and type of any group of video frames, applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, including:

[0028] In response to receiving the block-level coding information and type of any group of video frames, the third thread outputs the block-level coding information and type of the most recently received X groups of video frames and the X groups of video frames to the third firmware; in response to receiving the block-level coding information and type of the X groups of video frames and the X groups of video frames, the third firmware applies for resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames, determines quantization parameters of a group of video frames received earliest among the X groups of video frames according to the parsing results, and outputs the quantization parameters to the third thread.

[0029] In a possible implementation, the firmware layer includes a fourth firmware corresponding to the fourth thread, and the fourth thread, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, applies for the resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames, including: the fourth thread, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, outputs the group of video frames and the quantization parameters, block-level coding information and types of the group of video frames to the fourth firmware; the fourth firmware, in response to receiving at least one video frame and the quantization parameters, block-level coding information and type of the at least one video frame, applies for the resources of the hardware layer to perform second-stage encoding on the at least one video frame, and outputs the encoding result of the second-stage encoding of each video frame to the fourth thread.

[0030] In a possible implementation, when any thread applies for resources of the hardware layer through the firmware layer, the application priority of resources with a high load level is lower than the application priority of resources with a low load level.

[0031] According to another aspect of the present disclosure, a chip is provided, comprising the encoding system described above.

[0032] According to another aspect of the present disclosure, an electronic device is provided, comprising the chip described above.

[0033] According to the encoding system of the embodiment of the present disclosure, by setting a thread scheduling layer, a firmware layer and a hardware layer, the thread scheduling layer and the firmware layer use software resources, the hardware layer includes hardware resources, and the thread scheduling layer responds to receiving a video frame sequence, and applies for hardware layer resources through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames. Among them, the encoding results of any group of video frames encoded in the first stage are used to determine the encoding parameters used by the group of video frames in the second stage encoding, and the reference relationship used by each video frame included in the same group of video frames is the same in the first encoding stage and the second encoding stage. Since hardware resources are used in the encoding process, higher encoding complexity can be supported, so that the accuracy of the encoding parameters obtained by the first stage encoding is higher; and hardware encoding has higher encoding efficiency than software encoding, so the first stage encoding and the second stage encoding are supported to use the same reference relationship, so that the encoding parameters determined by the first stage encoding are more referenceable for the encoding of the second encoding stage; when the accuracy and referenceability of the used encoding parameters are relatively high, the encoding quality of the second stage encoding can be improved. In summary, the encoding system of the embodiment of the present disclosure can improve the encoding quality of the second stage encoding while ensuring a high encoding efficiency.

[0034] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0036] Figure 1a A schematic diagram showing the flow of two-stage encoding.

[0037] Figure 1b A schematic diagram showing the encoding time consumption of an existing video encoder model capable of implementing two-stage encoding.

[0038] Figure 2a An exemplary application scenario of the encoding system according to an embodiment of the present disclosure is shown.

[0039] Figure 2b A schematic diagram showing the structure of an encoding system according to an embodiment of the present disclosure.

[0040] Figure 3a A schematic diagram showing the structure of a thread scheduling layer according to an embodiment of the present disclosure.

[0041] Figure 3b A schematic diagram showing a data interaction mode of each thread in a thread scheduling layer according to an embodiment of the present disclosure.

[0042] Figure 4 A schematic diagram showing the time consumption of implementing two-stage encoding of a video frame sequence by an encoding system according to an embodiment of the present disclosure and a video encoder model in the prior art respectively.

[0043] Figure 5 A schematic diagram showing the structure of the firmware layer of the encoding system according to an embodiment of the present disclosure is shown.

[0044] Figure 6 A schematic diagram showing the process of the encoding method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0046] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0047] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0048] Figure 1a A schematic diagram showing the flow of two-stage encoding.

[0049] like Figure 1a As shown, in the first stage, the type of each video frame in the entire video frame sequence is predicted first, and the video frames are grouped based on the frame type. The frame type is used as the encoding parameter for one-pass encoding, and the video frames in each group are encoded once to calculate the block-level encoding information of each video frame, including the encoding method used for each encoding block, the encoding cost of the corresponding encoding method used for each encoding block, the quantization adjustment method, etc. After calculating the block-level encoding information of several groups of video frames, the block-level encoding information of the video frames is reversely parsed to establish a genetic information tree cu_tree. The genetic information tree cu_tree can indicate which encoding block of which frame the image content of each encoding block is inherited from, so that the importance of each encoding block can be determined according to the genetic information tree, and then the quantization parameter of the encoding block can be determined according to the importance of the encoding block. Thus, the quantization parameter qp_map of the video frame is obtained, including the quantization parameter delta_qp of each encoding block of each video frame. Then enter the second stage, use the frame type of the video frame, block-level coding information cu_info, and quantization parameter qp_map as the encoding parameters of the second-pass encoding, and perform second-pass encoding on the video frame. The encoding result of the second-pass encoding is output as the code stream corresponding to the group.

[0050] The existing two-stage encoding schemes all use software encoding, and each step is implemented by the central processing unit (CPU). However, the performance of the central processing unit (CPU) is limited, so it is necessary to reduce the encoding complexity and simplify the encoding process to ensure high encoding efficiency. Based on this, the existing two-stage encoding scheme designs encoding steps with strong dependencies to simplify the encoding process, and reduces the data processing cost required for one encoding by reducing the video frame quality, simplifying the encoding parameters, and using fixed simple reference relationships.

[0051] Take the existing open source video encoder model x264 / x265 that can realize two-stage encoding as an example. In this video encoder model, the frame type judgment function and the reverse parsing function are completed in the same software module, so the following steps need to be strictly executed in order: in the first stage, the types of a certain number of video frames are judged, and then the block-level encoding information of these video frames is calculated by encoding, and the reverse parsing module establishes the genetic information tree of these video frames and further obtains the quantization parameters of the video frames; the encoding parameters of these video frames counted in the first stage (including frame type, block-level encoding information, and quantization parameters) are used to perform the second-stage two-pass encoding. In order to reduce the time consumption of the first stage, the video encoder model x264 / x265 downsamples the video frames in one encoding, and uses a preset relatively simple fixed reference relationship to encode the downsampled video frames.

[0052] The reference relationship can be used to determine the encoding order of each video frame. The preset reference relationship can be, for example, that when there are multiple consecutive B frames in the video frame, the middle B frame is always used as the reference frame. Assuming that video frames 1, 2, 3, 4, 5, 6, and 7 are all B frames, video frame 4 in the middle of video frames 1 to 7 can be used as a reference frame, and then video frame 2 in the middle of video frames 1 to 4 can be used as a reference frame, and video frame 6 in the middle of video frames 4 to 7 can be used as a reference frame. Video frames 1, 3, 5, and 7 are not reference frames, so the encoding order can be video frames 4, 2, 6, 1, 3, 5, and 7.

[0053] Figure 1b A schematic diagram showing the encoding time consumption of an existing video encoder model capable of implementing two-stage encoding.

[0054] like Figure 1b As shown, the grouping of a group of video frames can be regarded as the task goal of a frame type judgment task to be completed by the video encoder model. Obtaining the block-level coding information of a group of video frames can be regarded as the task goal of a one-pass coding task to be completed by the video encoder model. Obtaining the quantization parameters of a group of video frames can be regarded as the task goal of a reverse parsing task to be completed by the video encoder model. Obtaining the encoding results of the second stage encoding of a group of video frames can be regarded as the task goal of a two-pass coding task to be completed by the video encoder model.

[0055] Each square represents the time it takes to complete a task. The numbers 1 to 6 above the square represent the number of video frames corresponding to the task. Figure 1b In the example, a reverse parsing task requires block-level coding information of three sets of video frames. Figure 1bIt can be seen that the strongly dependent frame type judgment task, one-pass encoding task, and reverse parsing task must be completed in series, which makes it difficult to use hardware acceleration, resulting in a relatively large delay t1 from the start of receiving the video frame sequence to the start of outputting the bitstream. Figure 1b The total time t2 for the corresponding code stream output to end is also relatively long (taking 4 groups as an example).

[0056] In addition, the accuracy of the block-level coding information obtained by the first-stage coding is reduced by reducing the quality of the video frame, simplifying the coding parameters, and adopting a fixed simple reference relationship, which leads to the fact that the quantization parameters determined by using the block-level coding information to establish the genetic information tree are not accurate enough, that is, the coding parameters determined by the first-stage coding are low in accuracy, and the second-stage coding encodes the video frame according to the coding parameters determined by the first-stage coding. Therefore, the effect of the second-stage coding cannot reach the optimal level, which reduces the coding quality. In addition, in the existing two-stage coding scheme, although a fixed reference relationship is adopted in the first-stage coding to ensure the coding efficiency, in order to improve the coding effect of the second-stage coding, the reference relationship used in the second-stage coding is usually determined according to the frame type. Since the first-stage coding and the second-stage coding adopt different reference relationships, the coding parameters determined by the first-stage coding are not completely consistent with the actual coding situation of the second-stage coding. The coding parameters determined by the first-stage coding have limited reference for the second-stage coding, which further reduces the coding effect of the second-stage coding.

[0057] In view of this, the present disclosure proposes a coding system, method, chip, and electronic device. The coding system of the present disclosure is divided into multiple levels, and uses hardware resources and software resources to jointly complete the two-stage coding of video frames, which can improve the coding quality of the second stage coding while ensuring high coding efficiency.

[0058] Figure 2a An exemplary application scenario of the encoding system according to an embodiment of the present disclosure is shown.

[0059] like Figure 2aAs shown, it is assumed that the video frame sequence stored on the first device needs to be transmitted to the second device. The first device may be provided with a coding system of an embodiment of the present disclosure. After the video frame sequence is input into the coding system, the coding system may perform two-stage coding on the video frame sequence. First, in the first stage, the coding parameters of each video frame are determined, including type, block-level coding information, quantization parameters, etc., and the reference relationship of each video frame is further determined according to the type. The reference relationship determines the coding order of the video frame. Then, in the second stage, each video frame is encoded based on the determined coding parameters and in accordance with the determined coding order, and the code stream corresponding to each video frame and the reference relationship, coding parameters, and other information of each video frame are transmitted to the second device. After the second device receives the code stream, it can decode the code stream according to the reference relationship, coding parameters, and other information of each video frame, thereby restoring each video frame, and then rearrange the video frames according to the acquisition order of the video frames themselves (such as recorded by the number of the video frames) to restore the video frame sequence.

[0060] Figure 2b A schematic diagram showing the structure of an encoding system according to an embodiment of the present disclosure.

[0061] like Figure 2b As shown, in a possible implementation, the system is used to implement two-stage encoding of video frames, including a thread scheduling layer, a firmware layer, and a hardware layer. The thread scheduling layer and the firmware layer use software resources, and the hardware layer includes hardware resources.

[0062] The thread scheduling layer is used to, in response to receiving the video frame sequence, apply for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding result of the second-stage encoding as the code stream corresponding to the group of video frames;

[0063] The encoding results of any group of video frames in the first encoding stage are used to determine the encoding parameters used by the group of video frames in the second encoding stage, and the reference relationships used by the video frames in the same group of video frames in the first encoding stage and the second encoding stage are the same.

[0064] For example, the coding system of the present disclosure is structurally divided into three layers, namely, the thread scheduling layer, the firmware layer, and the hardware layer. Among them, the thread scheduling layer and the firmware layer are both software layers, and the hardware layer is a hardware layer. The thread scheduling layer and the firmware layer can use the software computing resources provided by the central processing unit, and the hardware layer itself may include multiple (hardware) resources, such as a graphics processing unit (GPU) core. The embodiments of the present disclosure do not limit what specific resources the hardware layer includes. The firmware layer is located between the thread scheduling layer and the hardware layer, and can communicate with the thread scheduling layer and the hardware layer respectively.

[0065] Among them, the thread scheduling layer can be responsible for controlling the reception of the video frame sequence in the two-stage encoding, the startup and scheduling of each functional module of the firmware layer, the data interaction with the firmware layer, the output of the code stream, etc. The firmware layer is mainly responsible for the specific implementation and instantiation of each functional module in the two-stage encoding, the application of resources in the hardware layer, the collection and analysis of encoding results, the generation of encoding parameters, and the data interaction with the thread scheduling layer and the hardware layer. The hardware layer is mainly responsible for the encoding operation of the video frame and the data interaction with the firmware layer.

[0066] When the encoding system encodes the video frame, the thread scheduling layer, in response to receiving the video frame sequence, can apply for the hardware layer's resources through the firmware layer to complete the encoding. In this process, the thread scheduling layer can transmit the video frame to be encoded (optionally, also including encoding parameters) to the firmware layer, the firmware layer initiates a resource application, the hardware layer allocates resources in response to the resource application, and obtains the video frame to be encoded from the firmware layer (optionally, also including encoding parameters), uses the allocated resources, completes the encoding of the video frame based on the information obtained from the firmware layer, and transmits the encoding result to the firmware layer. If the encoding result is not the encoding result of the second stage encoding, the firmware layer generates new encoding parameters based on the encoding result and transmits it to the thread scheduling layer. If the encoding result is already the encoding result of the second stage encoding, the firmware layer can directly output the encoding result as a code stream to the thread scheduling layer. The thread scheduling layer can determine the next step to be performed based on the received encoding parameters, and when receiving the encoding result of the second stage encoding, it can output the encoding result as a code stream to the electronic device to receive the video frame sequence, such as the second device described above.

[0067] When the encoding system encodes the video frames, the encoding results of any group of video frames encoded in the first stage are used to determine the encoding parameters used by the group of video frames in the second stage. Since hardware resources are used for encoding, the encoding efficiency of hardware resources is usually high, so the video frames can still maintain a high encoding efficiency when a more complex reference relationship is used in the first encoding stage, that is, the reference relationship used by each video frame included in the same group of video frames in the first encoding stage and the second encoding stage can be the same. For examples of reference relationships, please refer to the relevant description below.

[0068] The structure of the thread scheduling layer and the exemplary manner in which the thread scheduling layer implements the above functions can be found in the following text and Figure 3a The structure of the firmware layer and the exemplary way in which the thread scheduling layer implements the above functions can be found in the following text and Figure 5 Related description.

[0069] According to the encoding system of the embodiment of the present disclosure, by setting a thread scheduling layer, a firmware layer and a hardware layer, the thread scheduling layer and the firmware layer use software resources, the hardware layer includes hardware resources, and the thread scheduling layer responds to receiving a video frame sequence, and applies for hardware layer resources through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames. Among them, the encoding results of any group of video frames encoded in the first stage are used to determine the encoding parameters used by the group of video frames in the second stage encoding, and the reference relationship used by each video frame included in the same group of video frames is the same in the first encoding stage and the second encoding stage. Since hardware resources are used in the encoding process, higher encoding complexity can be supported, so that the accuracy of the encoding parameters obtained by the first stage encoding is higher; and hardware encoding has higher encoding efficiency than software encoding, so the first stage encoding and the second stage encoding are supported to use the same reference relationship, so that the encoding parameters determined by the first stage encoding are more referenceable for the encoding of the second encoding stage; when the accuracy and referenceability of the used encoding parameters are relatively high, the encoding quality of the second stage encoding can be improved. In summary, the encoding system of the embodiment of the present disclosure can improve the encoding quality of the second stage encoding while ensuring a high encoding efficiency.

[0070] The structure of the thread scheduling layer and an exemplary method for the thread scheduling layer to implement the above functions are introduced below.

[0071] In a possible implementation, the thread scheduling layer includes a first thread, a second thread, a third thread, and a fourth thread, the hardware layer includes a plurality of resources, and the encoding parameters include block-level encoding information and a quantization parameter.

[0072] The first thread is used for, in response to receiving the video frame sequence, applying for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, grouping the video frame sequence according to the type, and outputting the type of each group of video frames to the second thread;

[0073] The second thread is used for, in response to receiving the type of any group of video frames, applying for resources of the hardware layer through the firmware layer to perform first-stage encoding on the group of video frames, the encoding results of the first-stage encoding are used to determine block-level encoding information of the group of video frames, and outputting the block-level encoding information and types of the group of video frames to the third thread;

[0074] The third thread is used for, in response to receiving the block-level coding information and type of any group of video frames, applying for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, and the parsing results are used to determine the quantization parameters of the earliest received group of video frames among the X groups of video frames, and outputting the quantization parameters, block-level coding information and type of each group of video frames to the fourth thread, where X represents the first preset number, and X is an integer greater than 1;

[0075] The fourth thread is used to, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, apply for hardware layer resources through the firmware layer to perform second-stage encoding on the group of video frames, and output the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames.

[0076] Figure 3a A schematic diagram showing the structure of a thread scheduling layer according to an embodiment of the present disclosure.

[0077] like Figure 3a As shown, Figure 3a As shown, the thread scheduling layer may include a first thread, a second thread, a third thread, and a fourth thread, wherein the first thread may start a frame type determination task in two-stage encoding, the second thread may start a one-pass encoding task in two-stage encoding, the third thread may start a reverse parsing task in two-stage encoding, and the fourth thread may start a two-pass encoding task in two-stage encoding. Since the hardware layer includes multiple (hardware) resources, different threads may apply for different resources of the hardware layer at the same time, and tasks started by different threads may be run in parallel; after a thread applies for resources of the hardware layer to complete a task started, it may continue to apply for other resources of the hardware layer to complete another task started, so multiple tasks started by a single thread may be run in parallel. For a single task, multiple resources may also be applied for at the same time to complete the task in a distributed manner, thereby greatly reducing encoding delay and improving encoding efficiency.

[0078] Figure 3b A schematic diagram showing a data interaction mode of each thread in a thread scheduling layer according to an embodiment of the present disclosure.

[0079] like Figure 3b As shown, the video frame sequence may be first received by the first thread. In response to receiving the video frame sequence, the first thread may apply for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence (i.e., start the frame type determination task), and the type of the video frame is one of the encoding parameters required for the second stage encoding of the video frame. The first thread may group the video frame sequence according to the type of the video frame, and output the type of each group of video frames to the second thread.

[0080] Determining the type of each video frame in the video frame sequence can be implemented based on the existing technology, which will not be described in detail here. Each video frame can be determined as an I frame, a P frame, or a B frame. The details of the first thread applying for resources at the hardware layer through the firmware layer, the specific work completed by the applied resources, and the details of the first thread grouping the video frame sequence according to the type are given later.

[0081] Each time a group of video frame types are received, the second thread can apply for hardware layer resources through the firmware layer to perform the first stage encoding of the group of video frames (i.e., start a one-pass encoding task). The encoding results of the first stage encoding of a group of video frames are used to determine the block-level encoding information of the group of video frames. The block-level encoding information of the video frames is one of the encoding parameters required for the second stage encoding of the video frames. Each time the block-level encoding information of a group of video frames is obtained, the second thread can output the block-level encoding information of the group of video frames together with the type of the group of video frames to the third thread.

[0082] The first stage encoding of a group of video frames can be implemented based on existing technologies, which will not be described in detail here. The details of the second thread applying for hardware layer resources through the firmware layer, the specific work completed by the applied resources, and the specific content of the block-level encoding information of a group of video frames are given in the following text.

[0083] Each time a group of video frames' block-level coding information and types are received, the third thread can apply for hardware layer resources once through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames (i.e., start the reverse parsing task), and the parsing results (i.e., the genetic information tree) are used to determine the quantization parameters of the earliest group of video frames received among the X groups of video frames. Each time a group of video frames' quantization parameters are obtained, the third thread can output the quantization parameters, block-level coding information, and types of the group of video frames to the fourth thread.

[0084] The quantization parameter of the video frame is one of the encoding parameters required for the second stage encoding of the video frame. If, when receiving a group of block-level encoding information and types of video frames, the number of video frame groups corresponding to the most recently received block-level encoding information and types is less than X, the block-level encoding information and types of the video frames may continue to be received until the number of video frame groups corresponding to the most recently received block-level encoding information and types is equal to X, and then the task is completed. Wherein X represents the first preset number, X may be an integer greater than 1, and the specific value of X may be determined according to the application scenario requirements, and the present disclosure does not limit the specific value of X.

[0085] Reverse parsing of the X groups of video frames and determining the quantization parameters of the video frames based on the parsing results can be implemented based on the existing technology. The principle can be seen in Figure 1aDetails of the third thread applying for resources at the hardware layer through the firmware layer, the specific work completed by the applied resources, and an example of the specific content of a set of quantization parameters of a video frame are given later.

[0086] Each time a group of video frames' quantization parameters, block-level coding information and types are received, the fourth thread can apply for hardware layer resources through the firmware layer to perform second-stage coding on the group of video frames (i.e., start the second-pass coding task), and the received quantization parameters, block-level coding information (and type) can be used as coding parameters for the second-stage coding. The fourth thread can output the obtained second-stage coding results as the bitstream corresponding to the group of video frames.

[0087] The second stage encoding of a group of video frames can be implemented based on the existing technology, which will not be described in detail here. The details of the fourth thread applying for resources at the hardware layer through the firmware layer and the specific work completed by the applied resources are given later.

[0088] Figure 4 A schematic diagram showing the time consumption of implementing two-stage encoding of a video frame sequence by an encoding system according to an embodiment of the present disclosure and a video encoder model in the prior art, respectively. Figure 4 In the example, each thread of the encoding system of the embodiment of the present disclosure applies for a resource at the hardware layer when starting a task.

[0089] like Figure 4 As shown, it is assumed that the time taken by the encoding system of the present disclosure to complete the task initiated by the thread is the same as the time taken by the video encoder model of the prior art to complete the same task. Each square represents the time taken to complete a task. The numbers 1 to 6 on the square represent the number of video frame groups corresponding to the information (encoding parameters or bitstream or video frame grouping) obtained by completing the task.

[0090] Combination Figure 4 It can be seen that, using the coding system of the present disclosure, after the first task started by the first thread is completed, other tasks started by the first thread can be run in parallel with the task started by the second thread; after the second task started by the first thread is completed, different tasks started by the second thread can be run in parallel; the first to third tasks started by the second thread (in Figure 4 In the example of X=3), the tasks started by the first thread, the second thread, and the third thread can be run in parallel; after the first task started by the third thread is completed, the tasks started by the first thread, the second thread, the third thread, and the fourth thread can be run in parallel. The delay from the start of receiving the video frame sequence to the start of outputting the bitstream is t3, and the delay from the start of receiving the video frame sequence to a certain number of video frame packets ( Figure 4 The total time taken for the code stream output to be completed is t4.

[0091] By comparison, it can be seen that compared with the video encoder model of the prior art, the encoding system of the embodiment of the present disclosure not only reduces the delay from the start of receiving the video frame sequence to the start of outputting the bitstream, but also reduces the overall time consumed from the start of receiving the video frame sequence to the end of outputting the bitstream corresponding to a certain number of video frame groups, which can greatly improve the encoding efficiency.

[0092] Those skilled in the art will understand that if each thread of the encoding system of the embodiment of the present disclosure applies for multiple resources at the hardware layer when starting a task, the time required to complete a task will be further shortened, thereby further reducing the encoding time and encoding delay of the video frame sequence and improving encoding efficiency.

[0093] The steps of decoupling two-stage encoding disclosed in the present invention are four threads, and each thread can apply for resources of the hardware layer to complete the corresponding function. Since the hardware layer includes multiple resources, different threads can apply for different resources of the hardware layer at the same time, and different threads can work in parallel; after a thread applies for resources of the hardware layer, it can continue to apply for other resources of the hardware layer, so a single thread can complete multiple tasks in parallel, further reducing encoding delay and improving encoding efficiency.

[0094] Figure 5 A schematic diagram of the structure of the firmware layer of the coding system according to an embodiment of the present disclosure is shown. Figure 5 An exemplary structure of the firmware layer of the coding system and an exemplary manner in which each thread applies for resources of the hardware layer through the firmware layer are introduced.

[0095] The following describes an exemplary method in which the first thread applies for resources at the hardware layer through the firmware layer.

[0096] like Figure 5 As shown, in a possible implementation, the firmware layer includes a first firmware corresponding to the first thread,

[0097] The first thread is specifically configured to, in response to receiving the video frame sequence, output the video frame sequence to the first firmware;

[0098] The first firmware is used to, in response to receiving at least one video frame, apply for hardware layer resources to encode at least one video frame, determine the type of the video frame in real time according to the encoding result of each video frame and output it to the first thread, or, when the encoding results of Y video frames are obtained, determine the types of Y video frames according to the encoding results of Y video frames and output them to the first thread, where Y represents a second preset number and Y is an integer greater than 1.

[0099] For example, the firmware layer may include a first firmware corresponding to the first thread. After the thread scheduling layer receives the video frame sequence, the first thread starts spontaneously, starts receiving the video frame sequence and outputs it to the first firmware. The first thread may output a video frame to the first firmware once each time a video frame is received, or may output a video frame to the first firmware once each time a certain number of video frames are received. The embodiment of the present disclosure does not limit the number of video frames output by the first thread to the first firmware each time.

[0100] After receiving at least one video frame, the first firmware may apply for resources from the hardware layer to encode the at least one video frame. Exemplarily, the first firmware may initiate a resource application to the hardware layer, and the hardware layer may allocate resources to the first firmware in response to the resource application initiated by the first firmware. After the resources applied by the first firmware obtain each video frame that is expected to be encoded from the first firmware, the first firmware may encode the video frame and send the encoding result of the video frame to the first firmware.

[0101] When the resources applied for by the first firmware obtain video frames from the first firmware, it can also query whether there are obtainable encoding parameters (such as the type of video frame, block-level encoding information, quantization parameters, etc.), and obtain the encoding parameters when the obtainable encoding parameters are queried. When the resources applied for by the first firmware do not obtain the type of video frame from the first firmware, it can be determined that the encoding requirement of the first firmware is frame type judgment. After the resources applied for by the first firmware obtain each video frame expected to be encoded from the first firmware, the video frame can be encoded according to the encoding mode corresponding to the encoding requirement of the first firmware (such as frame-level encoding). Frame-level encoding refers to encoding the entire frame using the same encoding parameters.

[0102] Each time the encoding result of a video frame is obtained, the resources applied for by the first firmware can output the encoding result to the first firmware. The first firmware can determine the type of the video frame in real time according to the encoding result of each video frame and output it to the first thread, further improving the encoding efficiency. When the first firmware obtains the encoding results of Y video frames, it can also determine the types of Y video frames according to the encoding results of Y video frames and output them to the first thread, thereby improving the accuracy of determining the types of the video frames. Wherein Y represents the second preset number, and Y is an integer greater than 1. Y can be less than or equal to the depth of the first firmware (that is, the maximum number of encoding results that the first firmware can store). The present disclosure does not limit the specific value of Y.

[0103] The first firmware can determine the type of the video frame according to the encoding result based on the existing technology, which will not be described in detail here.

[0104] Those skilled in the art should understand that as long as the first firmware can determine the type of video frame according to the encoding result, the embodiments of the present disclosure do not limit when the first firmware starts to determine the type of video frame according to the encoding result and how many types of video frames are determined at one time.

[0105] After receiving the type of the video frame, the first thread starts to group the video frame sequence. An exemplary manner in which the first thread groups the video frame sequence is described below.

[0106] In a possible implementation, the types of video frames include intra-frame coded frames, forward predictive coded frames, and bidirectional predictive interpolation coded frames, and the first thread is specifically used to:

[0107] When the type of the current video frame is an intra-frame coded frame, the current video frame is regarded as a group of video frames alone;

[0108] When the type of the current video frame is a forward predictive coding frame, the current video frame and video frames between the current video frame and the nearest forward intra-frame coding frame or forward predictive coding frame are taken as a group of video frames.

[0109] For example, when grouping video frames, the first thread can query the type of each video frame according to the acquisition order of the video frames. Since the intra-frame coded frame can be encoded independently without referring to other video frames, when the type of the current video frame is an intra-frame coded frame, the first thread can treat the current video frame as a group of video frames. Since the forward predictive coded frame refers to the previous frame, and the bidirectional predictive interpolation coded frame refers to the previous frame and the next frame, when the type of the current video frame is a forward predictive coded frame, the current video frame and the previous video frame need to be grouped into the same group; when the type of the current video frame is a bidirectional predictive interpolation coded frame, the current video frame, the previous video frame, and the next video frame need to be grouped into the same group. That is, each forward predictive coded frame and the adjacent video frames in a sequence consisting of M consecutive forward bidirectional predictive interpolation coded frames refer to each other during encoding, and M is an integer greater than or equal to 0.

[0110] In this case, when the type of the current video frame is a forward predictive coding frame, the current video frame, the current video frame and the nearest forward intra-frame coding frame or the video frames between the forward predictive coding frames can be taken as a group of video frames, that is, each forward predictive coding frame can be grouped with the forward consecutive M bidirectional predictive interpolated coding frames.

[0111] For example, if the video frames are sorted from early to late by the acquisition time of the video frames, and the type of each video frame in a certain video frame sequence is IPBPBBBP, then the first video frame is independently divided into the first group of video frames; the second video frame is independently divided into the second group of video frames (there are 0 forward consecutive bidirectional predictive interpolation coded frames); the third video frame and the fourth video frame are divided into the third group of video frames (there are 1 forward consecutive bidirectional predictive interpolation coded frames); the fifth to eighth video frames are divided into the fourth group of video frames (there are 4 forward consecutive bidirectional predictive interpolation coded frames).

[0112] After completing the grouping of the video frames, the coding order of each video frame in the group can be further determined, and the video frames in the group can be reordered according to the coding order. This step can be completed by the first thread or the second thread. The following first introduces an exemplary method for the first thread to complete the video frame reordering.

[0113] In a possible implementation, the first thread is further used to, after grouping the video frame sequence according to the type, determine a reference relationship corresponding to any group of video frames according to the number of video frames included in the group of video frames, and reorder the group of video frames according to the determined reference relationship;

[0114] The second thread is specifically used for, in response to receiving the type of any group of video frames, applying for resources of the hardware layer through the firmware layer to perform the first stage encoding on the group of video frames in the reordered order.

[0115] For example, after grouping the video frame sequence according to the type, the first thread may first determine the number of video frames included in each group of video frames. The encoding system may pre-store a mapping between the number of video frames in the group and the reference relationship. When the number of video frames is different, the reference relationship obtained by the mapping may be different. The first thread may determine the reference relationship corresponding to any group of video frames based on the number of video frames included in the group, and reorder the group of video frames based on the determined reference relationship.

[0116] For example, when a group of video frames includes 6 video frames, the corresponding reference relationship may be: the 2nd, 4th, and 6th video frames in the group are reference frames. When a group of video frames includes 9 video frames, the corresponding reference relationship may be: the 3rd, 6th, and 9th video frames in the group are reference frames.

[0117] When the first thread outputs the type of the video frame, it can be output in the reordered order or in the order in which the video frame is collected, and the present disclosure does not limit this. If the first thread outputs the type of the video frame in the order in which the video frame is collected, the corresponding video frame identifier can be added to the type of the video frame, so that the second thread, the third thread, and the fourth thread can determine which video frame the type corresponds to based on the identifier of the type.

[0118] Accordingly, in response to receiving the type of any group of video frames, the second thread can apply for resources of the hardware layer through the firmware layer to perform the first stage encoding of the group of video frames in the reordered order. In this case, when encoding the video frame group, the reference relationship actually used can be adapted to the number of video frames included in the video frame group, so that the encoding effect is better.

[0119] An exemplary method for the second thread to complete video frame reordering is described below.

[0120] In a possible implementation, the second thread is specifically used to, in response to receiving the type of any group of video frames, determine the reference relationship corresponding to the group of video frames according to the number of types of the group of video frames;

[0121] The video frames included in the group of video frames are reordered according to the determined reference relationship, and resources of the hardware layer are applied for through the firmware layer to perform the first stage encoding on the group of video frames in the reordered order.

[0122] For example, after receiving the type of a group of video frames, the second thread can first determine the number of types of the group of video frames. The determined number of types is also the number of video frames included in the group of video frames. The encoding system may pre-store a mapping of the number of video frames in the group and the reference relationship. When the number of video frames is different, the reference relationship obtained by the mapping may be different. The second thread can determine the reference relationship corresponding to the group of video frames based on the number of video frames included in any group of video frames, and reorder the group of video frames based on the determined reference relationship. When the second thread outputs the type of video frames to the third thread, it can be output in the reordered order or in the order in which the video frames are collected. The present disclosure does not limit this. If the second thread outputs the type of video frames according to the order in which the video frames are collected, the corresponding video frame identifier can be added to the type of the video frame, so that the third thread and the fourth thread can determine which video frame the type corresponds to based on the type identifier.

[0123] Accordingly, the second thread can apply for resources of the hardware layer through the firmware layer to perform the first stage encoding on the group of video frames in the reordered order. In this case, when encoding the video frame group, the reference relationship actually used can be adapted to the number of video frames included in the video frame group, so that the encoding effect is better.

[0124] In a possible implementation, the third thread is specifically used to, in response to receiving the block-level coding information and type of the X groups of video frames, apply for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the X groups of video frames in an order opposite to the reordered order;

[0125] The fourth thread is specifically used to, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, apply for hardware layer resources through the firmware layer to perform second-stage coding on the group of video frames in the reordered order.

[0126] For example, no matter whether the reordering of the video frames is completed by the first thread or the second thread, when the second thread, the third thread, and the fourth thread output the video frames to the firmware layer, they can all output the video frames in the reordered order. Therefore, the hardware layer resources applied for by the third thread through the firmware layer can reversely parse the block-level coding information of the X group of video frames in the order opposite to the reordered order; the hardware layer resources applied for by the fourth thread through the firmware layer can perform the second stage coding on the group of video frames in the reordered order.

[0127] In this case, the reference relationship used for each group of video frames in the first-stage encoding, reverse parsing and second-stage encoding of the two-stage encoding is the same, which can simplify the reference relationship management of different encoding stages and make the block-level coding information obtained by the first-stage encoding and the quantization parameters obtained by the reverse parsing closer to the actual encoding, the coding parameters are more referenceable, more accurate and reasonable, and the bit rate allocation compression rate and the coding quality of the second-stage encoding are better.

[0128] Those skilled in the art should understand that the embodiments of the present disclosure may also use a fixed reference relationship in the first stage of encoding, as in the prior art, and the embodiments of the present disclosure do not limit whether the second thread, the third thread, and the fourth thread must use the same reference relationship.

[0129] The following describes an exemplary method in which the second thread applies for resources at the hardware layer through the firmware layer.

[0130] like Figure 5 As shown, in a possible implementation, the firmware layer includes a second firmware corresponding to the second thread,

[0131] The second thread is specifically configured to, in response to receiving any group of video frame types, output the group of video frames and the type of the group of video frames to the second firmware;

[0132] The second firmware is used to, in response to receiving at least one video frame and at least one video frame type, apply for hardware layer resources to perform first-stage encoding on at least one video frame, determine block-level encoding information of the video frame based on the encoding result of the first-stage encoding of each video frame, and output it to the second thread.

[0133] For example, the firmware layer may include a second firmware corresponding to the second thread. When the second thread receives any set of video frame types, the set of video frames and the type of the set of video frames may be output to the second firmware. The second thread may output one video frame at a time, or may output multiple video frames at a time. The embodiment of the present disclosure does not limit the number of video frames output by the second thread to the second firmware each time.

[0134] After receiving at least one video frame and type, the second firmware can apply for resources from the hardware layer to perform the first stage encoding (one pass encoding) on ​​at least one video frame. Exemplarily, the second firmware can initiate a resource application to the hardware layer, and the hardware layer can allocate resources to the second firmware in response to the resource application initiated by the second firmware. The resources applied by the second firmware obtain each video frame that is expected to be encoded from the second firmware. And the corresponding type, the type of the video frame can be used as an encoding parameter to encode the video frame, and the encoding result of the video frame can be sent to the second firmware.

[0135] When the resources applied for by the second firmware obtain video frames from the second firmware, it can also query whether there are available encoding parameters (such as the type of video frame, block-level encoding information, quantization parameters, etc.), and obtain the encoding parameters when the available encoding parameters are queried. When the resources applied for by the second firmware obtain the type of video frame but fail to obtain block-level encoding information and quantization parameters, it can be determined that the encoding requirement of the second firmware is the first stage encoding. The encoding mode of the first stage encoding can be block-level encoding, that is, each encoding block is encoded with the same encoding parameters. The user can pre-configure the encoding block size. During the first stage encoding, the resources can divide each video frame into multiple encoding blocks according to the size of the encoding block, and encode each encoding block as a unit.

[0136] Each time a coding result of the first stage coding of a video frame (including the coding result of each coding block) is obtained, the resources applied for by the second firmware can output the coding result to the second firmware. The second firmware can determine the block-level coding information of the video frame according to the coding result of the first stage coding of each video frame.

[0137] The following introduces exemplary content of block-level coding information of a video frame determined by the second firmware of an embodiment of the present disclosure.

[0138] In a possible implementation, the block-level coding information of each video frame indicates the coding method, coding cost, and quantization adjustment method of each coding block of the video frame.

[0139] Since the first stage encoding is encoding at the encoding block level, the block-level encoding information obtained according to the encoding result of the first stage encoding also includes encoding information at the encoding block level, such as the encoding method of each encoding block of the video frame, the encoding cost of each encoding block under the corresponding encoding method, the quantization adjustment method, etc. The encoding method of the encoding block may include intra-frame coding and inter-frame coding. Among them, intra-frame coding refers to other encoding blocks included in the same video frame, and inter-frame coding refers to encoding blocks of other video frames. The encoding cost of the encoding block refers to the bit rate of the encoding block. The quantization adjustment method of the encoding block indicates the method of adaptively adjusting the quantization parameter of the encoding block.

[0140] Those skilled in the art should understand that the block-level coding information may also include more or less content, as long as the quantization parameters of the video frame can be reversely parsed based on the block-level coding information of the video frame. The embodiments of the present disclosure do not limit the specific content included in the block-level coding information.

[0141] The second firmware determines the block-level coding information of the video frame according to the coding result, which can be implemented based on the existing technology and will not be described in detail here.

[0142] Those skilled in the art should understand that as long as the second firmware can determine the block-level coding information of the video frame based on the coding result, the embodiments of the present disclosure do not limit when the second firmware starts to determine the block-level coding information of the video frame, how many video frames of block-level coding information are determined at one time, or how many coding blocks are determined at one time.

[0143] The following describes an exemplary method in which the third thread applies for resources at the hardware layer through the firmware layer.

[0144] like Figure 5 As shown, in a possible implementation, the firmware layer includes a third firmware corresponding to the third thread,

[0145] The third thread is specifically configured to, in response to receiving the block-level coding information and type of any group of video frames, output the block-level coding information, type and X groups of video frames of the most recently received X groups of video frames to the third firmware;

[0146] The third firmware is used for, in response to receiving block-level coding information and types of X groups of video frames and X groups of video frames, applying for resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames, and determining quantization parameters of a group of video frames received earliest among the X groups of video frames according to the parsing results and outputting them to the third thread.

[0147] For example, the firmware layer may include a third firmware corresponding to the third thread. When the third thread receives the block-level coding information and type of any group of video frames, the block-level coding information, type and X groups of video frames of the most recently received X groups of video frames may be output to the third firmware. Accordingly, if the block-level coding information and type of a group of video frames are received, and the number of video frame groups corresponding to the received block-level coding information and type is less than X, the block-level coding information and type of the video frames will continue to be received until the number of video frame groups corresponding to the received block-level coding information and type is equal to X, and then the block-level coding information, type and X groups of video frames of the most recently received X groups of video frames will be output to the third firmware.

[0148] If the third thread has already output a set of video frames and corresponding block-level coding information and types to the third firmware, there is no need to repeat the transmission when the same set of video frames and corresponding block-level coding information and types need to be output to the third firmware later.

[0149] The third thread may output one video frame at a time, or may output multiple video frames at a time. The embodiment of the present disclosure does not limit the number of video frames output by the third thread to the third firmware at a time.

[0150] After the third firmware receives the block-level coding information and type of the X groups of video frames and the X groups of video frames, it can apply for the resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames. Exemplarily, the third firmware can initiate a resource application to the hardware layer, and the hardware layer can allocate resources to the third firmware in response to the resource application initiated by the third firmware. After the resources applied by the third firmware obtain the X groups of video frames and the corresponding block-level coding information and type from the third firmware, the block-level coding information of the X groups of video frames can be reversely parsed to determine which coding block of which frame the image content of each coding block is inherited from, and then determine the quantization parameters of the earliest received group of video frames among the X groups of video frames according to the parsing results (i.e., the genetic information tree).

[0151] If the resource has already obtained at least one group of video frames in the X groups of video frames before obtaining the X groups of video frames and the corresponding block-level coding information and types from the third firmware, it is not necessary to repeatedly obtain a certain group of video frames and the corresponding block-level coding information and types that have been output to the third firmware, and when the same group of video frames and the corresponding block-level coding information and types need to be output to the third firmware later, it is not necessary to transmit them repeatedly.

[0152] Reference Figure 4 For example, when X=3, the block-level coding information of the first to third groups of video frames can be reversely parsed, and the quantization parameters of the first group of video frames can be determined according to the parsing results; the block-level coding information of the second to fourth groups of video frames can be reversely parsed, and the quantization parameters of the second group of video frames can be determined according to the parsing results.

[0153] The genetic information tree cannot record the genetic information of the coding block of the I frame, but can record the genetic information of the coding block of the P frame and the B frame. Therefore, during reverse parsing, the coding block of the I frame can be added to the genetic information tree as the coding block of the P frame according to the type of the video frame.

[0154] When the resources applied by the third firmware only obtain the video frame, type and block-level coding information from the third firmware, it can be determined that the coding requirement of the third firmware is reverse parsing. Each time a reverse parsing result of a group of video frames is obtained, the resources applied by the third firmware can output the parsing result to the third firmware. The third firmware can determine the quantization parameter of the video frame based on the reverse parsing results of the X groups of video frames. Reverse parsing to construct a genetic information tree and determine the quantization parameter can be implemented based on the existing technology, and the details of reverse parsing will not be repeated here.

[0155] The following introduces exemplary contents of the quantization parameters of the video frames determined by the third firmware of the embodiment of the present disclosure.

[0156] In a possible implementation manner, the quantization parameter of each video frame indicates a quantization parameter of each coding block of the video frame.

[0157] For example, since the block-level coding information as the parsing object includes coding information at the coding block level, the reverse parsing can be parsing at the coding block level. The genetic information tree obtained by the parsing uses coding blocks as nodes. When a coding block includes image content inherited from other coding blocks, there may be a directed edge between the two coding blocks, and the direction of the edge can be used to distinguish the inheritance direction. For example, when there is an edge pointing to coding block 1 between coding block 1 and coding block 2, it can be considered that coding block 1 includes image content inherited from coding block 2.

[0158] In this case, the importance of each coding block can be obtained based on the analysis result, and the quantization parameter of the coding block can be determined according to the importance of the coding block. The quantization parameter of a video frame can be a matrix or set composed of the quantization parameters of all coding blocks included in the video frame. The quantization parameter of a group of video frames can be a set of quantization parameters of each video frame included in the group of video frames.

[0159] The quantization parameter reflects the compression of spatial details. The smaller the value of the quantization parameter, the finer the encoding, the higher the bit rate, the higher the image quality, and the longer the bit stream generated. The larger the value of the quantization parameter, the coarser the encoding, the lower the bit rate, the lower the image quality, and the shorter the bit stream generated.

[0160] The following describes an exemplary method in which the fourth thread applies for resources at the hardware layer through the firmware layer.

[0161] like Figure 5 As shown, in a possible implementation, the firmware layer includes a fourth firmware corresponding to a fourth thread,

[0162] The fourth thread is specifically used to, in response to receiving the quantization parameter, block-level coding information and type of any group of video frames, output the group of video frames and the quantization parameter, block-level coding information and type of the group of video frames to the fourth firmware;

[0163] The fourth firmware is used to apply for hardware layer resources to perform second-stage encoding on at least one video frame in response to receiving at least one video frame and quantization parameters, block-level coding information, and type of at least one video frame, and output the encoding results of the second-stage encoding of each video frame to the fourth thread.

[0164] For example, the firmware layer may include a fourth firmware corresponding to the fourth thread. When the fourth thread receives the quantization parameters, block-level coding information, and type of any set of video frames, the set of video frames and the quantization parameters, block-level coding information, and type of the set of video frames may be output to the fourth firmware. The fourth thread may output one video frame at a time, or may output multiple video frames at a time. The embodiment of the present disclosure does not limit the number of video frames output by the fourth thread to the fourth firmware each time.

[0165] After receiving at least one video frame and the quantization parameter, block-level coding information, and type of the at least one video frame, the fourth firmware may apply for resources at the hardware layer to perform second-stage coding (two-pass coding) on ​​the at least one video frame. Exemplarily, the fourth firmware may initiate a resource application to the hardware layer, and the hardware layer may allocate resources to the fourth firmware in response to the resource application initiated by the fourth firmware. After the resources applied by the fourth firmware obtain each video frame to be encoded and the corresponding quantization parameter, block-level coding information, and type from the fourth firmware, the quantization parameter, block-level coding information, and type of the video frame may be used as coding parameters to encode the video frame, and the coding result of the video frame may be sent to the fourth firmware.

[0166] When the resources applied for by the fourth firmware obtain the video frame, type, block-level coding information, and quantization parameters from the fourth firmware, it can be determined that the coding requirement of the fourth firmware is the second-stage coding. The coding mode of the second-stage coding can be block-level coding, that is, each coding block is encoded using the same coding parameters. The second-stage coding is encoded in units of each coding block, and the coding parameters used when encoding each coding block include the type of video frame to which the coding block belongs, the coding method of the coding block, and the quantization parameters of the coding block. The coding result of the second-stage coding of a video frame can be obtained based on the coding results of all coding blocks included in the second-stage coding of a video frame.

[0167] Each time a coding result of the second stage coding of a video frame is obtained, the resources applied for by the fourth firmware can output the coding result to the fourth firmware. The fourth firmware can directly output the coding result to the fourth thread.

[0168] After receiving the encoding result from the fourth firmware, the fourth thread may start to output the code stream.

[0169] In the disclosed embodiment, the user can pre-configure the encoding delay requirements and encoding performance requirements, the firmware can query the configuration of the thread scheduling layer, and based on this, control the resources of the hardware layer to choose whether to downsample or skip certain encoding steps when encoding video frames.

[0170] In a possible implementation, when any thread applies for resources at the hardware layer through the firmware layer, the application priority of resources with a high load level is lower than the application priority of resources with a low load level.

[0171] For example, in order to balance the resource load of the hardware layer, when any thread applies for resources at the hardware layer through the firmware layer, the application priority of resources with high load level can be lower than that of resources with low load level. In this case, if there are idle resources when applying for resources, the idle resources can be applied first, reducing the probability of reduced coding efficiency due to resource congestion.

[0172] By decoupling the two-stage encoding steps into different firmware independent operations, it is convenient for multi-threaded scheduling of the thread scheduling layer to accelerate the two-stage encoding. It also facilitates the function expansion and function tailoring of the firmware, making the firmware function setting more flexible.

[0173] The present disclosure also proposes an encoding method, Figure 6 A schematic diagram showing the process of the encoding method according to an embodiment of the present disclosure.

[0174] like Figure 6 As shown, in a possible implementation, the encoding method is applied to an encoding system, the system is used to implement two-stage encoding of video frames, including a thread scheduling layer, a firmware layer and a hardware layer, the thread scheduling layer and the firmware layer use software resources, the hardware layer includes multiple resources, and the method includes:

[0175] In response to receiving the video frame sequence, the thread scheduling layer applies for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding result of the second-stage encoding as a code stream corresponding to the group of video frames;

[0176] The encoding results of any group of video frames in the first encoding stage are used to determine the encoding parameters used by the group of video frames in the second encoding stage, and the reference relationships used by the video frames in the same group of video frames in the first encoding stage and the second encoding stage are the same.

[0177] In a possible implementation, the thread scheduling layer includes a first thread, a second thread, a third thread, and a fourth thread, the hardware layer includes multiple resources, the encoding parameters include block-level encoding information and quantization parameters, and the thread scheduling layer, in response to receiving a video frame sequence, applies for the resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding results of the second stage encoding as the code stream corresponding to the group of video frames, including: the first thread, in response to receiving a video frame sequence, applies for the resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, and output the type of each group of video frames to the second thread; the second thread, in response to receiving the type of any group of video frames, applies for the resources of the hardware layer through the firmware layer The hardware layer uses resources of the hardware layer to perform first-stage encoding on the group of video frames, the encoding results of the first-stage encoding are used to determine block-level encoding information of the group of video frames, and the block-level encoding information and types of the group of video frames are output to the third thread; in response to receiving the block-level encoding information and types of any group of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level encoding information of the most recently received X groups of video frames, the parsing results are used to determine the quantization parameters of the earliest received group of video frames among the X groups of video frames, and the quantization parameters, block-level encoding information and types of the group of video frames are output to the fourth thread, where X represents a first preset number and X is an integer greater than 1; in response to receiving the quantization parameters, block-level encoding information and types of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames, and outputs the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames.

[0178] In a possible implementation, the types of the video frames include intra-frame coded frames, forward prediction coded frames, and bidirectional prediction interpolation coded frames, and the video frame sequence is grouped according to the types, including: when the type of the current video frame is an intra-frame coded frame, the current video frame is separately regarded as a group of video frames; when the type of the current video frame is a forward prediction coded frame, the current video frame, the current video frame and the video frames between the current video frame and the nearest forward intra-frame coded frame or forward prediction coded frame are regarded as a group of video frames.

[0179] In a possible implementation, the block-level coding information of each video frame indicates the coding method, coding cost, and quantization adjustment method of each coding block of the video frame, and the quantization parameter of each video frame indicates the quantization parameter of each coding block of the video frame.

[0180] In a possible implementation, the method also includes: after the first thread groups the video frame sequence according to the type, according to the number of video frames included in any group of video frames, determining a reference relationship corresponding to the group of video frames, and reordering the group of video frames according to the determined reference relationship; in response to receiving the type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: in response to receiving the type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames in the reordered order.

[0181] In one possible implementation, in response to receiving a type of any group of video frames, the second thread applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: in response to receiving a type of any group of video frames, the second thread determines a reference relationship corresponding to the group of video frames according to the number of types of the group of video frames; reorders the video frames included in the group of video frames according to the determined reference relationship, and applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames in the reordered order.

[0182] In a possible implementation, in response to receiving the block-level coding information and type of any group of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, including: in response to receiving the block-level coding information and type of the X groups of video frames, the third thread applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the X groups of video frames in an order opposite to the reordered order; in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames, including: in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, the fourth thread applies for resources of the hardware layer through the firmware layer to perform second-stage coding on the group of video frames in an order after the reordering.

[0183] In a possible implementation, the firmware layer includes a first firmware corresponding to the first thread, and the first thread, in response to receiving a video frame sequence, applies for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, including: the first thread, in response to receiving the video frame sequence, outputs the video frame sequence to the first firmware; the first firmware, in response to receiving at least one video frame, applies for resources of the hardware layer to encode the at least one video frame, determines the type of the video frame in real time according to the encoding result of each video frame and outputs it to the first thread, or, when the encoding results of Y video frames are obtained, determines the types of the Y video frames according to the encoding results of the Y video frames and outputs them to the first thread, where Y represents a second preset number, and Y is an integer greater than 1.

[0184] In one possible implementation, the firmware layer includes a second firmware corresponding to the second thread, and the second thread, in response to receiving the type of any group of video frames, applies for resources of the hardware layer through the firmware layer to perform a first-stage encoding on the group of video frames, including: the second thread, in response to receiving the type of any group of video frames, outputs the group of video frames and the type of the group of video frames to the second firmware; the second firmware, in response to receiving at least one video frame and the type of the at least one video frame, applies for resources of the hardware layer to perform a first-stage encoding on the at least one video frame, and determines the block-level encoding information of the video frame according to the encoding result of the first-stage encoding of each video frame and outputs it to the second thread.

[0185] In a possible implementation, the firmware layer includes a third firmware corresponding to the third thread, and the third thread, in response to receiving the block-level coding information and type of any group of video frames, applies for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, including:

[0186] In response to receiving the block-level coding information and type of any group of video frames, the third thread outputs the block-level coding information and type of the most recently received X groups of video frames and the X groups of video frames to the third firmware; in response to receiving the block-level coding information and type of the X groups of video frames and the X groups of video frames, the third firmware applies for resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames, determines quantization parameters of a group of video frames received earliest among the X groups of video frames according to the parsing results, and outputs the quantization parameters to the third thread.

[0187] In a possible implementation, the firmware layer includes a fourth firmware corresponding to the fourth thread, and the fourth thread, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, applies for the resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames, including: the fourth thread, in response to receiving the quantization parameters, block-level coding information and type of any group of video frames, outputs the group of video frames and the quantization parameters, block-level coding information and types of the group of video frames to the fourth firmware; the fourth firmware, in response to receiving at least one video frame and the quantization parameters, block-level coding information and type of the at least one video frame, applies for the resources of the hardware layer to perform second-stage encoding on the at least one video frame, and outputs the encoding result of the second-stage encoding of each video frame to the fourth thread.

[0188] In a possible implementation, when any thread applies for resources of the hardware layer through the firmware layer, the application priority of resources with a high load level is lower than the application priority of resources with a low load level.

[0189] The present disclosure also provides a chip, comprising the encoding system described above.

[0190] The present disclosure also provides an electronic device, including the chip described above. The electronic device may be a terminal device or a server. The schematic diagram of its structure can be found in Figure 2a .

[0191] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0192] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A coding system, characterized in that The system is used to implement two-stage encoding of video frames, including a thread scheduling layer, a firmware layer and a hardware layer, wherein the thread scheduling layer and the firmware layer use software resources, and the hardware layer includes hardware resources. The thread scheduling layer is used to, in response to receiving the video frame sequence, apply for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding result of the second-stage encoding as a code stream corresponding to the group of video frames; The encoding results of any group of video frames in the first stage are used to determine the encoding parameters used by the group of video frames in the second stage, and the reference relationships used by the video frames in the same group of video frames in the first stage and the second stage are the same.

2. The coding system according to claim 1, characterized in that The thread scheduling layer includes a first thread, a second thread, a third thread, and a fourth thread, the hardware layer includes a plurality of resources, and the encoding parameters include block-level encoding information and quantization parameters. The first thread is used for, in response to receiving the video frame sequence, applying for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, grouping the video frame sequence according to the type, and outputting the type of each group of video frames to the second thread; The second thread is used for, in response to receiving the type of any group of video frames, applying for resources of the hardware layer through the firmware layer to perform first-stage encoding on the group of video frames, the encoding results of the first-stage encoding are used to determine block-level encoding information of the group of video frames, and outputting the block-level encoding information and types of the group of video frames to the third thread; The third thread is used for, in response to receiving the block-level coding information and type of any group of video frames, applying for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the most recently received X groups of video frames, and the parsing results are used to determine the quantization parameters of the earliest received group of video frames among the X groups of video frames, and outputting the quantization parameters, block-level coding information and type of the group of video frames to the fourth thread, where X represents a first preset number and X is an integer greater than 1; The fourth thread is used to, in response to receiving quantization parameters, block-level coding information and types of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames, and output the encoding results of the second-stage encoding as the code stream corresponding to the group of video frames.

3. The coding system according to claim 2, characterized in that The types of the video frames include intra-frame coded frames, forward predictive coded frames, and bidirectional predictive interpolation coded frames, and the first thread is specifically used for: When the type of the current video frame is an intra-frame coded frame, the current video frame is regarded as a group of video frames alone; When the type of the current video frame is a forward predictive coding frame, the current video frame and video frames between the current video frame and the nearest forward intra-frame coding frame or forward predictive coding frame are taken as a group of video frames.

4. The coding system according to claim 2, characterized in that The block-level coding information of each video frame indicates the coding method, coding cost, and quantization adjustment method of each coding block of the video frame. The quantization parameter of each video frame indicates the quantization parameter of each coding block of the video frame.

5. The coding system according to claim 2, characterized in that The first thread is further used to, after grouping the video frame sequence according to the type, determine a reference relationship corresponding to any group of video frames according to the number of video frames included in the group of video frames, and reorder the group of video frames according to the determined reference relationship; The second thread is specifically configured to, in response to receiving a type of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform first-stage encoding on the group of video frames in a reordered order.

6. The coding system according to claim 2, characterized in that The second thread is specifically used for: In response to receiving the type of any group of video frames, determining a reference relationship corresponding to the group of video frames according to the number of types of the group of video frames; The video frames included in the group of video frames are reordered according to the determined reference relationship, and the resources of the hardware layer are applied for through the firmware layer to perform the first stage encoding on the group of video frames in the reordered order.

7. The coding system according to claim 5 or 6, characterized in that: The third thread is specifically configured to, in response to receiving the block-level coding information and type of the X groups of video frames, apply for resources of the hardware layer through the firmware layer to reversely parse the block-level coding information of the X groups of video frames in an order opposite to the reordered order; The fourth thread is specifically used to, in response to receiving quantization parameters, block-level coding information and types of any group of video frames, apply for resources of the hardware layer through the firmware layer to perform second-stage encoding on the group of video frames in a reordered order.

8. The coding system according to claim 2, characterized in that The firmware layer includes first firmware corresponding to the first thread, The first thread is specifically configured to, in response to receiving the video frame sequence, output the video frame sequence to the first firmware; The first firmware is used for, in response to receiving at least one video frame, applying for resources of the hardware layer to encode the at least one video frame, judging the type of the video frame in real time according to the encoding result of each video frame and outputting it to the first thread, or, when the encoding results of Y video frames are obtained, judging the types of the Y video frames according to the encoding results of the Y video frames and outputting them to the first thread, where Y represents a second preset number and Y is an integer greater than 1.

9. The coding system according to claim 2, characterized in that: The firmware layer includes second firmware corresponding to the second thread, The second thread is specifically configured to, in response to receiving a type of any group of video frames, output the group of video frames and the type of the group of video frames to the second firmware; The second firmware is used to, in response to receiving at least one video frame and the type of the at least one video frame, apply for resources of the hardware layer to perform first-stage encoding on the at least one video frame, determine block-level encoding information of the video frame based on the encoding result of the first-stage encoding of each video frame, and output it to the second thread.

10. The coding system according to claim 2, characterized in that The firmware layer includes a third firmware corresponding to the third thread, The third thread is specifically configured to, in response to receiving the block-level coding information and type of any group of video frames, output the block-level coding information and type of the most recently received X groups of video frames and the X groups of video frames to the third firmware; The third firmware is used for, in response to receiving the block-level coding information and type of the X groups of video frames and the X groups of video frames, applying for resources of the hardware layer to reversely parse the block-level coding information of the X groups of video frames, and determining quantization parameters of a group of video frames received earliest among the X groups of video frames according to the parsing results and outputting them to the third thread.

11. The coding system according to claim 2, characterized in that The firmware layer includes fourth firmware corresponding to the fourth thread, The fourth thread is specifically configured to, in response to receiving the quantization parameters, block-level coding information and types of any group of video frames, output the group of video frames and the quantization parameters, block-level coding information and types of the group of video frames to the fourth firmware; The fourth firmware is used to, in response to receiving at least one video frame and the quantization parameter, block-level coding information, and type of the at least one video frame, apply for resources of the hardware layer to perform second-stage encoding on the at least one video frame, and output the encoding result of the second-stage encoding of each video frame to the fourth thread.

12. The coding system according to claim 2, characterized in that When any thread applies for resources of the hardware layer through the firmware layer, the application priority of resources with a high load level is lower than the application priority of resources with a low load level.

13. A coding method, characterized in that: Applied to a coding system, the system is used to implement two-stage coding of video frames, including a thread scheduling layer, a firmware layer and a hardware layer, the thread scheduling layer and the firmware layer use software resources, the hardware layer includes hardware resources, and the method includes: In response to receiving the video frame sequence, the thread scheduling layer applies for resources of the hardware layer through the firmware layer to determine the type of each video frame in the video frame sequence, group the video frame sequence according to the type, perform first-stage encoding and second-stage encoding on each group of video frames, and output the encoding result of the second-stage encoding as a code stream corresponding to the group of video frames; The encoding results of any group of video frames in the first stage are used to determine the encoding parameters used by the group of video frames in the second stage, and the reference relationships used by the video frames in the same group of video frames in the first stage and the second stage are the same.

14. A chip comprising the encoding system according to any one of claims 1-12.

15. An electronic device comprising the chip according to claim 14.

Images