Time domain extensible video coding method, system and device and storage medium

Abstract

The embodiment of the invention discloses a time domain extensible video coding method, system and device and a storage medium. According to the technical scheme provided by the embodiment of the invention, the hierarchical structure of the to-be-coded picture group is determined, the hierarchical structure is provided with a plurality of hierarchies from high to low according to the set hierarchical rule, each hierarchy comprises a corresponding number of image frames, and the image frame of the higher hierarchy takes the image frame of the lower hierarchy as the reference frame; the image frame at the lowest level takes the first frame of the picture group to be coded as a reference frame; under the condition that time domain scalable video coding is triggered, screening out the image frames of at least one highest level in the to-be-coded picture group, and coding the residual image frames in the to-be-coded picture group; and storing the coded image frame in an image cache, and selecting a reference frame of the to-be-coded image frame from the image cache according to the hierarchical structure, so as to code the to-be-coded image frame based on the selected reference frame. By adopting the technical means, the transmission code rate and CPU (Central Processing Unit) overhead of the extensible video coding can be saved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and in particular, to a time-domain scalable video coding method, system, device and storage medium. Background Art

[0002] At present, in real-time audio and video (RTC) services, SVC (scalable video coding) technology is often used for video encoding. H.264 (H.264 standard, highly compressed digital video codec standard) SVC is based on the H.264AVC video codec standard and utilizes various efficient algorithm tools of the AVC codec (Advanced Video Coding). The encoded video generated by the encoding is scalable in time (frame rate) and space (resolution), and can generate decoded videos with different frame rates, resolutions or quality levels. For example, in a real-time audio and video call scenario, according to the network conditions of the other end, based on the quality and frame rate of the video stream sent on demand based on SVC decisions, it is possible to encode multiple streams of different quality and frame rates, thereby optimizing the audio and video call effect.

[0003] However, the SVC extension standard of H.264 is not supported in hardware decoding and most software decoding, and its popularity is not high. In the RTC group call scenario, not using a mechanism similar to SVC will lead to a significant increase in the bit rate, which will greatly increase the calculation and cache overhead of the video processing CPU (processor), thus affecting the efficiency and quality of video processing. Summary of the invention

[0004] The embodiments of the present application provide a time-domain scalable video coding method, system, device and storage medium, which can save the transmission bit rate and CPU overhead in the RTC group call scenario, and solve the technical problem of excessive video transmission bit rate during RTC group calls.

[0005] In a first aspect, an embodiment of the present application provides a time-domain scalable video coding method, including:

[0006] Determine a hierarchical structure of a group of pictures to be encoded, wherein the hierarchical structure is arranged in multiple levels from high to low according to a set hierarchical rule, each level includes a corresponding number of image frames, an image frame of a high level uses an image frame of a low level as a reference frame, and an image frame of a lowest level uses the first frame of the group of pictures to be encoded as a reference frame;

[0007] In the case of triggering time-domain scalable video coding, the image frames of at least one highest level in the group of pictures to be coded are filtered out, and the remaining image frames in the group of pictures to be coded are coded;

[0008] The encoded image frames are stored in an image buffer, and a reference frame of the image frame to be encoded is selected from the image buffer according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

[0009] In a second aspect, an embodiment of the present application provides a time-domain scalable video coding system, including:

[0010] A hierarchical module is configured to determine a hierarchical structure of a group of pictures to be encoded, wherein the hierarchical structure is arranged in a plurality of levels from high to low according to a set hierarchical rule, wherein each level includes a corresponding number of image frames, wherein image frames of a high level use image frames of a low level as reference frames, and image frames of a lowest level use the first frame of the group of pictures to be encoded as a reference frame;

[0011] A screening module is configured to screen out image frames of at least one highest level in the group of pictures to be encoded and encode the remaining image frames in the group of pictures to be encoded when the time-domain scalable video coding is triggered;

[0012] The encoding module is configured to store the encoded image frame in the image cache, and select a reference frame of the image frame to be encoded from the image cache according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

[0013] In a third aspect, an embodiment of the present application provides a time-domain scalable video encoding device, including:

[0014] memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the time-domain scalable video coding method as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the time-domain scalable video coding method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the time-domain scalable video coding method as described in the first aspect.

[0019] The embodiment of the present application determines the hierarchical structure of the picture group to be encoded, and the hierarchical structure is set up with multiple levels from high to low according to the set hierarchical rules, each level contains a corresponding number of image frames, the image frames of the high level use the image frames of the low level as reference frames, and the image frames of the lowest level use the first frame of the picture group to be encoded as reference frames; in the case of triggering time-domain scalable video coding, the image frames of at least one of the highest levels in the picture group to be encoded are filtered out, and the remaining image frames in the picture group to be encoded are encoded; the encoded image frames are stored in the image cache, and the reference frames of the image frames to be encoded are selected from the image cache according to the hierarchical structure, so as to encode the image frames to be encoded based on the selected reference frames. By adopting the above technical means, by filtering out the image frames of at least one of the highest levels of the picture group to be encoded according to the hierarchical structure, and encoding the image frames to be encoded and caching the encoded image frames according to the reference frame relationship identified by the hierarchical structure, the frame rate adjustment of video encoding can be realized in the scalable video coding process, and the video transmission bit rate can be saved by reducing the video frame rate, thereby adapting to the video encoding requirements in a specific network environment and improving the video processing efficiency and quality. In addition, by reducing the number of image frames to be sent, bandwidth can be saved, the CPU overhead of decoding on the other end can be saved, and the video encoding efficiency can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a time-domain scalable video coding method provided by an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of the hierarchical structure of the screen group in the embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of image frame encoding in an embodiment of the present application;

[0023] Figure 4 is a reference frame selection flow chart in an embodiment of the present application;

[0024] Figure 5 It is a structural diagram of a time-domain scalable video coding system provided by an embodiment of the present application;

[0025] Figure 6 It is a structural diagram of a time-domain scalable video encoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0027] The time-domain scalable video coding method provided in the present application aims to achieve frame rate adjustment of video coding in the scalable video coding process by screening out image frames of at least one highest level of the picture group to be encoded according to the hierarchical structure, and encoding the image frames to be encoded and caching the encoded image frames according to the reference frame relationship identified by the hierarchical structure, so as to save the video transmission bit rate by reducing the video frame rate.

[0028] In video encoding scenarios, the H.264 standard (Highly Compressed Digital Video Codec Standard) is a commonly used codec standard. H.264 encoders often use SVC (scalable video coding) technology for video encoding. H.264SVC is based on the H.264AVC video codec standard and uses various efficient algorithm tools of the AVC codec (Advanced Video Coding). The encoded video generated by the encoding is scalable in time (frame rate) and space (resolution), and can generate decoded videos with different frame rates, resolutions or quality levels. For example, in a real-time audio and video call scenario, based on the network conditions of the other end, the quality and frame rate of the video stream sent on demand based on SVC decisions can be used to encode multiple streams of different quality and frame rates, thereby optimizing the audio and video call effect.

[0029] In addition, the SVC extension standard of H.264 is not supported in hardware decoding and most soft decoding, and its popularity is not high. In the RTC group call scenario, not using a mechanism similar to SVC will lead to a significant increase in the bit rate, thereby greatly increasing the calculation and cache overhead of the video processing CPU (processor), thereby affecting the video processing efficiency and quality. Based on this, a time-domain scalable video coding method of an embodiment of the present application is provided to solve the technical problem of excessive video transmission bit rate during scalable video coding.

[0030] Example:

[0031] Figure 1 A flow chart of a time-domain scalable video coding method provided in an embodiment of the present application is given. The time-domain scalable video coding method provided in this embodiment can be executed by a time-domain scalable video coding device, which can be implemented by software and / or hardware. The time-domain scalable video coding device can be composed of two or more physical entities, or can be composed of one physical entity. Generally speaking, the time-domain scalable video coding device can be a video coding device such as a computer, a mobile phone, a tablet, a video coding server, etc.

[0032] The following description is made by taking the time domain scalable video coding device as an example of the main body for executing the time domain scalable video coding method. Figure 1 , the time-domain scalable video coding method specifically includes:

[0033] S110, determining a hierarchical structure of the picture group to be encoded, wherein the hierarchical structure is arranged in a plurality of levels from high to low according to a set hierarchical rule, wherein each level contains a corresponding number of image frames, and image frames of high levels use image frames of low levels as reference frames, and image frames of the lowest level use the first frame of the picture group to be encoded as a reference frame.

[0034] In the process of video encoding, the present application implements the time domain SVC function in the H.264 encoder core based on the reference frame management method of the H.264 standard to adjust the video encoding frame rate. For a group of pictures (GOP), the hierarchical structure is constructed to perform the time domain SVC function through the hierarchical structure.

[0035] Depending on the number of video frames contained in the picture group, it can be expanded into any GOP hierarchical structure, such as GOP4, GOP8, GOP16, GOP32, etc. Figure 2 , taking the time domain SVC GOP16 structure as an example, it is divided into 4 layers, layer0: I0; layer1: P4, P8, P12; layer2: P2, P6, P10, P14; layer3: P1, P3, P5, P7, P9, P11, P13, P15. In the hierarchical structure, high-level frames can only refer to frames in lower layers. In addition, since real-time audio and video applications require the network to have stronger anti-frame loss capabilities, only one reference frame is used for video encoding. Based on the above hierarchical structure, the reference frame relationship of each image frame can be clarified. For example, the P4 reference frame is I0, and I0 is the first frame of the picture group to be encoded. The first frame may be a key frame or a P frame.

[0036] Optionally, for reference relationships of other GOP structures, such as GOP4 structure, there are three layers, layer0: I0; layer1: P2; layer2: P1, P3, and reference frame relationships are set correspondingly for image frames of each layer. This application does not impose fixed restrictions on the hierarchical structure of a specific picture group, i.e., the reference frame relationship, and will not be elaborated here.

[0037] For each group of pictures (GOP) to be encoded, its hierarchical structure is first determined so as to subsequently perform a time domain SVC function according to the hierarchical structure.

[0038] S120: When time-domain scalable video coding is triggered, image frames of at least one highest level in the group of pictures to be encoded are filtered out, and remaining image frames in the group of pictures to be encoded are encoded.

[0039] Furthermore, based on the hierarchical structure of the picture group to be encoded determined as above, when time-domain scalable video coding (that is, time-domain SVC) is triggered, the image frames of at least one of the highest levels in the picture group to be encoded can be deleted to reduce the number of image frames to be encoded, thereby achieving the effect of reducing the encoding frame rate.

[0040] It can be understood that the time domain SVC of video coding refers to the effect of allowing different frame rates to be transmitted during the encoding of a video stream through layered GOP structure coding. Time domain SVC requires that high-level frames can only refer to lower-level image frames, thereby allowing the transmission process to discard high-level frames and only transmit low-level frames, thereby achieving the effect of different frame rates. For example, Figure 2 In the hierarchical structure shown, after the highest level frames P1, P3, P5, P7, etc. are discarded, the frame rate of the picture group is halved, and because these image frames are not used as reference frames, they will not affect the decoding of the code stream. In this way, by reducing the video frame rate, the effect of transmitting video streams with different frame rates can be achieved. According to actual needs, when the video frame rate needs to be reduced more, the second highest level image frames can be further screened out to further reduce the encoding frame rate.

[0041] Based on this, according to the actual coding requirements, when the video coding frame rate needs to be reduced, the time domain scalable video coding function can be triggered. The time domain scalable video coding function can be triggered automatically based on manual judgment or when certain conditions are met.

[0042] Optionally, before filtering out the image frames of at least one highest level in the group of pictures to be encoded, the method further includes:

[0043] When it is detected that the current data transmission network meets the set network condition, the time domain scalable video coding of the current group of images to be encoded is triggered.

[0044] It is understandable that when the data transmission network is poor and meets the set network conditions (such as packet loss rate and jitter reaching the threshold), the time domain scalable video encoding function can be triggered to reduce the video encoding frame rate, thereby reducing the video bit rate and optimizing audio and video calls.

[0045] Optionally, encoding the remaining image frames in the group of pictures to be encoded includes:

[0046] The encoding bit rate of the layer to which each image frame belongs is preconfigured, and the remaining image frames in the to-be-encoded picture group are encoded based on the corresponding encoding bit rate.

[0047] It is understandable that since high-level frames can only refer to low-level frames, the low-level frames have a greater impact on the overall image quality, so these image frames should be encoded with higher quality. Based on this feature, the present application sets different bitrates [layer I d] for each layer of audio frames, the lower the bitrate of the lower-level frames, the lower the bitrate of the higher-level frames, thereby reusing limited bandwidth and making the encoding quality better.

[0048] Specifically, the encoding bit rate of the layer to which each image frame belongs is pre-configured, including:

[0049] The quantization parameters of the image frames at each level are adjusted to configure the encoding bit rates of the image frames at each level based on the adjusted quantization parameters.

[0050] This application adjusts the quantization parameters of image frames at each level according to the GOP hierarchical structure to increase the bit rate allocation weight of the underlying frame, thereby achieving the effect of configuring the encoding bit rate at each level. The quantization parameter (QP) is the serial number of the quantization step size Qstep, which reflects the compression of spatial details. The smaller the QP value, the smaller the quantization step size, and the higher the quantization accuracy, which means that the amount of data generated may be larger under the same image quality. QP and bit rate are inversely proportional. As QP increases, some details are lost and the bit rate is reduced, but the image distortion is enhanced and the quality is reduced; as QP decreases, more details are retained, the bit rate increases, and the image quality improves.

[0051] By adjusting the QP, the output target bit rate can be adjusted while maintaining a certain image quality, thereby achieving a balance between compression efficiency and quality. In scenarios where bandwidth is limited or image quality requirements are not high, the QP value can be appropriately increased to reduce the encoding bit rate and reduce the amount of data. Therefore, the present application adjusts the quantization parameters of image frames at each level to configure the encoding bit rate of image frames at each level based on the adjusted quantization parameters.

[0052] In addition, according to actual needs, the image size can also be adjusted. The smaller the image size, the lower the frame rate, and the encoding bit rate will usually be lower. On the other hand, different encoding algorithms and encoder settings can also have a significant impact on the encoding bit rate. By adjusting the quantization parameters, this application can affect the encoding bit rate to a certain extent to achieve the best compression effect and quality performance, which can improve the video encoding quality.

[0053] Based on the above-mentioned screened-out picture group of at least one highest level image frame, combined with the above-configured encoding bit rate, encoding can be performed frame by frame to improve encoding efficiency and encoding quality in a real-time network environment.

[0054] S130 , storing the encoded image frame in an image cache, and selecting a reference frame of the image frame to be encoded from the image cache according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

[0055] Furthermore, during the video encoding process, an image cache is constructed to store the encoded image frames of the group of pictures to be encoded. Figure 3 After the current frame is encoded, the encoded image frame is stored in the image cache, and the image cache selects the reference frame of the subsequent image frame to be encoded through the encoded image frame. For example, taking the GOP16 structure as an example, after the encoding of the first frame I0 of the group of pictures to be encoded is completed, I0 enters the image cache, and then I0 is used as the reference frame of the subsequent frame for encoding the subsequent image frame.

[0056] Image cache Decode Picture Buffer (DPB), that is, decoded picture buffer. The encoded image will be cached in the image cache as a reference image for subsequent frames. The maximum supported DPB size (image cache space) of the H.264 standard is 16 frames, and it supports the reference frame reordering function. Therefore, the present application utilizes the characteristics of the above-mentioned image cache to manage the image cache. Among them, since the top-level frame layer3 of GOP 16 is not used as a reference frame, that is, it is not referenced by any subsequent frames, it does not need to be retained in the image cache. Therefore, the present application uses a DPB size of 8 frames to meet the required image cache requirements. After encoding, the frames of Layer0, layer1, and layer2 are retained in the image cache for reference by subsequent frames.

[0057] Optionally, after storing the encoded image frame in the image buffer, the method further includes:

[0058] The encoded image frames in the image buffer are managed based on a sliding window of a fixed length, and the encoded image frames exceeding the sliding window are removed from the image buffer.

[0059] The update process of the image cache within a GOP is as follows: I0, P2, P4, P6, P8, P10, P12, P14. The subsequent encoding process uses a sliding window to manage the image cache. When the image cache is full, the earliest encoded frame will be removed from the image cache.

[0060] The sliding window is based on the number of frames that can be stored in the image cache. For the currently decoded image, the new decoded image is moved in in a first-in-first-out manner, and the decoded image that exceeds the window is moved out. Therefore, the image cache stores multiple recently decoded images. In addition, according to the hierarchical structure, the encoded images in the cache can be divided into reference images (such as I0, P4, P8 and P12) and non-reference images (such as P2, P6, P10, P14, which are the highest-level image frames among the remaining image frames and are not used as reference frames. In addition, P16 directly refers to I0 and can also be set as a non-reference image). Further, the reference image can be set as a short-term reference frame (such as P4, P8 and P12) and a long-term reference frame (such as I0) according to actual needs. It can be understood that I0 needs to be used as a reference frame for P1, P2, P4, P8, P12 and P16, so it needs to be managed as a long-term reference frame.

[0061] It should be noted that the sliding window is mainly used to remove expired short-term reference frames from the image cache and does not involve operations on long-term reference frames.

[0062] Optionally, after storing the encoded image frame in the image buffer, the method further includes:

[0063] The reference frame status of the encoded images in the image cache is marked according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure, so as to selectively remove the corresponding encoded images according to the reference frame status.

[0064] This application is based on MMCO (Adaptive Memory Control) to manage the image cache. MMCO transmits control commands in the code stream through the encoder to notify the status marking of the image in the image cache. It supports marking an encoded image from the reference state "Used for reference" to the non-reference state "Unused for reference", and can also mark the short-term reference state "Used for short-term referenceene" of the current frame or short-term reference frame as the long-term reference state "Used for long-term reference". MMCO instructions are transmitted through the syntax element memory_management_control_operation.

[0065] Since the memory overhead of the image cache memory for 8 frames is relatively large (especially for ultra-high-definition images), for scenes with strict memory constraints but good network, the MMCO technology of H.264 can be used to optimize the image cache and reduce memory overhead. The H.264 standard allows the current frame to carry MMCO information to mark the reference frame status ("Used for short-term reference", "Used for long-term reference", "Unused for reference"). When the next frame is encoded, the image cache status can be updated according to the MMCO information. For the GOP16 layered structure, MMCO can be used to mark the frames that are no longer used as reference frames as non-reference states "Unused for reference". In the next frame, the frames that are no longer used as reference frames will be removed from the image cache, thereby reducing the memory overhead of the image cache.

[0066] Optionally, refer to Figure 4 , during the encoding process, a reference frame of the image frame to be encoded is selected from the image cache according to the hierarchical structure, including:

[0067] S1301, constructing a reference frame list based on the image cache, and sorting the reference frame list according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure;

[0068] S1302: Select an encoded image at a specified position from the reference frame list as a reference frame for the current image frame to be encoded.

[0069] A reference frame list is constructed from the image cache for reference by the current frame. The reference frame list includes the process of reference frame sorting and reference frame rearrangement. Among them, the reference frames in the short-term reference state are sorted in descending order, that is, the reference frames closer to the current frame are placed at the front. The purpose of reference frame rearrangement is to put the reference frame that the current frame mainly refers to at the front of the reference list to save the bit rate overhead of the reference frame list. By moving the specified reference frame order to the front of the reference frame list, it is convenient to select the encoded image from the specified position as the reference frame of the current image frame to be encoded.

[0070] Exemplarily, the target reference frame is moved to the first position of the reference frame list by reordering the reference frame, thereby ensuring that the reference frame list construction process is correct. Since there is only one reference frame, each time the header element is taken from the reference frame list, it is the reference frame of the current frame. For example, when encoding the P8 frame, the image cache state is [I0, P2, P4, P6], and the list of reference frames sorted in descending order at this time is ref_sort_list[P6, P4, P2, I0]. Since the real-time audio and video call application is limited to using only one reference frame, if the reference frame is constructed directly using the list, the nearest P6 will be selected as the only reference frame, and the constructed reference frame list is ref_lsit[P6], which does not match the order of P8 reference I0 in the GOP hierarchical structure. By reordering the reference frame, the order of ref_sort_list can be adjusted to [I0, P6, P4, P2], so that I0 is taken as the reference frame.

[0071] In the above, by determining the hierarchical structure of the picture group to be encoded, the hierarchical structure is set up with multiple levels from high to low according to the set hierarchical rules, each level contains a corresponding number of image frames, the image frames of the high level use the image frames of the low level as reference frames, and the image frames of the lowest level use the first frame of the picture group to be encoded as reference frames; in the case of triggering time domain scalable video coding, the image frames of at least one of the highest levels in the picture group to be encoded are filtered out, and the remaining image frames in the picture group to be encoded are encoded; the encoded image frames are stored in the image cache, and the reference frames of the image frames to be encoded are selected from the image cache according to the hierarchical structure, so as to encode the image frames to be encoded based on the selected reference frames. By adopting the above technical means, by filtering out the image frames of at least one of the highest levels of the picture group to be encoded according to the hierarchical structure, and encoding the image frames to be encoded and caching the encoded image frames according to the reference frame relationship identified by the hierarchical structure, the frame rate adjustment of video encoding can be realized in the scalable video coding process, and the video transmission bit rate can be saved by reducing the video frame rate, thereby adapting to the video encoding requirements in a specific network environment and improving the video processing efficiency and quality. In addition, by reducing the number of image frames to be sent, bandwidth can be saved, the CPU overhead of decoding on the other end can be saved, and the video encoding efficiency can be further improved.

[0072] Based on the above embodiments, Figure 5 A schematic diagram of the structure of a time-domain scalable video coding system provided by this application. Figure 5 The time-domain scalable video coding system provided in this embodiment specifically includes: a layering module 21, a screening module 22 and a coding module 23.

[0073] The hierarchical module is configured to determine a hierarchical structure of the group of pictures to be encoded, wherein the hierarchical structure is arranged in multiple levels from high to low according to a set hierarchical rule, each level includes a corresponding number of image frames, and the image frames of the high level use the image frames of the low level as reference frames, and the image frames of the lowest level use the first frame of the group of pictures to be encoded as a reference frame;

[0074] A screening module is configured to screen out image frames of at least one highest level in the group of pictures to be encoded and encode the remaining image frames in the group of pictures to be encoded when the time-domain scalable video coding is triggered;

[0075] The encoding module is configured to store the encoded image frame in the image cache, and select a reference frame of the image frame to be encoded from the image cache according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

[0076] Specifically, before filtering out the image frames of at least one highest level in the group of pictures to be encoded, the method further includes:

[0077] When it is detected that the current data transmission network meets the set network condition, the time domain scalable video coding of the current group of images to be encoded is triggered.

[0078] Specifically, encoding the remaining image frames in the group of pictures to be encoded includes:

[0079] The encoding bit rate of the layer to which each image frame belongs is preconfigured, and the remaining image frames in the to-be-encoded picture group are encoded based on the corresponding encoding bit rate.

[0080] Specifically, the encoding bit rate of the layer to which each image frame belongs is pre-configured, including:

[0081] The quantization parameters of the image frames at each level are adjusted to configure the encoding bit rates of the image frames at each level based on the adjusted quantization parameters.

[0082] Specifically, after storing the encoded image frame in the image buffer, the method further includes:

[0083] The encoded image frames in the image buffer are managed based on a sliding window of a fixed length, and the encoded image frames exceeding the sliding window are removed from the image buffer.

[0084] Specifically, after storing the encoded image frame in the image buffer, the method further includes:

[0085] The reference frame status of the encoded images in the image cache is marked according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure, so as to selectively remove the corresponding encoded images according to the reference frame status.

[0086] Specifically, selecting a reference frame of a to-be-encoded image frame from an image cache according to a hierarchical structure includes:

[0087] Building a reference frame list based on the image cache, and sorting the reference frame list according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure;

[0088] Select an encoded image at a specified position from the reference frame list as a reference frame for the current image frame to be encoded.

[0089] In the above, by determining the hierarchical structure of the picture group to be encoded, the hierarchical structure is set up with multiple levels from high to low according to the set hierarchical rules, each level contains a corresponding number of image frames, the image frames of the high level use the image frames of the low level as reference frames, and the image frames of the lowest level use the first frame of the picture group to be encoded as reference frames; in the case of triggering time domain scalable video coding, the image frames of at least one of the highest levels in the picture group to be encoded are filtered out, and the remaining image frames in the picture group to be encoded are encoded; the encoded image frames are stored in the image cache, and the reference frames of the image frames to be encoded are selected from the image cache according to the hierarchical structure, so as to encode the image frames to be encoded based on the selected reference frames. By adopting the above technical means, by filtering out the image frames of at least one of the highest levels of the picture group to be encoded according to the hierarchical structure, and encoding the image frames to be encoded and caching the encoded image frames according to the reference frame relationship identified by the hierarchical structure, the frame rate adjustment of video encoding can be realized in the scalable video coding process, and the video transmission bit rate can be saved by reducing the video frame rate, thereby adapting to the video encoding requirements in a specific network environment and improving the video processing efficiency and quality. In addition, by reducing the number of image frames to be sent, bandwidth can be saved, the CPU overhead of decoding on the other end can be saved, and the video encoding efficiency can be further improved.

[0090] The time-domain scalable video coding system provided in the embodiment of the present application can be configured to execute the time-domain scalable video coding method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0091] Based on the above practical example, the present application embodiment also provides a time domain scalable video encoding device, referring to Figure 6, the time-domain scalable video coding device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the time-domain scalable video coding method described in any embodiment of the present application (for example, a layering module, a screening module and a coding module in a time-domain scalable video coding system). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizing the above-mentioned time-domain scalable video coding method. The input device can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device may include a display device such as a display screen. The above-mentioned time-domain scalable video coding device can be configured to execute the time-domain scalable video coding method provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0092] On the basis of the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute a time-domain scalable video coding method when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the computer-executable instructions of the computer-readable storage medium provided in the embodiments of the present application are not limited to the time-domain scalable video coding method described above, and may also execute related operations in the time-domain scalable video coding method provided in any embodiment of the present application.

[0093] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the time-domain scalable video coding method described in each embodiment of the present application.

Claims

1. A time-domain scalable video coding method, characterized in that: include: Determine a hierarchical structure of a group of pictures to be encoded, wherein the hierarchical structure is arranged in a plurality of levels from high to low according to a set hierarchical rule, each level includes a corresponding number of image frames, an image frame of a high level uses an image frame of a low level as a reference frame, and an image frame of a lowest level uses a first frame of the group of pictures to be encoded as a reference frame; In the case of triggering time-domain scalable video coding, filtering out image frames of at least one highest level in the group of pictures to be encoded, and encoding the remaining image frames in the group of pictures to be encoded; The encoded image frames are stored in an image buffer, and a reference frame of the image frame to be encoded is selected from the image buffer according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

2. The temporal scalable video coding method according to claim 1, characterized in that: After storing the encoded image frame in the image buffer, the method further includes: The encoded image frames in the image buffer are managed based on a sliding window of fixed length, and the encoded image frames exceeding the sliding window are moved out of the image buffer.

3. The time-domain scalable video coding method according to claim 1, characterized in that: The encoding of the remaining image frames in the group of pictures to be encoded comprises: The encoding bit rate of the layer to which each image frame belongs is preconfigured, and the remaining image frames in the to-be-encoded picture group are encoded based on the corresponding encoding bit rate.

4. The temporal scalable video coding method according to claim 3, characterized in that: The pre-configuration of the encoding bit rate of the layer to which each image frame belongs includes: The quantization parameters of the image frames at each level are adjusted to configure the encoding bit rates of the image frames at each level based on the adjusted quantization parameters.

5. The temporal scalable video coding method according to any one of claims 1 to 4, characterized in that: After storing the encoded image frame in the image buffer, the method further includes: The reference frame status of the encoded images in the image cache is marked according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure, so as to select and remove the corresponding encoded images according to the reference frame status.

6. The temporal scalable video coding method according to any one of claims 1 to 4, characterized in that: The step of selecting a reference frame of a to-be-encoded image frame from the image cache according to the hierarchical structure comprises: constructing a reference frame list based on the image cache, and sorting the reference frame list according to the reference frame relationship of the current image frame to be encoded identified by the hierarchical structure; An encoded image at a specified position is selected from the reference frame list as a reference frame for the current image frame to be encoded.

7. The temporal scalable video coding method according to any one of claims 1 to 4, characterized in that: Before filtering out the image frames of at least one highest level in the group of pictures to be encoded, the method further includes: When it is detected that the current data transmission network meets the set network condition, the time domain scalable video coding of the current group of images to be encoded is triggered.

8. A temporal scalable video coding system, characterized in that: include: a hierarchical module configured to determine a hierarchical structure of a group of pictures to be encoded, wherein the hierarchical structure is arranged in a plurality of levels from high to low according to a set hierarchical rule, each level includes a corresponding number of image frames, an image frame of a high level uses an image frame of a low level as a reference frame, and an image frame of a lowest level uses a first frame of the group of pictures to be encoded as a reference frame; a screening module configured to screen out image frames of at least one highest level in the group of pictures to be encoded and encode the remaining image frames in the group of pictures to be encoded when time-domain scalable video coding is triggered; The encoding module is configured to store the encoded image frame in an image cache, and select a reference frame of the image frame to be encoded from the image cache according to the hierarchical structure, so as to encode the image frame to be encoded based on the selected reference frame.

9. A time-domain scalable video coding device, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the time-domain scalable video coding method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer processor, they are configured to execute the temporal scalable video coding method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the time-domain scalable video coding method according to any one of claims 1 to 7.

Images