Image processing method and apparatus, and device

Abstract

The present application provides an image processing method and apparatus, and a device. The method comprises: dividing all P frames in a GOP sequence into at least two types of P frames, the at least two types of P frames comprising P frames that cannot be used as reference frames and P frames that can be used as reference frames; and if it is determined that the P frames in the GOP sequence need to be subjected to step printing, performing step printing on the P frames that cannot be used as reference frames, or if it is determined that the P frames in the GOP sequence need to be subjected to step printing, performing step printing on the P frames that cannot be used as reference frames and the P frames that can be used as reference frames. By means of the technical solution of the present application, frame dropping can be performed according to a certain rule, such that a video frame sequence can be ensured to be played in order, and some P frames can also be dropped in some special scenarios to save bandwidth.

Description

Image processing method, device and equipment Technical Field

[0001] The present application relates to the field of video coding and decoding, and in particular to an image processing method, apparatus and device. Background Art

[0002] To save space, video images are encoded before transmission. A complete video encoding method can include prediction, transformation, quantization, entropy coding, filtering, and other processes. Predictive coding can include intra-frame coding and inter-frame coding. Inter-frame coding exploits temporal correlation in the video, using pixels from adjacent encoded images to predict the pixels of the current image, thereby removing temporal redundancy in the video. Intra-frame coding takes into account the strong spatial correlation between adjacent blocks and uses surrounding reconstructed pixels as reference pixels to predict the current uncoded block. Only the residual values ​​need to be encoded, rather than the original values. This effectively removes spatial redundancy and greatly improves compression efficiency.

[0003] During the video encoding process, all images can be divided into I-frames and P-frames. I-frames are images that use intra-frame coding, meaning they can be independently encoded without reference to other images. P-frames are images that use inter-frame coding, meaning they require reference to other images to complete encoding, such as relying on the previous image to complete encoding. For example, consecutive video frames include frames 1-10, where frame 1 is an I-frame, frames 2-5 are P-frames, frame 6 is an I-frame, and frames 7-10 are P-frames. In this case, frame 1 is intra-frame coded, frame 2 is inter-coded with reference to frame 1, frame 3 is inter-coded with reference to frame 2, frame 4 is inter-coded with reference to frame 3, frame 5 is inter-coded with reference to frame 4, frame 6 is intra-frame coded, frame 7 is inter-coded with reference to frame 6, and so on.

[0004] Obviously, in this method, all P frames cannot be discarded. Once a P frame is discarded, subsequent P frames cannot be processed correctly. For example, if frame 3 is discarded, frame 4 cannot reference frame 3 to complete inter-frame coding, and frame 5 cannot reference frame 4 to complete inter-frame coding.

[0005] Summary of the Invention

[0006] The present application provides an image processing method, which includes: dividing all P frames in a group of pictures (GOP) sequence into at least two types of P frames, wherein there are P frames that cannot be used as reference frames among the at least two types of P frames, and there are P frames that can be used as reference frames among the at least two types of P frames; if it is determined that the P frames in the GOP sequence need to be extracted, the P frames that cannot be used as reference frames are extracted, or if it is determined that the P frames in the GOP sequence need to be extracted, the P frames that cannot be used as reference frames and the P frames that can be used as reference frames are extracted.

[0007] In one possible embodiment, the method includes: dividing all P frames in the GOP sequence into a first category of P frames and a second category of P frames; wherein, the first category of P frames cannot be used as reference frames, the second category of P frames can be used as reference frames for the first category of P frames, and the second category of P frames can be used as reference frames for the second category of P frames; if it is determined that the P frames in the GOP sequence need to be extracted, then the first category of P frames are extracted, or, the first category of P frames and the second category of P frames are extracted.

[0008] In a possible implementation, dividing all P frames in the GOP sequence into first-category P frames and second-category P frames includes: dividing all P frames in the GOP sequence into first-category P frames and second-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between the I frame of the GOP sequence and the first second-category P frame; wherein M1 is a positive integer, and M2 is a positive integer.

[0009] For each first-category P frame, if there is a second-category P frame preceding the first-category P frame, the second-category P frame closest to the first-category P frame is used as a reference frame for the first-category P frame; if there is no second-category P frame preceding the first-category P frame, the I frame of the GOP sequence is used as a reference frame for the first-category P frame;

[0010] For each second-class P frame, if there is a second-class P frame before the second-class P frame, the second-class P frame most recently before the second-class P frame is used as the reference frame of the second-class P frame; if there is no second-class P frame before the second-class P frame, the I frame of the GOP sequence is used as the reference frame of the second-class P frame.

[0011] Exemplarily, if it is determined that the P frames in the GOP sequence need to be decimated, then the first type of P frames are decimated, or the first type of P frames and the second type of P frames are decimated, including: if it is determined based on service information that the P frames in the GOP sequence need to be decimated using a first sampling ratio, then all or part of the first type of P frames are decimated; if it is determined based on service information that the P frames in the GOP sequence need to be decimated using a second sampling ratio, then all the first type of P frames are decimated and all or part of the second type of P frames are decimated; or,

[0012] If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a first sampling ratio, some of the first-category P frames are decimated; if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a second sampling ratio, all of the first-category P frames are decimated; if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a third sampling ratio, all of the first-category P frames are decimated, and all or some of the second-category P frames are decimated;

[0013] Among them, the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

[0014] In a possible implementation, the method further includes: if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determining whether it is necessary to perform frame extraction on the P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images at the resolution and the frame rate, determining to perform frame extraction on the P frames in the GOP sequence; and if so, determining a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability; or,

[0015] If the service information includes network quality corresponding to a target network between a transmitting end and a receiving end, determining whether to perform frame extraction on the P frames in the GOP sequence based on the network quality; wherein, if the network quality is not an optimal level of network quality, determining to perform frame extraction on the P frames in the GOP sequence; and if so, determining a sampling ratio corresponding to the GOP sequence based on the network quality;

[0016] Wherein, the sampling ratio is the first sampling ratio or the second sampling ratio; or,

[0017] The sampling ratio is the first sampling ratio, the second sampling ratio, or the third sampling ratio.

[0018] In one possible implementation, the method includes: dividing all P frames in the GOP sequence into a first category of P frames, a second category of P frames, and a third category of P frames; wherein the first category of P frames cannot be used as reference frames, the second category of P frames can only be used as reference frames for the first category of P frames, and the third category of P frames can be used as reference frames for the first category of P frames, and can be used as reference frames for the second category of P frames, and can be used as reference frames for the third category of P frames; if it is determined that the P frames in the GOP sequence need to be extracted, then the first category of P frames are extracted, or the first category of P frames and the second category of P frames are extracted, or the first category of P frames, the second category of P frames, and the third category of P frames are extracted.

[0019] In a possible implementation, the dividing all P frames in the GOP sequence into first-category P frames, second-category P frames, and third-category P frames includes: dividing all P frames in the GOP sequence into first-category P frames, second-category P frames, and third-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, there are two first-category P frames and one second-category P frame between two adjacent third-category P frames, and the one second-category P frame is located between the two first-category P frames; there are two first-category P frames and one second-category P frame between the I frame of the GOP sequence and the first third-category P frame, and the second-category P frame is located between the two first-category P frames.

[0020] Exemplarily, for each first-category P frame, if the previous frame of the first-category P frame is not a first-category P frame, the previous frame of the first-category P frame is used as a reference frame for the first-category P frame; if the previous frame of the first-category P frame is a first-category P frame, the most recent non-first-category P frame is used as a reference frame for the first-category P frame; for each second-category P frame, if there is a third-category P frame before the second-category P frame, the most recent third-category P frame before the second-category P frame is used as a reference frame for the second-category P frame; if there is no third-category P frame before the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame; for each third-category P frame, if there is a third-category P frame before the third-category P frame, the most recent third-category P frame before the third-category P frame is used as a reference frame for the third-category P frame; if there is no third-category P frame before the third-category P frame, the I frame of the GOP sequence is used as a reference frame for the third-category P frame.

[0021] Exemplarily, if it is determined that P frames in the GOP sequence need to be extracted, then extracting the first category of P frames, or extracting the first category of P frames and the second category of P frames, or extracting the first category of P frames, the second category of P frames, and the third category of P frames includes:

[0022] If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the first sampling ratio, all or part of the first-category P frames are decimated;

[0023] If it is determined based on the service information that a second sampling ratio is required to be used to decimate the P frames in the GOP sequence, decimating all first-category P frames and decimating all or part of the second-category P frames; wherein the second sampling ratio is greater than the first sampling ratio;

[0024] If it is determined based on business information that a third sampling ratio is required to sample the P frames in the GOP sequence, all first-category P frames are sampled, all second-category P frames are sampled, and all third-category P frames or some third-category P frames are sampled; the third sampling ratio is greater than the second sampling ratio.

[0025] The method further includes: if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determining whether it is necessary to extract P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images with the resolution and the frame rate, determining to extract P frames in the GOP sequence; if so, determining a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, if the service information includes a network quality corresponding to a target network between a transmitting end and a receiving end, determining whether it is necessary to extract P frames in the GOP sequence based on the network quality; wherein, if the network quality is not an optimal level of network quality, determining to extract P frames in the GOP sequence; and if so, determining a sampling ratio corresponding to the GOP sequence based on the network quality, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0026] Exemplarily, the method further includes: when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the I frame of the first GOP sequence is used as a reference frame for the first frame of the GOP sequence.

[0027] The present application provides an image processing device, which includes: a division module, used to divide all P frames in a GOP sequence into at least two types of P frames, wherein there are P frames that cannot be used as reference frames among the at least two types of P frames, and there are P frames that can be used as reference frames among the at least two types of P frames; a processing module, used to, if it is determined that the P frames in the GOP sequence need to be extracted, extract the P frames that cannot be used as reference frames, or, if it is determined that the P frames in the GOP sequence need to be extracted, extract the P frames that cannot be used as reference frames and the P frames that can be used as reference frames.

[0028] In a possible implementation, the division module is specifically used to divide all P frames in the GOP sequence into a first category of P frames and a second category of P frames; wherein, the first category of P frames cannot be used as reference frames, the second category of P frames can be used as reference frames for the first category of P frames, and the second category of P frames can be used as reference frames for the second category of P frames; the processing module is specifically used to extract the first category of P frames if it is determined that the P frames in the GOP sequence need to be extracted, or to extract the first category of P frames and the second category of P frames.

[0029] Among them, when the division module divides all P frames in the GOP sequence into first-category P frames and second-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames and second-category P frames based on the position sorting relationship between the P frames in the GOP sequence; among them, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between the I frame of the GOP sequence and the first second-category P frame; M1 and M2 are both positive integers.

[0030] Among them, for each first-category P frame, if there is a second-category P frame before the first-category P frame, the second-category P frame closest before the first-category P frame is used as the reference frame of the first-category P frame; if there is no second-category P frame before the first-category P frame, the I frame of the GOP sequence is used as the reference frame of the first-category P frame; for each second-category P frame, if there is a second-category P frame before the second-category P frame, the second-category P frame closest before the second-category P frame is used as the reference frame of the second-category P frame; if there is no second-category P frame before the second-category P frame, the I frame of the GOP sequence is used as the reference frame of the second-category P frame.

[0031] Wherein, when the processing module determines that the P frames in the GOP sequence need to be decimated, the first type of P frames are decimated, or when decimating the first type of P frames and the second type of P frames, it is specifically used to: if it is determined based on business information that the P frames in the GOP sequence need to be decimated using a first sampling ratio, all the first type of P frames or part of the first type of P frames are decimated; if it is determined based on business information that the P frames in the GOP sequence need to be decimated using a second sampling ratio, all the first type of P frames are decimated, and all the second type of P frames or part of the second type of P frames are decimated; or If it is determined based on business information that the first sampling ratio needs to be used to sample the P frames in the GOP sequence, some of the first-category P frames are sampled; if it is determined based on business information that the second sampling ratio needs to be used to sample the P frames in the GOP sequence, all of the first-category P frames are sampled; if it is determined based on business information that the third sampling ratio needs to be used to sample the P frames in the GOP sequence, all of the first-category P frames are sampled, and all or some of the second-category P frames are sampled; wherein the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

[0032] The processing module is further configured to, if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determine whether it is necessary to extract P frames within the GOP sequence based on the resolution, the frame rate, and the decoding capability; if the decoding capability does not support decoding images with the resolution and the frame rate, determine to extract P frames within the GOP sequence; if so, determine a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability; or, if the service information includes network quality corresponding to a target network between a transmitting end and a receiving end, determine whether it is necessary to extract P frames within the GOP sequence based on the network quality; if the network quality is not optimal, determine to extract P frames within the GOP sequence; if so, determine a sampling ratio corresponding to the GOP sequence based on the network quality; the sampling ratio is a first sampling ratio or a second sampling ratio; or, the sampling ratio is the first sampling ratio, the second sampling ratio, or the third sampling ratio.

[0033] Among them, the division module is specifically used to divide all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames; the first-category P frames cannot be used as reference frames, the second-category P frames can only be used as reference frames for the first-category P frames, and the third-category P frames can be used as reference frames for the first-category P frames, can be used as reference frames for the second-category P frames, and can be used as reference frames for the third-category P frames; the processing module is specifically used to extract the first-category P frames, or extract the first-category P frames and the second-category P frames, or extract the first-category P frames, the second-category P frames and the third-category P frames if it is determined that the P frames in the GOP sequence need to be extracted.

[0034] Among them, when the division module divides all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, for all P frames in the GOP sequence, there are two first-category P frames and one second-category P frame between two adjacent third-category P frames, and the one second-category P frame is located between the two first-category P frames; there are two first-category P frames and one second-category P frame between the I frame of the GOP sequence and the first third-category P frame, and this second-category P frame is located between the two first-category P frames.

[0035] Among them, for each first-category P frame, if the previous frame of the first-category P frame is not a first-category P frame, the previous frame of the first-category P frame is used as a reference frame for the first-category P frame; if the previous frame of the first-category P frame is a first-category P frame, the most recent non-first-category P frame is used as a reference frame for the first-category P frame; for each second-category P frame, if there is a third-category P frame before the second-category P frame, the most recent third-category P frame before the second-category P frame is used as a reference frame for the second-category P frame; if there is no third-category P frame before the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame; for each third-category P frame, if there is a third-category P frame before the third-category P frame, the most recent third-category P frame before the third-category P frame is used as a reference frame for the third-category P frame; if there is no third-category P frame before the third-category P frame, the I frame of the GOP sequence is used as a reference frame for the third-category P frame.

[0036] Wherein, when the processing module determines that the P frames in the GOP sequence need to be decimalized, the first type of P frames are decimalized, or the first type of P frames and the second type of P frames are decimalized, or the first type of P frames, the second type of P frames and the third type of P frames are decimalized. Specifically, if it is determined based on service information that the P frames in the GOP sequence need to be decimalized using a first sampling ratio, all or part of the first type of P frames are decimalized; if it is determined based on service information that the P frames in the GOP sequence need to be decimalized using a second sampling ratio, all the first type of P frames are decimalized, and all or part of the second type of P frames are decimalized; the second sampling ratio is greater than the first sampling ratio; if it is determined based on service information that the P frames in the GOP sequence need to be decimalized using a third sampling ratio, all the first type of P frames are decimalized, all the second type of P frames are decimalized, and all or part of the third type of P frames are decimalized; the third sampling ratio is greater than the second sampling ratio.

[0037] The processing module is further configured to, if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determine whether P frames in the GOP sequence need to be decimated based on the resolution, the frame rate, and the decoding capability; if the decoding capability does not support decoding images with the resolution and the frame rate, determine whether P frames in the GOP sequence need to be decimated; if so, determine a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, where the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, if the service information includes network quality corresponding to a target network between an encoding end and a decoding end, determine whether P frames in the GOP sequence need to be decimated based on the network quality; if the network quality is not optimal, determine whether P frames in the GOP sequence need to be decimated; if so, determine a sampling ratio corresponding to the GOP sequence based on the network quality, where the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0038] When the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the division module is further used to use the I frame of the first GOP sequence as a reference frame for the first frame of the GOP sequence.

[0039] The present application provides an image processing device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method disclosed in the above embodiment of the present application.

[0040] It can be seen from the above technical solution that in the embodiment of the present application, all P frames in the GOP (Group of Pictures) sequence can be divided into P frames that cannot be used as reference frames and P frames that can be used as reference frames. On this basis, the P frames that cannot be used as reference frames can be extracted, or the P frames that cannot be used as reference frames and the P frames that can be used as reference frames can be extracted. The above method provides a new type of video frame sequence, which includes different types of P frames. In the video frame sequence, frames can be dropped according to a certain rule, which can ensure that the video frame sequence is played in order and has the characteristics of intermittently dropping P frames according to the rules without affecting the viewing experience. In some special scenarios, some P frames can be dropped to save bandwidth. The video frame sequence has better usage prospects in decoding, network transmission, and storage, and realizes the characteristics of decoding multiplication, low-bandwidth transmission, and low-bitrate storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] 1A and 1B are schematic diagrams of a GOP sequence;

[0042] FIG2 is a schematic flow chart of an image processing method in one embodiment of the present application;

[0043] 3A and 3B are schematic diagrams of a GOP sequence;

[0044] 4A-4C are schematic diagrams of extracting P frames within a GOP sequence;

[0045] 5A to 5E are schematic diagrams of displaying images through multiple channels;

[0046] FIG6 is a schematic diagram of a set of image sequences;

[0047] FIG7 is a flow chart of an image processing method in one embodiment of the present application;

[0048] 8A and 8B are schematic diagrams of a GOP sequence;

[0049] 9A-9C are schematic diagrams of extracting P frames within a GOP sequence;

[0050] FIG10 is a schematic diagram of an image sequence set;

[0051] FIG11 is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0052] FIG12 is a hardware structure diagram of an image processing device in one embodiment of the present application. DETAILED DESCRIPTION

[0053] During video encoding and decoding, all images can be divided into I-frames and P-frames. I-frames are images that use intra-frame coding, that is, they can be encoded independently without referring to other images. P-frames are images that use inter-frame coding, that is, they need to refer to other images to complete the encoding, such as relying on the previous image to complete the encoding.

[0054] Video encoding is a compression technology developed for image storage. It compresses images into a smaller bitstream and stores it on a hard drive. When the video is played, the bitstream is decoded back into an image and displayed. Video encoding methods include H.264 and H.265.

[0055] Video decoding is a technology that re-decodes a code stream into an image. After decoding a code stream (such as a code stream stored on a hard disk or transmitted from a network) into an image, the image can be displayed to achieve video playback.

[0056] During video encoding and decoding, an I frame and all P frames between the I frame and the next I frame can be combined into a GOP sequence. That is, the first frame of a GOP sequence is an I frame, and there is only one I frame in a GOP sequence. The I frame in the GOP sequence may be followed by a P frame. For example, consecutive video frames include frames 1-10, where frame 1 is an I frame, frames 2-5 are P frames, frame 6 is an I frame, and frames 7-10 are P frames. Then, GOP sequence 1 includes frames 1-5, and GOP sequence 2 includes frames 6-10. See Figure 1A, which shows GOP sequence 1 and GOP sequence 2. Each GOP sequence may include one I frame and multiple P frames.

[0057] For each GOP sequence, each P frame within the GOP sequence needs to refer to the previous frame. For example, for GOP sequence 1, frame 2 references frame 1 to complete inter-frame coding, frame 3 references frame 2 to complete inter-frame coding, frame 4 references frame 3 to complete inter-frame coding, and frame 5 references frame 4 to complete inter-frame coding. However, in the above structure, all P frames cannot be discarded. Once a P frame is discarded, subsequent P frames cannot be processed correctly. For example, as shown in Figure 1B, if frame 3 is discarded, frame 4 cannot complete inter-frame coding with reference to frame 3, and frame 5 cannot complete inter-frame coding with reference to frame 4. In other words, all P frames after frame 3 cannot be processed correctly. Under normal circumstances, there will be a time of 2 to 10 seconds between two I frames. If a P frame is discarded, the image for several seconds cannot be displayed normally.

[0058] In response to the above problems, a new video frame sequence is provided in an embodiment of the present application. The video frame sequence may include different types of P frames, such as P frames that cannot be used as reference frames and P frames that can be used as reference frames. The video frame sequence may drop frames according to a certain rule, which can ensure that the video frame sequence is played in order and that the viewing is not affected when P frames are dropped. The purpose of saving bandwidth can also be achieved by dropping some P frames. The new video frame sequence in the embodiment of the present application can divide all P frames in a GOP sequence into at least two types of P frames, at least two types of P frames include P frames that cannot be used as reference frames, and at least two types of P frames include P frames that can be used as reference frames. On this basis, if it is determined that the P frames in the GOP sequence need to be extracted, the P frames that cannot be used as reference frames are extracted, or if it is determined that the P frames in the GOP sequence need to be extracted, the P frames that cannot be used as reference frames and the P frames that can be used as reference frames are extracted.

[0059] The technical solutions of the embodiments of the present application are described below in conjunction with specific embodiments.

[0060] In an embodiment of the present application, an image processing method is proposed. The method can be applied to any device. FIG2 is a flowchart of the image processing method. The method may include:

[0061] Step 201: Divide all P frames in the GOP sequence into first-category P frames and second-category P frames; illustratively, the first-category P frames cannot be used as reference frames, the second-category P frames can be used as reference frames for the first-category P frames, and the second-category P frames can also be used as reference frames for the second-category P frames.

[0062] Exemplarily, for each GOP sequence, the GOP sequence may include an I frame and multiple P frames. In an embodiment of the present application, all P frames in the GOP sequence may be divided into two types of P frames, and these two types of P frames may be referred to as first-type P frames and second-type P frames, respectively.

[0063] Among them, first-class P frames, also known as ordinary P frames, cannot be used as reference frames for other frames. First-class P frames can use other frames as reference frames for themselves. For example, a first-class P frame can use a second-class P frame as a reference frame, and a first-class P frame can also use an I frame in a GOP sequence as a reference frame. However, first-class P frames cannot use other first-class P frames as reference frames.

[0064] Among them, the second type of P frame can also be called a referenced P frame (that is, the second type of P frame can be used as a reference frame). The second type of P frame can serve as a reference frame for the first type of P frame, and the second type of P frame can also serve as a reference frame for the second type of P frame. The second type of P frame can use other frames as reference frames for the second type of P frame. For example, the second type of P frame can use other second type of P frames as reference frames, and the second type of P frame can also use the I frame of the GOP sequence as a reference frame. However, the second type of P frame cannot use the first type of P frame as a reference frame.

[0065] In one possible implementation, when all P frames in a GOP sequence are divided into first-category P frames and second-category P frames, any division method may be used, without limitation, as long as both first-category P frames and second-category P frames exist. For example, all P frames in a GOP sequence may be divided into first-category P frames and second-category P frames based on a positional ordering relationship between the P frames in the GOP sequence, i.e., the division method is based on a positional ordering relationship. The positional ordering relationship refers to which P frame in the GOP sequence the P frame belongs to, such as the first P frame, the second P frame, etc., and is used to indicate the position of the P frame in the GOP sequence.

[0066] For example, the first P frame is a first-category P frame, the second P frame is a second-category P frame, the third P frame is a first-category P frame, the fourth P frame is a second-category P frame, and so on. For another example, the first P frame and the second P frame are first-category P frames, the third P frame and the fourth P frame are second-category P frames, and so on. For another example, the first P frame and the second P frame are first-category P frames, the third P frame is second-category P frame, and so on. For another example, the first P frame, the second P frame, and the third P frame are first-category P frames, the fourth P frame is second-category P frame, and so on. For another example, the first P frame is a first-category P frame, the second P frame and the third P frame are second-category P frames, and so on. Of course, the above division method is just an example.

[0067] For example, when all P frames are divided into first-category P frames and second-category P frames, the number of first-category P frames may be the same as the number of second-category P frames, or the number of first-category P frames may be different from the number of second-category P frames. When the number of first-category P frames is different from the number of second-category P frames, the number of first-category P frames may be greater than the number of second-category P frames, or the number of first-category P frames may be less than the number of second-category P frames. For example, the following example uses the number of first-category P frames being greater than the number of second-category P frames as an example.

[0068] For example, when all P frames are divided into first-category P frames and second-category P frames, at least two first-category P frames may appear consecutively, but two second-category P frames may not appear consecutively. For example, a first-category P frame may be followed by a first-category P frame or a second-category P frame, and a second-category P frame may be followed by a first-category P frame, but a second-category P frame may not be followed by a second-category P frame.

[0069] For example, the I frame of the GOP sequence may be followed by a first-type P frame, and the I frame of the GOP sequence may be followed by a second-type P frame. For example, the first P frame following the I frame of the GOP sequence is a first-type P frame.

[0070] In one possible implementation, when all P frames within a GOP sequence are divided into first-category P frames and second-category P frames, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between an I-frame (i.e., the first frame) of the GOP sequence and the first second-category P frame. M1 is a positive integer, and M2 is a positive integer. Both M1 and M2 can be configured based on experience, and M1 and M2 can be the same or different. When there are M1 first-category P frames between two adjacent second-category P frames, M1 can vary. For example, there are three first-category P frames between the first second-category P frame and the second second-category P frame, four first-category P frames between the second second-category P frame and the third second-category P frame, three first-category P frames between the third second-category P frame and the fourth second-category P frame, and so on.

[0071] Exemplarily, a first-category P frame needs to be included between two adjacent second-category P frames, and the number of first-category P frames can be kept fixed, such as the number of first-category P frames is more than one. However, in order to prevent poor image quality, the number of first-category P frames cannot be too large. For the convenience of description, in the embodiment of the present application, it is taken as an example that three first-category P frames are included between two adjacent second-category P frames, and there are three first-category P frames between the I frame of the GOP sequence and the first second-category P frame. Of course, the above division method is only an example and is not limited to this. For example, four first-category P frames can be included between two adjacent second-category P frames, or two first-category P frames can be included between two adjacent second-category P frames.

[0072] For ease of description, in the embodiment of the present application, taking the GOP sequence shown in FIG3A as an example, the first frame following an I frame is a first-category P frame. There are three first-category P frames between the I frame and the first second-category P frame. There are three first-category P frames between two adjacent second-category P frames. Referring to FIG3A , the frames within the GOP sequence are, in order: I frame, first-category P frame, first-category P frame, first-category P frame, second-category P frame, first-category P frame, first-category P frame, first-category P frame, second-category P frame, and so on.

[0073] In one possible implementation, based on the characteristic that "first-category P frames cannot be used as reference frames, second-category P frames can be used as reference frames for first-category P frames, and second-category P frames can be used as reference frames for second-category P frames," the following applies: for each first-category P frame, if a second-category P frame precedes the first-category P frame, the immediately preceding second-category P frame is used as the reference frame for the first-category P frame; if no second-category P frame precedes the first-category P frame, the I frame in the GOP sequence is used as the reference frame for the first-category P frame. For each second-category P frame, if a second-category P frame precedes the second-category P frame, the immediately preceding second-category P frame is used as the reference frame for the second-category P frame; if no second-category P frame precedes the second-category P frame, the I frame in the GOP sequence is used as the reference frame for the second-category P frame.

[0074] 3B shows the relationship between each P frame and the reference frame. For the first type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of P frame is the previous I frame, the reference frame of the third type of P frame is the previous I frame, the reference frame of the fourth type of P frame is the previous second type of P frame, the reference frame of the fifth type of P frame is the previous second type of P frame, the reference frame of the sixth type of P frame is the previous second type of P frame, and so on. For the second type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of P frame is the previous second type of P frame, the reference frame of the third type of P frame is the previous second type of P frame, and so on.

[0075] Step 202: If it is determined that P frames within the GOP sequence need to be extracted, then the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted. Exemplarily, extracting a P frame means not processing the P frame. For example, when the encoder extracts a P frame, the P frame is not encoded, and when the decoder extracts a P frame, the P frame is not decoded.

[0076] In one possible implementation, if it is determined that P frames within a GOP sequence need to be extracted, all or some of the first-category P frames may be extracted; alternatively, all or some of the first-category P frames may be extracted, and all or some of the second-category P frames may be extracted. As can be seen from the above, when extracting first-category P frames, it is not necessary to consider whether second-category P frames are extracted, and first-category P frames may be directly extracted. When extracting second-category P frames, it is necessary to consider whether first-category P frames are extracted, and only after all first-category P frames are extracted can second-category P frames be extracted.

[0077] In one possible implementation, in order to extract P frames within a GOP sequence, multiple sampling ratios can be divided, and the sampling ratio is used to indicate how many P frames are extracted. For example, when the sampling ratio is x, it means that x of all P frames are extracted. Different sampling ratios represent different sampling methods.

[0078] For example, two sampling ratios can be divided, which are recorded as a first sampling ratio and a second sampling ratio, and the first sampling ratio is smaller than the second sampling ratio. For the first sampling ratio, all or some of the first-category P frames can be decimated; for the second sampling ratio, all the first-category P frames can be decimated and all or some of the second-category P frames can be decimated.

[0079] For another example, three sampling ratios can be divided: a first sampling ratio, a second sampling ratio, and a third sampling ratio. The first sampling ratio is smaller than the second sampling ratio, and the second sampling ratio is smaller than the third sampling ratio. For the first sampling ratio, some first-category P frames can be decimated; for the second sampling ratio, all first-category P frames can be decimated; and for the third sampling ratio, all first-category P frames can be decimated, and all or some second-category P frames can be decimated.

[0080] For another example, four sampling ratios can be divided, denoted as the first sampling ratio, the second sampling ratio, the third sampling ratio, and the fourth sampling ratio. The first sampling ratio is smaller than the second sampling ratio, the second sampling ratio is smaller than the third sampling ratio, and the third sampling ratio is smaller than the fourth sampling ratio. For the first sampling ratio, some first-category P frames are decimated; for the second sampling ratio, all first-category P frames are decimated; for the third sampling ratio, all first-category P frames are decimated and some second-category P frames are decimated; for the fourth sampling ratio, all first-category P frames are decimated and all second-category P frames are decimated.

[0081] Of course, the above is only an example of the division of sampling ratios. There is no limitation to this. Any number of sampling ratios can be divided. For the convenience of description, the following explanation will be given by taking the division of 2 or 3 sampling ratios as an example.

[0082] Exemplarily, based on the division of the first sampling ratio, the second sampling ratio and the third sampling ratio, if it is necessary to use the first sampling ratio to sample the P frames in the GOP sequence, then some of the first-category P frames are sampled; if it is necessary to use the second sampling ratio to sample the P frames in the GOP sequence, then all of the first-category P frames are sampled; if it is necessary to use the third sampling ratio to sample the P frames in the GOP sequence, then all of the first-category P frames are sampled, and all or some of the second-category P frames are sampled.

[0083] Assuming that the total number of P frames of the first category accounts for x1 of the total number of P frames, the total number of P frames of the second category accounts for x2 of the total number of P frames, and the sum of x1 and x2 is 1, then the first sampling ratio is less than x1, the second sampling ratio is less than or equal to x1, and taking the second sampling ratio as an example, the third sampling ratio is less than or equal to x1 + x2. Referring to Figures 3A and 3B, where x1 is 0.75 and x2 is 0.25, if the sampling ratio is less than or equal to 0.5, the sampling ratio can be determined to be the first sampling ratio; if the sampling ratio is greater than 0.5 and less than or equal to 0.75, the sampling ratio can be determined to be the second sampling ratio; and if the sampling ratio is greater than 0.75 and less than or equal to 1, the sampling ratio can be determined to be the third sampling ratio.

[0084] In summary, when decimating P frames, all or part of the first-category P frames are prioritized for decimation. After all first-category P frames are decimated, all or part of the second-category P frames are decimated. When decimating part of the second-category P frames, the preceding second-category P frames are retained, and subsequent second-category P frames are decimated. For example, all second-category P frames after the kth (greater than 1) second-category P frame are decimated.

[0085] As shown in FIG4A , if it is determined that the first sampling ratio needs to be used to decimate P frames within a GOP sequence, some of the first-category P frames are decimated. In this case, the first sampling ratio is less than or equal to 0.5. Taking the first sampling ratio of 0.5 as an example, two first-category P frames between two second-category P frames need to be decimated, that is, one first-category P frame is retained between the two second-category P frames. For example, three first-category P frames between two second-category P frames are decimated according to a strategy of discarding one, retaining one, and discarding one. As shown in FIG4B , if it is determined that the second sampling ratio needs to be used to decimate P frames within a GOP sequence, all first-category P frames are decimated. In this case, the second sampling ratio is less than or equal to 0.75 and greater than 0.5. As shown in Figure 4C, if it is determined that the third sampling ratio needs to be used to sample the P frames in the GOP sequence, all first-category P frames can be sampled, and all second-category P frames can be sampled. In this case, the third sampling ratio is less than or equal to 1 and greater than 0.75.

[0086] As shown in Figure 4A, a GOP sequence can achieve 1 / 2 frame extraction. When 1 / 2 frame extraction cannot meet the business needs, as shown in Figure 4B, a GOP sequence can achieve 3 / 4 frame extraction. When 3 / 4 frame extraction cannot meet the business needs, as shown in Figure 4C, a GOP sequence can achieve full frame extraction.

[0087] For example, based on the division of the first sampling ratio and the second sampling ratio, if it is determined that the first sampling ratio needs to be used to sample the P frames in the GOP sequence, all first-category P frames can be sampled; if it is determined that the second sampling ratio needs to be used to sample the P frames in the GOP sequence, all first-category P frames are sampled, and all second-category P frames or some second-category P frames are sampled.

[0088] Assuming that the total number of P frames of the first category accounts for x1 of the total number of P frames, the total number of P frames of the second category accounts for x2 of the total number of P frames, and the sum of x1 and x2 is 1, then the first sampling ratio is less than or equal to x1, and the second sampling ratio is less than or equal to x1 + x2. Referring to Figures 3A and 3B, where x1 is 0.75 and x2 is 0.25, if the sampling ratio is less than or equal to 0.75, the sampling ratio is determined to be the first sampling ratio. If the sampling ratio is greater than 0.75 and less than or equal to 1, the sampling ratio is determined to be the second sampling ratio.

[0089] 4B , if it is determined that the first sampling ratio is required for decimation of P frames within a GOP sequence, all first-category P frames may be decimated. In this case, the first sampling ratio is less than or equal to 0.75. As shown in FIG4C , if it is determined that the second sampling ratio is required for decimation of P frames within a GOP sequence, all first-category P frames may be decimated, and all second-category P frames may be decimated. In this case, the second sampling ratio is less than or equal to 1 and greater than 0.75.

[0090] As shown in FIG4B , a GOP sequence can implement 3 / 4 frame extraction. When 3 / 4 frame extraction cannot meet the service demand, as shown in FIG4C , a GOP sequence can implement full frame extraction.

[0091] In a possible implementation, based on the division of the first sampling ratio, the second sampling ratio and the third sampling ratio, the sampling ratio corresponding to the GOP sequence can be determined based on the business information. For example, based on the business information, it is determined that the first sampling ratio is used to sample the P frames in the GOP sequence, or based on the business information, it is determined that the second sampling ratio is used to sample the P frames in the GOP sequence, or based on the business information, it is determined that the third sampling ratio is used to sample the P frames in the GOP sequence.

[0092] If the first sampling ratio is used to extract P frames within a GOP sequence and the service requirements corresponding to the service information are met, for example, extracting 50% of the P frames from the GOP sequence can meet the service requirements, then the first sampling ratio is determined to be adopted based on the service information. If the first sampling ratio cannot meet the service requirements, but the second sampling ratio is used to extract P frames from the GOP sequence and the service requirements corresponding to the service information are met, that is, extracting 75% of the P frames from the GOP sequence can meet the service requirements, then the second sampling ratio is determined to be adopted based on the service information. If the first sampling ratio cannot meet the service requirements and the second sampling ratio cannot meet the service requirements, then the third sampling ratio is determined to be adopted based on the service information.

[0093] In another possible implementation, based on the division of the first sampling ratio and the second sampling ratio, the sampling ratio corresponding to the GOP sequence can be determined based on service information. For example, based on the service information, it can be determined that the first sampling ratio is used to extract P frames within the GOP sequence, or based on the service information, it can be determined that the second sampling ratio is used to extract P frames within the GOP sequence. For example, if extracting P frames within the GOP sequence using the first sampling ratio can meet the service requirements corresponding to the service information, that is, extracting 75% of the P frames from the GOP sequence can meet the service requirements, then the first sampling ratio is determined to be used based on the service information. If the first sampling ratio cannot meet the service requirements, then the second sampling ratio is determined to be used based on the service information.

[0094] Based on the division of the first sampling ratio, the second sampling ratio, and the third sampling ratio, the above-mentioned frame extraction process is described below in conjunction with two specific application scenarios of business information. Of course, the implementation methods based on the division of the first sampling ratio and the second sampling ratio can be similar, and will not be repeated in this article.

[0095] Application scenario 1: The service information may include the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end. Based on the resolution, the frame rate, and the decoding capability, it is determined whether the P frames in the GOP sequence need to be extracted. If not, the P frames in the GOP sequence are not extracted. If so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate, and the decoding capability. For example, the sampling ratio corresponding to the GOP sequence may be the first sampling ratio, the second sampling ratio, or the third sampling ratio. Alternatively, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate, and the decoding capability, such as the first sampling ratio or the second sampling ratio.

[0096] For example, assume that the NVR device has a 4*1080P30 decoding capability, that is, the NVR device can decode 4*30 frames of 1920*1080 images per second. Images that exceed this decoding capability cannot be decoded.

[0097] a. If the GOP sequence corresponds to a 4*30 frame image (resolution of 1920*1080, frame rate of 4*30 frames), then based on the resolution, frame rate, and decoding capability, it is determined that there is no need to extract P frames in the GOP sequence. That is, the resolution, frame rate, and decoding capability match. Therefore, the NVR device does not extract P frames in the GOP sequence. It can decode 4*30 frames of images in one second and display 1080P30 images through four channels, as shown in Figure 5A.

[0098] b. If the GOP sequence corresponds to an 8*30 frame image (resolution of 1920*1080, frame rate of 8*30 frames), then based on the resolution, frame rate and decoding capability, it is determined that the P frames in the GOP sequence need to be decimated. That is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, the NVR device determines that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the first sampling ratio (that is, 1 / 2 decimation). When 4*30 frames of images can be decoded in one second, 1080P15 images can be displayed through 8 channels, as shown in Figure 5B. Obviously, the 15 frames of the 8 channels correspond to 4*30 frames.

[0099] Obviously, when the number of decoding channels increases, such as from 4 to 8, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0100] c. If the GOP sequence corresponds to a 4*60 frame image (resolution of 1920*1080, frame rate of 4*60 frames), then based on the resolution, frame rate and decoding capability, it is determined that the P frames in the GOP sequence need to be decimated. That is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, the NVR device determines that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the first sampling ratio (that is, 1 / 2 decimation). When 4*30 frames of images can be decoded in one second, 1080P30 images can be displayed through 4 channels, as shown in Figure 5C. Obviously, after the 4*60 frames are decimated by 1 / 2, 4*30 frames remain.

[0101] Obviously, when playing back at double speed, such as changing from 30 frames to 60 frames, the decoding capability can support decoding images of this resolution and frame rate by extracting frames from each image to meet business needs. Obviously, 30 frames at double speed is equivalent to 60 frames per second, and extracting frames at 1 / 2 is equal to 30 frames per second.

[0102] d. If the GOP sequence corresponds to a 12*30 frame image (resolution of 1920*1080, frame rate of 12*30 frames), then based on the resolution, frame rate and decoding capability, it is determined that the P frames in the GOP sequence need to be decimated. That is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, the NVR device determines that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the second sampling ratio (that is, 3 / 4 decimation). When 4*30 frames of images can be decoded in one second, 1080P10 images can be displayed through 12 channels, as shown in Figure 5D. Obviously, the 10 frames of the 12 channels correspond to 4*30 frames.

[0103] Obviously, when the number of decoding channels increases, such as from 4 to 12, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0104] e. If the GOP sequence corresponds to a 16*30 frame image (resolution of 1920*1080, frame rate of 16*30 frames), then based on the resolution, frame rate and decoding capability, it is determined that the P frames in the GOP sequence need to be decimated. That is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, the NVR device determines that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the third sampling ratio (that is, 4 / 4 decimation). When 4*30 frames of images can be decoded in one second, 1080P7.5 images can be displayed through 16 channels, as shown in Figure 5E. Obviously, the 7.5 frames of 16 channels correspond to 4*30 frames.

[0105] Obviously, when the number of decoding channels increases, such as from 4 to 16 channels, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0106] From the above, it can be seen that when the conditions of multiple speeds and / or multiple channels change arbitrarily, frames can be sampled according to the first sampling ratio, the second sampling ratio, or the third sampling ratio to meet business needs.

[0107] Application Scenario 2: Service information may include the network quality of the target network between the transmitter and receiver. Based on the network quality, it is determined whether P frames within the GOP sequence need to be extracted. If not, the P frames within the GOP sequence are not extracted. If so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality. For example, the sampling ratio corresponding to the GOP sequence may be the first sampling ratio, the second sampling ratio, or the third sampling ratio. Alternatively, the sampling ratio corresponding to the GOP sequence is determined based on the network quality. The sampling ratio corresponding to the GOP sequence may be the first sampling ratio or the second sampling ratio.

[0108] For example, when a transmitter sends a GOP sequence to a receiver, it is necessary to detect the network quality (such as jitter, packet loss rate, etc.) corresponding to the target network used to carry the GOP sequence, and there is no restriction on the detection method.

[0109] In one possible implementation, four levels of network quality may be designed. If the network quality corresponding to the target network is the first level (i.e., the optimal level) of network quality, it is determined that the P frames in the GOP sequence will not be sampled. If the network quality corresponding to the target network is the second level of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the first sampling ratio, such as a 1 / 2 sample method. If the network quality corresponding to the target network is the third level of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the second sampling ratio, such as a 3 / 4 sample method. If the network quality corresponding to the target network is the fourth level (i.e., the worst level) of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the third sampling ratio, such as a 4 / 4 sample method.

[0110] In another possible implementation, two levels of network quality can be designed. If the network quality corresponding to the target network is the first level (i.e., the optimal level), it is determined that the P frames in the GOP sequence will not be extracted. If the network quality corresponding to the target network is the second level, it is determined that the P frames in the GOP sequence will be extracted, and the sampling ratio is the first sampling ratio. During the transmission of the GOP sequence, if the network quality is still the second level, the sampling ratio is adjusted to the second sampling ratio. During the transmission of the GOP sequence, if the network quality is still the second level, the sampling ratio is adjusted to the third sampling ratio. In addition, during the transmission of the GOP sequence, if the network quality returns to the first level, the P frames in the GOP sequence will not be extracted.

[0111] Of course, the two application scenarios described above are merely examples of determining the sampling ratio. Other methods for determining the sampling ratio corresponding to a GOP sequence may also be used, and are not limited thereto. For example, the sampling ratio corresponding to a GOP sequence may be determined based on the size of each frame. For example, when the frame size is less than a first threshold, no P-frames within the GOP sequence are sampled. When the frame size is greater than the first threshold and less than a second threshold, the sampling ratio is the first sampling ratio. When the frame size is greater than the second threshold and less than a third threshold, the sampling ratio is the second sampling ratio. When the frame size is greater than the third threshold, the sampling ratio is the third sampling ratio.

[0112] In one possible implementation, at least two GOP sequences may be combined into an image sequence set. Based on this, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence in the image sequence set, the I frame of the first GOP sequence may be used as a reference frame for the first frame of the GOP sequence. In this implementation, if the GOP sequence is not the first GOP sequence in the image sequence set, the first frame of the GOP sequence may be referred to as a key P frame (i.e., no longer referred to as an I frame), and the reference frame of the key P frame is the most recent preceding I frame.

[0113] Exemplarily, for an image sequence set, the image sequence set may include at least one key P frame, that is, at least one key P frame is allowed to exist, and the key P frame is used to replace the I frame. The key P frame can only refer to the previous I frame closest to it, and the frame structure of the key P frame can effectively reduce the overall bandwidth.

[0114] FIG6 is a schematic diagram of an image sequence set that may include key P frames. Between an I frame and a key P frame, a first-class P frame and a second-class P frame may exist. The structural relationship between the first-class P frame and the second-class P frame can be described in the above embodiment. Between two adjacent key P frames, a first-class P frame and a second-class P frame may exist. The structural relationship between the first-class P frame and the second-class P frame can be described in the above embodiment. Based on this, when extracting P frames, first-class P frames and second-class P frames may be extracted, but key P frames cannot be extracted.

[0115] As can be seen from the above technical solution, in the embodiment of the present application, all P frames in the GOP sequence can be divided into first-class P frames and second-class P frames. The first-class P frames cannot be used as reference frames, the second-class P frames can be used as reference frames for the first-class P frames, and the second-class P frames can also be used as reference frames for the second-class P frames. On this basis, the first-class P frames can be extracted, or the first-class P frames and the second-class P frames can be extracted. The above method provides a new video frame sequence, which includes different types of P frames, namely first-class P frames and second-class P frames. In the video frame sequence, frames can be dropped according to a certain rule, which can ensure that the video frame sequence is played in order and has the characteristics of intermittently dropping P frames according to the rules without affecting the viewing experience. In some special scenarios, some P frames can be dropped to save bandwidth. The video frame sequence has better usage prospects in decoding, network transmission, and storage, and realizes the characteristics of decoding multiplication, low-bandwidth transmission, low-bitrate storage, etc. In a weak network environment, it can also solve the problems of long-term black screen and flowery screen after frame loss. Frame drop strategies can be dynamically implemented and adjusted based on site conditions, flexibly and effectively reducing consumption without compromising decoding integrity. This video frame sequence ensures flexible frame extraction and stable image quality, effectively improving the customer experience. Frame drop can be implemented regularly during the decoding process to achieve decoding patterns such as 1 / 2 or 3 / 4 frames. Alternatively, 1 / 2 or 3 / 4 frames can be used for low-bandwidth transmission over the network. This frame drop strategy can be dynamically implemented and adjusted based on site conditions, flexibly and effectively reducing consumption without compromising decoding integrity.

[0116] An image processing method is proposed in an embodiment of the present application. The method can be applied to any device. FIG7 is a flowchart of the image processing method. The method may include:

[0117] Step 701: Divide all P frames in the GOP sequence into first-category P frames, second-category P frames, and third-category P frames; illustratively, the first-category P frames cannot be used as reference frames, the second-category P frames can only be used as reference frames for the first-category P frames, and the third-category P frames can be used as reference frames for the first-category P frames, as well as for the second-category P frames and as reference frames for the third-category P frames.

[0118] Exemplarily, for each GOP sequence, the GOP sequence may include an I frame and multiple P frames. In an embodiment of the present application, all P frames in the GOP sequence may be divided into three types of P frames, which are referred to as first-class P frames, second-class P frames, and third-class P frames.

[0119] Among them, the first type of P frame can also be called a normal P frame. The first type of P frame cannot be used as a reference frame for other frames. The first type of P frame can use other frames as reference frames for the first type of P frame. For example, the first type of P frame can use the second type of P frame as a reference frame, the first type of P frame can use the third type of P frame as a reference frame, and the first type of P frame can use the I frame of the GOP sequence as a reference frame. However, the first type of P frame cannot use the first type of P frame (i.e., other first type of P frames other than the first type of P frame) as a reference frame.

[0120] Among them, the second type of P frame can also be called 2X type P frame. The second type of P frame can only serve as a reference frame for the first type of P frame, but cannot serve as a reference frame for the second type of P frame or the third type of P frame. The second type of P frame can use other frames as reference frames for the second type of P frame. For example, the second type of P frame can use the third type of P frame as a reference frame. The second type of P frame can also use the I frame of the GOP sequence as a reference frame. However, the second type of P frame cannot use the second type of P frame (i.e., other second type of P frames other than the second type of P frame) as a reference frame, nor can the second type of P frame use the first type of P frame as a reference frame.

[0121] Among them, the third type of P frame can also be called a 4X type P frame. The third type of P frame can serve as a reference frame for the first type of P frame, as a reference frame for the second type of P frame, and as a reference frame for the third type of P frame. The third type of P frame can use other frames as reference frames for the third type of P frame. For example, the third type of P frame can use the third type of P frame (i.e., other third type of P frames other than the third type of P frame) as a reference frame. The third type of P frame can also use the I frame of the GOP sequence as a reference frame. However, the third type of P frame cannot use the second type of P frame as a reference frame, nor can the third type of P frame use the first type of P frame as a reference frame.

[0122] In one possible implementation, when all P frames in a GOP sequence are divided into first-category P frames, second-category P frames, and third-category P frames, any division method may be used. This embodiment does not impose any limitation on this division method, as long as first-category P frames, second-category P frames, and third-category P frames can coexist. Exemplarily, all P frames in a GOP sequence may be divided into first-category P frames, second-category P frames, and third-category P frames based on a positional ordering relationship between the P frames in the GOP sequence, i.e., the division method is based on a positional ordering relationship. The positional ordering relationship refers to which P frame in the GOP sequence the P frame is, such as the first P frame, the second P frame, the third P frame, etc., and is used to indicate the position of the P frame in the GOP sequence.

[0123] For example, the first P frame is a first-category P frame, the second P frame is a second-category P frame, the third P frame is a third-category P frame, the fourth P frame is a first-category P frame, the fifth P frame is a second-category P frame, the sixth P frame is a third-category P frame, and so on. For example, the first and second P frames are first-category P frames, the third and fourth P frames are second-category P frames, the fifth and sixth P frames are third-category P frames, and so on. For another example, the first P frame is a first-category P frame, the second P frame is a second-category P frame, the third P frame is a first-category P frame, the fourth P frame is a third-category P frame, the fifth P frame is a first-category P frame, and the sixth P frame is a second-category P frame, and so on. For example, the first and second P frames are first-category P frames, the third P frame is a second-category P frame, the fourth P frame is a third-category P frame, the fifth and sixth P frames are first-category P frames, and so on. Of course, the above divisions are just a few examples.

[0124] For example, when all P frames are divided into first-category P frames, second-category P frames, and third-category P frames, the number of first-category P frames may be the same as the number of second-category P frames, and the number of first-category P frames may be different from the number of second-category P frames. The number of first-category P frames may be the same as the number of third-category P frames, and the number of first-category P frames may be different from the number of third-category P frames. The number of second-category P frames may be the same as the number of third-category P frames, and the number of second-category P frames may be different from the number of third-category P frames.

[0125] For example, the number of P frames of the first category may be greater than the number of P frames of the second category, and the number of P frames of the first category may be less than the number of P frames of the second category. The number of P frames of the first category may be greater than the number of P frames of the third category, and the number of P frames of the first category may be less than the number of P frames of the third category. The number of P frames of the second category may be greater than the number of P frames of the third category, and the number of P frames of the second category may be less than the number of P frames of the third category.

[0126] For example, the number of the first type of P frames is greater than the number of the second type of P frames, the number of the first type of P frames is greater than the number of the third type of P frames, and the number of the second type of P frames is equal to the number of the third type of P frames.

[0127] For another example, the number of the second type P frames is equal to the number of the third type P frames, and the number of the first type P frames is greater than or equal to the sum of the number of the second type P frames and the number of the third type P frames.

[0128] For example, when all P frames are divided into first-category P frames, second-category P frames, and third-category P frames, the first-category P frames, second-category P frames, and third-category P frames may appear alternately, i.e., two first-category P frames will not appear consecutively, two second-category P frames will not appear consecutively, and two third-category P frames will not appear consecutively. For example, a first-category P frame may be followed by a second-category P frame or a third-category P frame, but not a first-category P frame; a second-category P frame may be followed by a first-category P frame or a third-category P frame, but not a second-category P frame; and a third-category P frame may be followed by a first-category P frame or a second-category P frame, but not a third-category P frame.

[0129] For example, the first type of P frame may be followed by the second type of P frame or the third type of P frame, the second type of P frame may be followed by the first type of P frame, and the third type of P frame may be followed by the first type of P frame.

[0130] Exemplarily, the I frame of the GOP sequence may be followed by a first-category P frame, the I frame of the GOP sequence may be followed by a second-category P frame, and the I frame of the GOP sequence may be followed by a third-category P frame, and there is no limitation to this.

[0131] For example, the first P frame following the I frame in a GOP sequence is a first-type P frame.

[0132] In one possible implementation, when all P frames in a GOP sequence are divided into first-category P frames, second-category P frames, and third-category P frames, two first-category P frames and one second-category P frame may exist between two adjacent third-category P frames, and one second-category P frame may be located between two first-category P frames. Furthermore, two first-category P frames and one second-category P frame may exist between an I-frame (i.e., the first frame) of a GOP sequence and the first third-category P frame, and the second-category P frame may be located between the two first-category P frames.

[0133] Of course, the above division method is only an example and is not intended to be limiting. For example, between two adjacent third-category P frames, there may be three first-category P frames and two second-category P frames, with the second-category P frames located between the two first-category P frames. For another example, between two adjacent third-category P frames, there may be two first-category P frames and three second-category P frames, with the first-category P frames located between the two second-category P frames. For another example, between two adjacent third-category P frames, there may be two first-category P frames and two second-category P frames, with the second-category P frames located between the two first-category P frames, and the first-category P frames located between the two second-category P frames. For another example, between two adjacent third-category P frames, there may be three first-category P frames and three second-category P frames, with the second-category P frames located between the two first-category P frames, and the first-category P frames located between the two second-category P frames.

[0134] For ease of description, in this embodiment, taking the GOP sequence shown in FIG8A as an example, the first frame following an I frame is a first-category P frame. Between the I frame and the first third-category P frame, there are two first-category P frames and one second-category P frame, and the second-category P frame is located between the two first-category P frames. Between two adjacent third-category P frames, there are two first-category P frames and one second-category P frame, and the second-category P frame is located between the two first-category P frames. Referring to FIG8A , the frames within the GOP sequence are, in order: I frame, first-category P frame, second-category P frame, first-category P frame, third-category P frame, first-category P frame, second-category P frame, first-category P frame, third-category P frame, first-category P frame, second-category P frame, first-category P frame, third-category P frame, and so on.

[0135] In one possible implementation, based on the characteristic that "first-category P frames are not used as reference frames, second-category P frames are used as reference frames for first-category P frames, and third-category P frames are used as reference frames for first-category P frames, second-category P frames, and third-category P frames," the following applies: for each first-category P frame, if the previous frame of the first-category P frame is not a first-category P frame, the previous frame of the first-category P frame (which may be an I-frame, a second-category P frame, or a third-category P frame) is used as the reference frame for the first-category P frame. If the previous frame of the first-category P frame is a first-category P frame, the most recent non-first-category P frame (such as an I-frame, a second-category P frame, or a third-category P frame) preceding the first-category P frame is used as the reference frame for the first-category P frame.

[0136] For each second-type P frame, if there is a third-type P frame preceding the second-type P frame, the third-type P frame closest to the second-type P frame is used as the reference frame for the second-type P frame. If there is no third-type P frame preceding the second-type P frame, the I frame in the GOP sequence is used as the reference frame for the second-type P frame.

[0137] For each third-category P frame, if there is a third-category P frame preceding it, the most recent third-category P frame preceding it is used as a reference frame for the third-category P frame. If there is no third-category P frame preceding it, the I frame in the GOP sequence is used as a reference frame for the third-category P frame.

[0138] 8B shows the relationship between each P frame and its reference frame. For the first type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of P frame is the previous second type of P frame, the reference frame of the third type of P frame is the previous third type of P frame, the reference frame of the fourth type of P frame is the previous second type of P frame, the reference frame of the fifth type of P frame is the previous third type of P frame, and so on. For the second type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of P frame is the previous third type of P frame, the reference frame of the third type of P frame is the previous third type of P frame, and so on. For the third type of P frame, the reference frame of the first type of P frame is the previous I frame, the reference frame of the second type of P frame is the previous third type of P frame, the reference frame of the third type of P frame is the previous third type of P frame, and so on.

[0139] Step 702: If it is determined that P frames within the GOP sequence need to be extracted, then the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted, or the first type of P frames, the second type of P frames, and the third type of P frames are extracted. Exemplarily, extracting a P frame may mean not processing the P frame. For example, when the encoder extracts a P frame, the P frame is not encoded, and when the decoder extracts a P frame, the P frame is not decoded.

[0140] In a possible implementation, if it is determined that the P frames in the GOP sequence need to be extracted, all first-category P frames or some first-category P frames may be extracted; alternatively, all first-category P frames may be extracted, and all second-category P frames or some second-category P frames may be extracted; alternatively, all first-category P frames may be extracted, all second-category P frames may be extracted, and all third-category P frames or some third-category P frames may be extracted. In summary, it can be seen that when extracting first-category P frames, it is not necessary to consider whether second-category P frames and third-category P frames are extracted, and first-category P frames may be directly extracted. When extracting second-category P frames, it is necessary to consider whether first-category P frames are extracted, and it is not necessary to consider whether third-category P frames are extracted. Only after all first-category P frames are extracted can second-category P frames be extracted. When extracting the third type of P frames, it is necessary to consider whether the first type of P frames are extracted and whether the second type of P frames are extracted. Only after all the first type of P frames are extracted and all the second type of P frames are extracted can the third type of P frames be extracted.

[0141] In one possible implementation, in order to extract P frames within a GOP sequence, multiple sampling ratios can be divided, and the sampling ratio is used to indicate how many P frames are extracted. For example, when the sampling ratio is x, it means that x of all P frames are extracted. Different sampling ratios represent different sampling methods.

[0142] For example, three sampling ratios are defined, namely the first sampling ratio, the second sampling ratio, and the third sampling ratio. The first sampling ratio is smaller than the second sampling ratio, and the second sampling ratio is smaller than the third sampling ratio. For the first sampling ratio, all or some of the first-category P frames are decimated. For the second sampling ratio, all the first-category P frames are decimated, and all or some of the second-category P frames are decimated. For the third sampling ratio, all the first-category P frames are decimated, all the second-category P frames are decimated, and all or some of the third-category P frames are decimated.

[0143] For another example, there are 6 sampling proportions, recorded as the first sampling proportion, the second sampling proportion, the third sampling proportion, the fourth sampling proportion, the fifth sampling proportion and the sixth sampling proportion. The first sampling proportion is smaller than the second sampling proportion, the second sampling proportion is smaller than the third sampling proportion, the third sampling proportion is smaller than the fourth sampling proportion, the fourth sampling proportion is smaller than the fifth sampling proportion, and the fifth sampling proportion is smaller than the sixth sampling proportion. For the first sampling ratio, some of the first category P frames are sampled; for the second sampling ratio, all of the first category P frames are sampled; for the third sampling ratio, all of the first category P frames are sampled, and some of the second category P frames are sampled; for the fourth sampling ratio, all of the first category P frames are sampled, and all of the second category P frames are sampled; for the fifth sampling ratio, all of the first category P frames are sampled, all of the second category P frames are sampled, and some of the third category P frames are sampled; for the sixth sampling ratio, all of the first category P frames are sampled, all of the second category P frames are sampled, and all of the third category P frames are sampled.

[0144] Of course, the above is only an example of the division of sampling ratios. There is no limitation to this. Any number of sampling ratios can be divided. For the convenience of description, the following explanation will be given by taking the division of 3 sampling ratios as an example.

[0145] On this basis, if it is determined that the first sampling ratio needs to be used to sample the P frames in the GOP sequence, then all the first-category P frames or some of the first-category P frames can be sampled; if it is determined that the second sampling ratio needs to be used to sample the P frames in the GOP sequence, then all the first-category P frames can be sampled, and all the second-category P frames or some of the second-category P frames can be sampled; if it is determined that the third sampling ratio needs to be used to sample the P frames in the GOP sequence, then all the first-category P frames can be sampled, all the second-category P frames can be sampled, and all the third-category P frames or some of the third-category P frames can be sampled.

[0146] For example, assuming that the total number of P frames of the first category accounts for x1 of the total number of P frames, the total number of P frames of the second category accounts for x2 of the total number of P frames, the total number of P frames of the third category accounts for x3 of the total number of P frames, and the sum of x1, x2 and x3 is 1, then the first sampling ratio is less than or equal to x1, the second sampling ratio is greater than x1 and less than or equal to x1+x2, and the third sampling ratio is greater than x1+x2 and less than or equal to x1+x2+x3.

[0147] Referring to Figures 8A and 8B, x1 is 0.5, x2 is 0.25, and x3 is 0.25. On this basis, if the sampling ratio is less than or equal to 0.5, it means that the sampling ratio is the first sampling ratio. If the sampling ratio is greater than 0.5 and less than or equal to 0.75, it means that the sampling ratio is the second sampling ratio. If the sampling ratio is greater than 0.75 and less than or equal to 1, it means that the sampling ratio is the third sampling ratio.

[0148] In summary, when extracting P frames, all or part of the first type of P frames are preferentially extracted. After all the first type of P frames are extracted, all or part of the second type of P frames can be extracted. After all the second type of P frames are extracted, all or part of the third type of P frames can be extracted. When extracting part of the third type of P frames, the preceding third type of P frames are retained, and the following third type of P frames are extracted. For example, all third type P frames after the kth (k is greater than 1) third type P frame are extracted.

[0149] For example, referring to FIG9A , if it is determined that a first sampling ratio is required for decimation of P frames within a GOP sequence, all first-category P frames may be decimated. In this case, the first sampling ratio is less than or equal to 0.5. Referring to FIG9B , if it is determined that a second sampling ratio is required for decimation of P frames within a GOP sequence, all first-category P frames may be decimated, and all second-category P frames may be decimated. In this case, the second sampling ratio is less than or equal to 0.75 and greater than 0.5. Referring to FIG9C , if it is determined that a third sampling ratio is required for decimation of P frames within a GOP sequence, all first-category P frames may be decimated, all second-category P frames may be decimated, and all third-category P frames may be decimated. In this case, the third sampling ratio is less than or equal to 1 and greater than 0.75.

[0150] As shown in Figure 9A, a GOP sequence can achieve 1 / 2 frame extraction. When 1 / 2 frame extraction cannot meet the business needs, as shown in Figure 9B, a GOP sequence can achieve 3 / 4 frame extraction. When 3 / 4 frame extraction cannot meet the business needs, as shown in Figure 9C, a GOP sequence can achieve full frame extraction.

[0151] In one possible implementation, the sampling ratio corresponding to the GOP sequence can be determined based on business information. For example, based on the business information, it is determined that the P frames in the GOP sequence are sampled using a first sampling ratio, or based on the business information, it is determined that the P frames in the GOP sequence are sampled using a second sampling ratio, or based on the business information, it is determined that the P frames in the GOP sequence are sampled using a third sampling ratio.

[0152] If the first sampling ratio is used to extract P frames within a GOP sequence, the business requirements corresponding to the business information can be met, that is, extracting x1 (such as 0.5) P frames from the GOP sequence can meet the business requirements, and the first sampling ratio is determined to be used based on the business information. If the first sampling ratio cannot meet the business requirements, but the second sampling ratio is used to extract P frames within the GOP sequence, the business requirements corresponding to the business information can be met, that is, extracting x1+x2 (such as 0.75) P frames from the GOP sequence can meet the business requirements, and the second sampling ratio is determined to be used based on the business information. If the first sampling ratio cannot meet the business requirements, and the second sampling ratio cannot meet the business requirements, the third sampling ratio is determined to be used based on the business information.

[0153] The following describes the frame extraction process in conjunction with two specific application scenarios of service information.

[0154] Application Scenario 1: Service information includes the resolution corresponding to the GOP sequence (e.g., 4*30 frames represent the frame rate), the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end. Based on the resolution, frame rate, and decoding capability, it is determined whether to perform frame extraction on the P frames within the GOP sequence. If not, the P frames within the GOP sequence are not extracted. If so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, frame rate, and decoding capability, such as the first sampling ratio, the second sampling ratio, or the third sampling ratio.

[0155] For example, assume that the NVR device has a 4*1080P30 decoding capability, that is, the NVR device can decode 4*30 frames of 1920*1080 images per second. Images that exceed this decoding capability cannot be decoded.

[0156] a. If the GOP sequence corresponds to a 4*30 frame image (resolution 1920*1080), then based on the resolution, frame rate, and decoding capability, it is determined that there is no need to extract P frames in the GOP sequence. That is, the resolution, frame rate, and decoding capability match. Therefore, the NVR device does not extract P frames in the GOP sequence. It can decode 4*30 frames of images in one second and display 1080P30 images through four channels, as shown in Figure 5A.

[0157] b. If the GOP sequence corresponds to an 8*30 frame image (resolution is 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be decimated, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be decimated, and each image channel is decimated according to the first sampling ratio (that is, 1 / 2 decimation). When 4*30 frames of images can be decoded in one second, 1080P15 images can be displayed through 8 channels, as shown in Figure 5B. Obviously, the 15 frames of the 8 channels correspond to 4*30 frames.

[0158] Obviously, when the number of decoding channels increases, such as from 4 to 8, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0159] c. If the GOP sequence corresponds to a 4*60 frame image (resolution 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be decimated, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images with this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the first sampling ratio (that is, 1 / 2 decimation). When 4*30 frames of images can be decoded in one second, 1080P30 images can be displayed through 4 channels, as shown in FIG5C . Obviously, after the 4*60 frames are decimated by 1 / 2, 4*30 frames remain.

[0160] Obviously, when playing back at double speed, such as changing from 30 frames to 60 frames, the decoding capability can support decoding images of this resolution and frame rate by extracting frames from each image to meet business needs. That is to say, 30 frames at double speed is equivalent to 60 frames per second, and extracting 1 / 2 frames is equal to 30 frames per second.

[0161] d. If the GOP sequence corresponds to an image of 12*30 frames (resolution of 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be decimated, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding of images with this resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be decimated, and each image channel is decimated according to the second sampling ratio (that is, 3 / 4 decimation). When 4*30 frames of images can be decoded in one second, 1080P10 images can be displayed through 12 channels, as shown in Figure 5D. Obviously, the 10 frames of the 12 channels correspond to 4*30 frames.

[0162] Obviously, when the number of decoding channels increases, such as from 4 to 12, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0163] e. If the GOP sequence corresponds to an image of 16*30 frames (resolution of 1920*1080), it is determined based on the resolution, frame rate and decoding capability that the P frames in the GOP sequence need to be decimated, that is, the resolution, frame rate and decoding capability do not match (that is, the decoding capability does not support decoding images of the resolution and frame rate). Therefore, it is determined that the P frames in the GOP sequence need to be decimated, and each image is decimated according to the third sampling ratio (that is, 4 / 4 decimation). When 4*30 frames of images can be decoded in one second, 1080P7.5 images can be displayed through 16 channels, as shown in Figure 5E. Obviously, the 7.5 frames of the 16 channels correspond to 4*30 frames.

[0164] Obviously, when the number of decoding channels increases, such as from 4 to 16 channels, the decoding capability can support decoding images of the resolution and frame rate by extracting frames from each channel to meet business needs.

[0165] From the above, it can be seen that when the conditions of multiple speeds and / or multiple channels change arbitrarily, frames can be sampled according to the first sampling ratio, the second sampling ratio, or the third sampling ratio to meet business needs.

[0166] Application Scenario 2: If the service information includes the network quality of the target network between the transmitter and receiver, whether P frames within the GOP sequence need to be decimated is determined based on the network quality. If not, P frames within the GOP sequence are not decimated. If so, a sampling ratio corresponding to the GOP sequence is determined based on the network quality, such as a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0167] For example, when a transmitter sends a GOP sequence to a receiver, it is necessary to detect the network quality (such as jitter, packet loss rate, etc.) corresponding to the target network used to carry the GOP sequence, and there is no restriction on the detection method.

[0168] In one possible implementation, four levels of network quality may be designed. If the network quality corresponding to the target network is the first level (i.e., the optimal level) of network quality, it is determined that the P frames in the GOP sequence will not be sampled. If the network quality corresponding to the target network is the second level of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the first sampling ratio, such as a 1 / 2 sample method. If the network quality corresponding to the target network is the third level of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the second sampling ratio, such as a 3 / 4 sample method. If the network quality corresponding to the target network is the fourth level (i.e., the worst level) of network quality, it is determined that the P frames in the GOP sequence will be sampled, and the sampling ratio will be the third sampling ratio, such as a 4 / 4 sample method.

[0169] In another possible implementation, two levels of network quality can be designed. If the network quality corresponding to the target network is the first level (i.e., the optimal level), it is determined that the P frames in the GOP sequence will not be extracted. If the network quality corresponding to the target network is the second level, it is determined that the P frames in the GOP sequence will be extracted, and the sampling ratio is the first sampling ratio. During the transmission of the GOP sequence, if the network quality is still the second level, the sampling ratio is adjusted to the second sampling ratio. During the transmission of the GOP sequence, if the network quality is still the second level, the sampling ratio is adjusted to the third sampling ratio. In addition, during the transmission of the GOP sequence, if the network quality returns to the first level, the P frames in the GOP sequence will not be extracted.

[0170] Of course, the two application scenarios described above are merely examples of determining the sampling ratio. Other methods for determining the sampling ratio corresponding to a GOP sequence may also be used, and are not limited thereto. For example, the sampling ratio corresponding to a GOP sequence may be determined based on the size of each frame. For example, when the frame size is less than a first threshold, no P-frames within the GOP sequence are sampled. When the frame size is greater than the first threshold and less than a second threshold, the sampling ratio is the first sampling ratio. When the frame size is greater than the second threshold and less than a third threshold, the sampling ratio is the second sampling ratio. When the frame size is greater than the third threshold, the sampling ratio is the third sampling ratio.

[0171] In one possible implementation, at least two GOP sequences may be combined into an image sequence set. Based on this, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence in the image sequence set, the I frame of the first GOP sequence may be used as a reference frame for the first frame of the GOP sequence. In this implementation, if the GOP sequence is not the first GOP sequence in the image sequence set, the first frame of the GOP sequence may be referred to as a key P frame (i.e., no longer referred to as an I frame), and the reference frame of the key P frame is the most recent preceding I frame.

[0172] Exemplarily, for an image sequence set, the image sequence set may include at least one key P frame, that is, at least one key P frame is allowed to exist, and the key P frame is used to replace the I frame. The key P frame can only refer to the previous I frame closest to it, and the frame structure of the key P frame can effectively reduce the overall bandwidth.

[0173] FIG10 is a schematic diagram of an image sequence set that may include key P frames. Between an I-frame and a key P frame, there may be first, second, and third type P frames. The structural relationship among these first, second, and third types of P frames can be found in the above-described embodiments. Between two adjacent key P frames, there may be first, second, and third type P frames. The structural relationship among these first, second, and third types of P frames can be found in the above-described embodiments.

[0174] When extracting P frames, the extraction can be performed on the first type of P frames, the second type of P frames, and the third type of P frames, but the extraction cannot be performed on key P frames.

[0175] It can be seen from the above technical solutions that this embodiment provides a new type of video frame sequence, which includes a first type of P frame, a second type of P frame and a third type of P frame. Frames can be dropped according to a certain rule, which can ensure that the video frame sequence is played in order and has the characteristics of intermittently dropping P frames according to the rules without affecting the viewing experience. In addition, some P frames can be dropped in some special scenarios to save bandwidth. The video frame sequence has better prospects for use in decoding, network transmission, and storage, and can achieve the characteristics of multiplied decoding paths, low-bandwidth transmission, and low-bitrate storage. In a weak network environment, it can also solve problems such as long-term black screen and flowery screen after frame loss. The frame dropping strategy can be implemented dynamically, and the frame dropping strategy can be adjusted arbitrarily according to the on-site situation. Without affecting the decoding integrity, consumption can be flexibly and effectively reduced. The video frame sequence can ensure the flexibility of frame extraction and the stability of image quality, which can effectively improve customer experience.

[0176] Based on the same application concept as the above method, an image processing device is proposed in an embodiment of the present application. FIG11 is a schematic structural diagram of the image processing device. The device may include:

[0177] a dividing module 111 configured to divide all P frames in a group of pictures (GOP) sequence into at least two types of P frames, wherein some of the at least two types of P frames cannot be used as reference frames, and some of the at least two types of P frames can be used as reference frames;

[0178] The processing module 112 is configured to, if it is determined that the P frames in the GOP sequence need to be extracted, extract the P frames that cannot be used as reference frames, or, if it is determined that the P frames in the GOP sequence need to be extracted, extract the P frames that cannot be used as reference frames and the P frames that can be used as reference frames.

[0179] Exemplarily, the division module 111 is specifically used to divide all P frames in the GOP sequence into first-category P frames and second-category P frames; wherein, the first-category P frames cannot be used as reference frames, the second-category P frames can be used as reference frames for the first-category P frames, and the second-category P frames can be used as reference frames for the second-category P frames; the processing module 112 is specifically used to extract the first-category P frames if it is determined that the P frames in the GOP sequence need to be extracted, or to extract the first-category P frames and the second-category P frames.

[0180] Exemplarily, when the division module 111 divides all P frames in a GOP sequence into first-category P frames and second-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames and second-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between the I frame of the GOP sequence and the first second-category P frame; M1 and M2 are both positive integers.

[0181] Exemplarily, for each first-category P frame, if there is a second-category P frame preceding the first-category P frame, the second-category P frame that is most immediately preceding the first-category P frame is used as a reference frame for the first-category P frame; if there is no second-category P frame preceding the first-category P frame, the I frame of the GOP sequence is used as a reference frame for the first-category P frame; for each second-category P frame, if there is a second-category P frame preceding the second-category P frame, the second-category P frame that is most immediately preceding the second-category P frame is used as a reference frame for the second-category P frame; if there is no second-category P frame preceding the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame.

[0182] Exemplarily, the processing module 112 determines that it is necessary to extract the P frames in the GOP sequence, then extract the first type of P frames, or extract the first type of P frames and the second type of P frames. Specifically, if it is determined based on the service information that the first sampling ratio is required to be used for extracting the P frames in the GOP sequence, then all the first type of P frames or part of the first type of P frames are extracted; if it is determined based on the service information that the second sampling ratio is required to be used for extracting the P frames in the GOP sequence, then all the first type of P frames are extracted, and all the second type of P frames or part of the second type of P frames are extracted; or if If it is determined based on business information that the first sampling ratio needs to be used to sample the P frames in the GOP sequence, some of the first-category P frames are sampled; if it is determined based on business information that the second sampling ratio needs to be used to sample the P frames in the GOP sequence, all of the first-category P frames are sampled; if it is determined based on business information that the third sampling ratio needs to be used to sample the P frames in the GOP sequence, all of the first-category P frames are sampled, and all or some of the second-category P frames are sampled; wherein the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

[0183] Exemplarily, the processing module 112 is further used to determine whether it is necessary to extract the P frames in the GOP sequence based on the resolution, frame rate and decoding capability if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end; wherein, if the decoding capability does not support the image of the decoding resolution and frame rate, it is determined to extract the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, frame rate and decoding capability; or, if the service information includes the network quality corresponding to the target network between the sending end and the receiving end, it is determined whether it is necessary to extract the P frames in the GOP sequence based on the network quality; wherein, if the network quality is not the optimal level of network quality, it is determined to extract the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality; the sampling ratio is the first sampling ratio, or the second sampling ratio; or, the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio.

[0184] Exemplarily, the division module 111 is specifically used to divide all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames; the first-category P frames cannot be used as reference frames, the second-category P frames can only be used as reference frames for the first-category P frames, and the third-category P frames can be used as reference frames for the first-category P frames, can be used as reference frames for the second-category P frames, and can be used as reference frames for the third-category P frames; the processing module 112 is specifically used to extract the first-category P frames, or extract the first-category P frames and the second-category P frames, or extract the first-category P frames, the second-category P frames and the third-category P frames if it is determined that the P frames in the GOP sequence need to be extracted.

[0185] Exemplarily, when the division module 111 divides all P frames in a GOP sequence into first-category P frames, second-category P frames, and third-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames, second-category P frames, and third-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, for all P frames in the GOP sequence, there are two first-category P frames and one second-category P frame between two adjacent third-category P frames, and one second-category P frame is located between the two first-category P frames; there are two first-category P frames and one second-category P frame between the I frame of the GOP sequence and the first third-category P frame, and this second-category P frame is located between the two first-category P frames.

[0186] Exemplarily, for each first-category P frame, if the previous frame of the first-category P frame is not a first-category P frame, the previous frame of the first-category P frame is used as a reference frame for the first-category P frame; if the previous frame of the first-category P frame is a first-category P frame, the most recent non-first-category P frame is used as a reference frame for the first-category P frame; for each second-category P frame, if a third-category P frame exists before the second-category P frame, the most recent third-category P frame before the second-category P frame is used as a reference frame for the second-category P frame; if no third-category P frame exists before the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame; for each third-category P frame, if a third-category P frame exists before the third-category P frame, the most recent third-category P frame before the third-category P frame is used as a reference frame for the third-category P frame; if no third-category P frame exists before the third-category P frame, the I frame of the GOP sequence is used as a reference frame for the third-category P frame;

[0187] Exemplarily, when the processing module 112 determines that it is necessary to extract the P frames in the GOP sequence, it extracts the first type of P frames, or extracts the first type of P frames and the second type of P frames, or extracts the first type of P frames, the second type of P frames and the third type of P frames. Specifically, if it is determined based on business information that the first sampling ratio is required to extract the P frames in the GOP sequence, all the first type of P frames or part of the first type of P frames are extracted; if it is determined based on business information that the second sampling ratio is required to extract the P frames in the GOP sequence, all the first type of P frames are extracted, and all the second type of P frames or part of the second type of P frames are extracted; the second sampling ratio is greater than the first sampling ratio; if it is determined based on business information that the third sampling ratio is required to extract the P frames in the GOP sequence, all the first type of P frames are extracted, all the second type of P frames are extracted, and all the third type of P frames or part of the third type of P frames are extracted; the third sampling ratio is greater than the second sampling ratio.

[0188] Exemplarily, the processing module 112 is further used to determine whether it is necessary to extract the P frames in the GOP sequence based on the resolution, frame rate and decoding capability if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence and the decoding capability corresponding to the decoding end; wherein, if the decoding capability does not support the image of the decoding resolution and frame rate, it is determined to extract the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the resolution, frame rate and decoding capability, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, it is determined whether it is necessary to extract the P frames in the GOP sequence based on the network quality; wherein, if the network quality is not the optimal level of network quality, it is determined to extract the P frames in the GOP sequence; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio.

[0189] Exemplarily, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the division module 111 is further configured to use the I frame of the first GOP sequence as a reference frame for the first frame of the GOP sequence.

[0190] Based on the same application concept as the above method, an image processing device is proposed in an embodiment of the present application, as shown in Figure 12, the image processing device includes a processor 1201 and a machine-readable storage medium 1202, and the machine-readable storage medium 1202 stores machine-executable instructions that can be executed by the processor 1201; the processor 1201 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0191] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0192] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0193] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Dividing all P frames in a group of pictures (GOP) sequence into at least two types of P frames, wherein there is a P frame that cannot be used as a reference frame among the at least two types of P frames, and there is a P frame that can be used as a reference frame among the at least two types of P frames; If it is determined that the P frames in the GOP sequence need to be extracted, then the P frames that cannot be used as reference frames are extracted; or, if it is determined that the P frames in the GOP sequence need to be extracted, then the P frames that cannot be used as reference frames and the P frames that can be used as reference frames are extracted.

2. The method according to claim 1, characterized in that The method comprises: Classifying all P frames in the GOP sequence into a first category of P frames and a second category of P frames; wherein the first category of P frames cannot be used as reference frames, the second category of P frames can be used as reference frames for the first category of P frames, and the second category of P frames can be used as reference frames for the second category of P frames; If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted.

3. The method according to claim 2, characterized in that The dividing all P frames in the GOP sequence into first-category P frames and second-category P frames comprises: Based on the position sorting relationship between the P frames in the GOP sequence, all P frames in the GOP sequence are divided into first-category P frames and second-category P frames; wherein, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between the I frame of the GOP sequence and the first second-category P frame; wherein, M1 is a positive integer, and M2 is a positive integer.

4. The method according to claim 2 or 3, characterized in that For each first-category P frame, if there is a second-category P frame preceding the first-category P frame, the second-category P frame closest to the first-category P frame is used as a reference frame for the first-category P frame; if there is no second-category P frame preceding the first-category P frame, the I frame of the GOP sequence is used as a reference frame for the first-category P frame; For each second-class P frame, if there is a second-class P frame before the second-class P frame, the second-class P frame most recently before the second-class P frame is used as the reference frame of the second-class P frame; if there is no second-class P frame before the second-class P frame, the I frame of the GOP sequence is used as the reference frame of the second-class P frame.

5. The method according to claim 2, characterized in that If it is determined that the P frames in the GOP sequence need to be extracted, then the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted, including: If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the first sampling ratio, all or part of the first-category P frames are decimated; if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the second sampling ratio, all or part of the first-category P frames are decimated; or, If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a first sampling ratio, some of the first-category P frames are decimated; if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a second sampling ratio, all of the first-category P frames are decimated; if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using a third sampling ratio, all of the first-category P frames are decimated, and all or some of the second-category P frames are decimated; Among them, the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

6. The method according to claim 5, characterized in that The method further comprises: If the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determining whether it is necessary to perform frame extraction on the P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images with the resolution and the frame rate, determining to perform frame extraction on the P frames in the GOP sequence; and if so, determining a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability; or, If the service information includes network quality corresponding to a target network between a transmitting end and a receiving end, determining whether to perform frame extraction on the P frames in the GOP sequence based on the network quality; wherein, if the network quality is not an optimal level of network quality, determining to perform frame extraction on the P frames in the GOP sequence; and if so, determining a sampling ratio corresponding to the GOP sequence based on the network quality; Wherein, the sampling ratio is the first sampling ratio or the second sampling ratio; or, The sampling ratio is the first sampling ratio, the second sampling ratio, or the third sampling ratio.

7. The method according to claim 1, characterized in that The method comprises: Classifying all P frames in the GOP sequence into a first category of P frames, a second category of P frames, and a third category of P frames; wherein the first category of P frames cannot be used as reference frames, the second category of P frames can only be used as reference frames for the first category of P frames, and the third category of P frames can be used as reference frames for the first category of P frames, can be used as reference frames for the second category of P frames, and can be used as reference frames for the third category of P frames; If it is determined that the P frames in the GOP sequence need to be extracted, the first type of P frames are extracted, or the first type of P frames and the second type of P frames are extracted, or the first type of P frames, the second type of P frames and the third type of P frames are extracted.

8. The method according to claim 7, characterized in that The dividing all P frames in the GOP sequence into first-category P frames, second-category P frames, and third-category P frames includes: Based on the position order relationship between the P frames in the GOP sequence, all P frames in the GOP sequence are divided into a first category of P frames, a second category of P frames, and a third category of P frames; Wherein, there are two first-category P frames and one second-category P frame between two adjacent third-category P frames, and the one second-category P frame is located between the two first-category P frames; There are two first-category P frames and one second-category P frame between the I frame of the GOP sequence and the first third-category P frame, and the second-category P frame is located between the two first-category P frames.

9. The method according to claim 7 or 8, characterized in that For each first-class P frame, if the previous frame of the first-class P frame is not a first-class P frame, the previous frame of the first-class P frame is used as the first-class P frame. The reference frame of the frame, if the previous frame of the first-type P frame is a first-type P frame, the most recent non-first-type P frame is used as the reference frame of the first-type P frame; For each second-category P frame, if there is a third-category P frame preceding the second-category P frame, the third-category P frame closest to the second-category P frame is used as a reference frame for the second-category P frame; if there is no third-category P frame preceding the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame; For each third-category P frame, if there is a third-category P frame before the third-category P frame, the third-category P frame most recently before the third-category P frame is used as a reference frame for the third-category P frame; if there is no third-category P frame before the third-category P frame, the I frame of the GOP sequence is used as a reference frame for the third-category P frame.

10. The method according to claim 7, characterized in that If it is determined that the P frames in the GOP sequence need to be extracted, then the first-category P frames are extracted, or the first-category P frames and the second-category P frames are extracted, or the first-category P frames, the second-category P frames, and the third-category P frames are extracted, including: If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the first sampling ratio, all or part of the first-category P frames are decimated; If it is determined based on the service information that a second sampling ratio is required to be used to decimate the P frames in the GOP sequence, decimating all first-category P frames and decimating all or part of the second-category P frames; wherein the second sampling ratio is greater than the first sampling ratio; If it is determined based on business information that a third sampling ratio is required to sample the P frames in the GOP sequence, all first-category P frames are sampled, all second-category P frames are sampled, and all third-category P frames or some third-category P frames are sampled; the third sampling ratio is greater than the second sampling ratio.

11. The method according to claim 10, characterized in that The method further comprises: If the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determining whether it is necessary to perform frame extraction on the P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images with the resolution and the frame rate, determining to perform frame extraction on the P frames in the GOP sequence; if so, determining a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, If the service information includes the network quality corresponding to the target network between the transmitting end and the receiving end, it is determined based on the network quality whether the P frames in the GOP sequence need to be extracted; wherein, if the network quality is not the optimal level of network quality, it is determined that the P frames in the GOP sequence need to be extracted; if so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio.

12. The method according to claim 2 or 7, characterized in that The method further comprises: When the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the I frame of the first GOP sequence is used as a reference frame for the first frame of the GOP sequence.

13. An image processing device, characterized in that: The device comprises: a dividing module, configured to divide all P frames in a group of pictures (GOP) sequence into at least two types of P frames, wherein some of the at least two types of P frames cannot be used as reference frames, and some of the at least two types of P frames can be used as reference frames; The processing module is used to extract P frames that cannot be used as reference frames if it is determined that P frames in the GOP sequence need to be extracted, or to extract P frames that cannot be used as reference frames and P frames that can be used as reference frames if it is determined that P frames in the GOP sequence need to be extracted.

14. The device according to claim 13, characterized in that The division module is specifically used to divide all P frames in the GOP sequence into a first category of P frames and a second category of P frames; wherein the first category of P frames cannot be used as reference frames, the second category of P frames can be used as reference frames for the first category of P frames, and the second category of P frames can be used as reference frames for the second category of P frames; the processing module is specifically used to extract the first category of P frames if it is determined that the P frames in the GOP sequence need to be extracted, or to extract the first category of P frames and the second category of P frames.

15. The device according to claim 14, characterized in that When the division module divides all P frames in the GOP sequence into first-category P frames and second-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames and second-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, there are M1 first-category P frames between two adjacent second-category P frames, and there are M2 first-category P frames between the I frame of the GOP sequence and the first second-category P frame; M1 and M2 are both positive integers.

16. The device according to claim 14 or 15, characterized in that For each first-category P frame, if there is a second-category P frame preceding the first-category P frame, the second-category P frame closest to the first-category P frame is used as the reference frame of the first-category P frame. If there is no second-category P frame preceding the first-category P frame, the I frame of the GOP sequence is used as the reference frame of the first-category P frame. For each second-class P frame, if there is a second-class P frame before the second-class P frame, the second-class P frame most recently before the second-class P frame is used as the reference frame of the second-class P frame; if there is no second-class P frame before the second-class P frame, the I frame of the GOP sequence is used as the reference frame of the second-class P frame.

17. The device according to claim 14, characterized in that The processing module determines that it is necessary to perform frame decimation on the P frames in the GOP sequence, and then decimates the first-category P frames, or decimates the first-category P frames and the second-category P frames, and is specifically configured to: if it is determined based on service information that it is necessary to perform frame decimation on the P frames in the GOP sequence using a first sampling ratio, then decimate all or part of the first-category P frames; If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the second sampling ratio, all the first-category P frames are decimated, and all or part of the second-category P frames are decimated; Alternatively, if it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the first sampling ratio, then some of the first type of P frames are decimated; If it is determined based on the service information that the second sampling ratio needs to be used to decimate the P frames in the GOP sequence, all the first-category P frames are decimated; If it is determined based on business information that a third sampling ratio is required to be used to sample the P frames in the GOP sequence, all first-category P frames are sampled, and all second-category P frames or some second-category P frames are sampled; wherein the second sampling ratio is greater than the first sampling ratio, and the third sampling ratio is greater than the second sampling ratio.

18. The device according to claim 17, characterized in that The processing module is further configured to, if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determine whether it is necessary to extract P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images of the resolution and the frame rate, determine to extract P frames in the GOP sequence; if so, determine a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability; or, if the service information includes a network quality corresponding to a target network between a transmitting end and a receiving end, determine whether it is necessary to extract P frames in the GOP sequence based on the network quality; wherein, if the network quality is not an optimal level of network quality, determine to extract P frames in the GOP sequence; if so, determine a sampling ratio corresponding to the GOP sequence based on the network quality; the sampling ratio is a first sampling ratio or a second sampling ratio; or, the sampling ratio is the first sampling ratio, the second sampling ratio, or the third sampling ratio.

19. The device according to claim 12, characterized in that The division module is specifically used to divide all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames; the first-category P frames cannot be used as reference frames, the second-category P frames can only be used as reference frames for the first-category P frames, and the third-category P frames can be used as reference frames for the first-category P frames, can be used as reference frames for the second-category P frames, and can be used as reference frames for the third-category P frames; the processing module is specifically used to, if it is determined that the P frames in the GOP sequence need to be extracted, extract the first-category P frames, or extract the first-category P frames and the second-category P frames, or extract the first-category P frames, the second-category P frames and the third-category P frames.

20. The device according to claim 19, characterized in that When the division module divides all P frames in a GOP sequence into first-category P frames, second-category P frames and third-category P frames, it is specifically used to: divide all P frames in the GOP sequence into first-category P frames, second-category P frames and third-category P frames based on the position sorting relationship between the P frames in the GOP sequence; wherein, for all P frames in the GOP sequence, there are two first-category P frames and one second-category P frame between two adjacent third-category P frames, and the one second-category P frame is located between the two first-category P frames; there are two first-category P frames and one second-category P frame between the I frame of the GOP sequence and the first third-category P frame, and the second-category P frame is located between the two first-category P frames.

21. The device according to claim 19 or 20, characterized in that For each first-category P frame, if the previous frame of the first-category P frame is not a first-category P frame, the previous frame of the first-category P frame is used as a reference frame for the first-category P frame; if the previous frame of the first-category P frame is a first-category P frame, the most recent non-first-category P frame is used as a reference frame for the first-category P frame; For each second-category P frame, if there is a third-category P frame preceding the second-category P frame, the third-category P frame closest to the second-category P frame is used as a reference frame for the second-category P frame; if there is no third-category P frame preceding the second-category P frame, the I frame of the GOP sequence is used as a reference frame for the second-category P frame; For each third-category P frame, if there is a third-category P frame before the third-category P frame, the third-category P frame most recently before the third-category P frame is used as a reference frame for the third-category P frame; if there is no third-category P frame before the third-category P frame, the I frame of the GOP sequence is used as a reference frame for the third-category P frame.

22. The device according to claim 19, characterized in that The processing module determines that it is necessary to perform frame extraction on the P frames in the GOP sequence, then extracts the first type of P frames, or extracts the first type of P frames and the second type of P frames, or extracts the first type of P frames, the second type of P frames, and the third type of P frames. Specifically, if it is determined based on the service information that it is necessary to use a first sampling ratio to extract the P frames in the GOP sequence, then extracts all or part of the first type of P frames; If it is determined based on the service information that the P frames in the GOP sequence need to be decimated using the second sampling ratio, all the first-category P frames are decimated, and all or part of the second-category P frames are decimated; The second sampling ratio is greater than the first sampling ratio; if it is determined based on the service information that the third sampling ratio needs to be used to sample the P frames in the GOP sequence, all the first-category P frames are sampled, all the second-category P frames are sampled, and all or part of the third-category P frames are sampled; The third sampling proportion is greater than the second sampling proportion.

23. The device according to claim 22, characterized in that The processing module is further configured to, if the service information includes a resolution corresponding to a GOP sequence, a frame rate corresponding to the GOP sequence, and a decoding capability corresponding to a decoding end, determine whether it is necessary to extract P frames in the GOP sequence based on the resolution, the frame rate, and the decoding capability; wherein, if the decoding capability does not support decoding images with the resolution and the frame rate, determine whether to extract P frames in the GOP sequence; if so, determine a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, if the service information includes a network quality corresponding to a target network between an encoding end and a decoding end, determine whether it is necessary to extract P frames in the GOP sequence based on the network quality; wherein, if the network quality is not an optimal level of network quality, determine whether to extract P frames in the GOP sequence; and if so, determine a sampling ratio corresponding to the GOP sequence based on the network quality, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

24. The device according to claim 14 or 19, characterized in that in, When the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, the division module is further configured to use the I frame of the first GOP sequence as a reference frame for the first frame of the GOP sequence.

25. An image processing device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps described in any one of claims 1-12.